Questionnaires

The questionnaire has become one of the most used and abused means of collecting
information. If well constructed, a questionnaire permits the collection of reliable and
reasonably valid data in a simple, cheap and timely manner. However, there are many
sloppy questionnaires and these yield unreliable data of limited validity and utility. The
decision to use a questionnaire is often motivated by a need to collect routine data from a
large number of respondents who may be in one or several locations, for example, schools
within a district. In this situation a questionnaire can be administered to a group at one
school or it may be widely dispersed throughout the district. Questionnaires can be
efficiently administered by mail; however, unless sufficient care is taken, the rate of return
can be a gamble. Researchers who elect to use this form of data collection are cautioned to
think carefully before they use questionnaires to gather information which could be
gathered in another more efficient or valid way.

A good questionnaire is difficult to construct, and to do the job well it will probably take 10
to 50 hours, including pilot-testing, revision and formatting. While time-consuming,
constructing a questionnaire is a good way to sharpen what you really need to know, and fi
done thoroughly, it will greatly facilitate the data ana- lysis phase of the research. The task
of developing and implementing a questionnaire can be accomplished by following the six
essential steps outlined in this chapter: determine your questions, draft the questionnaire
items, sequence the items, design the questionnaire, pilot-test and revise the instrument, and
develop a strategy for data collection and analysis.

Step 1: Determine Your Questions

To develop valid questionnaires, you must first clearly identify your general information
needs. What types of information do you require and from whom? Accurately
defining your needs is critical and will make it easier to write questions that will
answer your research questions and achieve your goals. Different target groups may
provide different types of information and a clear understanding of the purpose of
the research will help you limit your questions to the relevant information and no
more. In a busy world there is no place for questions which might be interesting to
ask but are not related to the central theme.

Developing sharp general research questions (see Chapter 4) is an essential

step in constructing a questionnaire. As noted in Chapter 14 and illustrated in Exhibit 14.5,
the general questions or issues can be organized into an evaluation or research framework.
This framework helps define the overall topic categories for the questionnaire and provides
a funnel for developing sub questions. To illustrate, consider the specific, yet broad
question, 'Who are the program participants?' This question gives rise to immediate sub
questions such as, what are the demographic characteristics of the participants?' This in turn
can lead to even more specific questions: 'What is the highest level of formal education
held by the participants?' 'What is their gender?" "What is the age distribution of the
participants?" These questions themselves are not questions for the questionnaire; rather,
they indicate what you want to know.
Every broad research question asked should generate a number of sub questions. If your
research relates to a research framework, it will be relatively easy to list many sub
questions which are necessary to fill out your understanding of each main issue. Of course,
not every sub question will be addressed in the questionnaire, for certain sub questions may
be better answered using another technique such as studying files and documents or
interviewing a key informant.

Finally, before drafting the questionnaire items, you should highlight, underline or place a
check-mark beside each sub question that will be addressed through the questionnaire. This
will save you time later and help crystalize your thinking. Asking yourself whether each
question is absolutely necessary is a challenging task for it is easy to have a lot of
questions. It is much harder, yet essential, to restrict yourself to a manageable number of
important questions.

Step 2: Draft the Items

Once you have written your research questions and sub questions, questionnaire items will
become immediately apparent. Brainstorm as many items for each sub- question as possible
to help you gain a full understanding of the breadth and depth of each question. This task
complete, you can move on and select the formats) for the questions. There is an infinite
range of question types and many beginners waste time trying to generate creative
approaches to questionnaire wording and format. My advice is to master six basic question
formats before you attempt alternatives which may get you into trouble. Years of
experience have taught me that fill-in- the-blank, multiple choice, comment on, list, Likert
scales, and rank order questions will serve most needs and achieve reliable and valid
responses. Also, these question formats are familiar to respondents and they should
understand how to complete the items. Thus, these six common question formats should
serve as models for your work:

Fill-in-the-blank:

This common format asks a question and leaves a blank for the response. The stem should
be a complete sentence rather than just a phrase. Normally, the answer should not be more
than a word, number or phrase (see Exhibit 17.1).

Note that the answer blank follows the question. I prefer it to be on the same print line as
the last phrase of the question and of a length appropriate to the length of response
expected.

Multiple Choice:

Multiple choice questions are similar to fill-in-the-blank questions, except the respondent is
given a choice of answers and must check one. Sometimes there are discrete response
options (e.g. sex: male or female) in other instances a range of values. is presented (e.g.
annual income: $10,000-$50,000).
It is preferable to provide the response choices and this requires that you understand and be
aware of the range of possible responses. Normally four to eight defined response choices
are provided. Distinct choices may make the analysis easier and they provide natural
groupings for comparing respondents of various types. Occasionally, a blank response
option is included if you feel that the range of responses may be larger than you have
identified (Exhibit 17.2). Remember though, blank options in a multiple choice question
increase the analytic challenge.

The visual presentation of the multiple choice question is also important. Many people use
a line instead of a box, but this confuses some respondents for, in a long list of response
options, it is not readily apparent which line corresponds to which choice. When using the
box, it should follow the answer choice rather than precede it and when possible, answer
choices should be listed in a single column. In practice this often consumes too much space,
so two columns are used. The danger with more than one column is that people will miss
the second column and choose only from the first. The problem is not serious when the
choices are in numerical sequence or refer toa defined and inclusive set of options such as
one's district of residence.

Comment-on:

The comment-on question is similar in format to the fill-in-the-blank question, except for
one critical difference. The comment-on question attempts to elicit an extensive answer by
posing a question and leaving enough space for the respondent to write a short paragraph.
While this format is essential for in-depth understanding, such questions should not be
overused. They tend to bias the results by giving a greater weight to those who are verbally
expressive and tend to evoke responses only from those with extreme views. Furthermore,
the information is difficult to analyze and can often be obtained in other ways. Extensive
use of these items is a sign that the questionnaire writer has taken an easy route. Thoughtful
use of the comment-on question will pay dividends later on if used appropriately. The
example in Exhibit 17.3 illustrates this type of question.
As with all questions that require a narrative response, the amount of space you provide
will generally indicate the degree of detail you expect. Sometimes a parenthetical note is
added instructing the respondent to add extra sheets as necessary.

List:

Asking the respondent to list things is a good way to find out views in an unbiased way.
This open format forces the respondent to think up answers without having a list of
'acceptable' options from which to choose. In my experience it is best to encourage the
respondent to list a specified number of views, say three or five. Otherwise, the responses
of a few vocal individuals may outweigh the majority who list very little. If each
respondent lists three, then it is easy to conduct a content analysis by counting the number
of times each theme is mentioned. Exhibit 17.4 illustrates this approach.

A useful variation on this type of question is to ask for the list in order of importance. This
provides additional information on the potency of each suggestion and enables you to
weight the order during the analysis. Using the modified statement. 'List, in order of
importance, the three most important skills you acquired during this training session', the
most important skill can be assigned a weight of 3, the second 2 , and the third, 1. Thus, in
this example, first choices are three times as potent as third choices. Another variation on
the list question is to ask the respondent in a subsequent question to pick the one from the
list which is most important.

Likert Scales:

The Likert Scale is one of the most useful question forms. The scale is named after Rensis
Likert who developed this format in 1932. In its most popular form, the respondent is
presented a sentence and is asked to agree or disagree on a three, five or seven-point scale,
as shown in Exhibit 17.5.

Note that a question is not asked. Rather, a clear statement is made and the respondent is
asked to indicate whether the statement reflects his or her views. One secret to having
effective Likert scales is to observe the following rules for sentencing:

• Use single sentences containing only one complete thought.

• statements should be short, rarely exceeding 20 words.
• statements shouldn’t be in the past tense.
• statements should cover the entire range of expected responses. Those which are
likely to be endorsed by almost everyone or by almost no one should be avoided:
• avoid statements that are factual or capable of being interpreted as factual.
• avoid statements that may be interpreted in more than one way.
• avoid the use of universals such as all, always, none, and never.
• words such as only, just, merely should be used with care and moderation.
• avoid the use of words that may not be understood by the intended respondents; and
• do not use double negatives.

While Likert scales can have many response points (three to seven being most
common), a 5-point scale is the most practical for most common purposes. It is easy to
respond to, straightforward to analyze, and sufficient for most needs. Young
children, however, are more comfortable with a 3-point or even 2-point scale. The
issue of whether or not to have a neutral mid-point is often debated. I lean to having a
neutral position for two reasons. Without one, some people will leave the item blank or
mark a mid-point anyway, and second, research has shown that the proportion of people
responding to non-neutral positions when there is no neutral position is similar to the
proportion so responding when there is a neutral point and the neutral responders are
discarded. On certain items a category of ‘not applicable' or 'don't know' category is
sometimes added as an extra response option. It may be positioned to either the right or left
of the other boxes.

Likert scales provide an excellent means of gathering opinions and attitudes and they can
relate to terms other than agree or disagree. Other useful forms of Likert scales are
presented in Exhibit 17.6.
Likert scales provide a great deal of information in a short period of time, and they lend
themselves to simple and effective analysis. Additionally, the data can
provide descriptive information, or it may be manipulated in more complex way. For
example, one may list the findings or use a factor analysis to manipulate the data to look for
underlying patterns of responses.
When using a Likert scale, you must include instructions that describe how to complete the
scale. Typical instructions are included with the sample questionnaire in Appendix 17.3. In
earlier times, much more extensive instructions were given, but today most respondents are
familiar with Likert scales and how to complete them.

Rank:

In this type of question, the respondent is given a list of items and asked to rank them in
order of importance (see Exhibit 17.7). A variation on the rank question asks the respondent
to check all fitting answers. However, the true rank-order item is more powerful than
asking the respondent to check all answers that apply because the act of ranking forces
some difficult decisions. It is better to know that most people considered a certain problem
the most serious with the training program rather than just one of 10 problems. When using
rank questions, one should not normally present more than 10 or 12 items since people can
become confused. If more items are needed, try modifying the question by asking people to
rank the five most important from the whole list and ignore the remainder. Be cautioned
however that this may present difficulties. One difficulty with ranked items is the analysis
since different respondents will rank more items than will others. Thus, one person's second
choice may mean something quite different from another's. When considering the visual
presentation of the rank order question, it is imperative that all response items are listed on
one page. Do not split the list between pages or you will get many response errors.

Step 3: Sequence the Items

Good questionnaires, unless they are very short, generally contain subsections. If
questionnaire items have been developed within a research framework, then the
major questions and sub questions will already be grouped to some degree. If not,
you should attempt to refine the groupings into areas with common dimensions. The
process is not unlike writing a report with subheadings and corresponding sections
Such sections give structure to the questionnaire and communicate a sense of purpose and
order to the respondent. Within each section you will have to sequence the
questions to accommodate two different principles. The first principle, related to a common
debate about questionnaire writing, is whether or not to group questions on the same topic
one after the other, or disperse them throughout the questionnaire. My experience indicates
that questions on the same topic should be grouped together, and questions of similar form
should also be grouped. The second principle is particularly relevant for Likert scale items
since one does not want to repeat the instructions too often. Since these two principles are
not always compatible, you may have to rewrite some questions or perhaps change the
sectioning of the questionnaire so that question types are not constantly shifting back and
forth.
Sometimes it is necessary to use filter questions. Filter questions funnel out responses with
common characteristics and guide the respondent to different branches. For example,
participants who attended a particular management training course will be asked to answer
questions about it; those who did not take the course will skip those questions and move to
another section of the questionnaire. It is necessary when designing branches to make a
flow chart from the filter questions. Extensive page flipping should be avoided and
respondents should not become confused as they move forward through the questionnaire.

Another sequencing consideration concerns the overall organization of the

questionnaire. It should begin with easy, non-threatening questions (see Sudman &
Bradbum, 1982). Questions about age, gender, annual income can be threatening and are
generally best asked at the end rather than the beginning. The various sections should be
organized in logical fashion generally progressing from descriptive type information to
more involved attitudes and opinions. Never start a questionnaire with an open question
that requires much writing, as this will discourage people and lower the response rate.

Proper sequencing will involve question editing and will aid the overall construction of the
questionnaire by identifying redundant or unnecessary questions that can be removed. The
questionnaire should be as short as possible to achieve its purpose. Remember, lengthy
questionnaires require lengthy analyzes. The permissible length is related to the
respondents and their vested interest. in the questionnaire. Typically, questionnaires should
be limited to two to four pages unless the respondents are highly motivated, in which case
up to 16 pages are possible. What motivates a respondent? Certainly, if a respondent
believes that the results will affect them or something they believe in, they will be inclined
to give it their full attention. Alternately, some respondents are motivated by the promise of
a reward for returning the
questionnaire.
Step 4: Design the Questionnaire

The format of a questionnaire is extremely important because it is a major factor in

determining whether the questionnaire will be completed. An attractive, well laid-out and
easy-to -use questionnaire is taken more seriously than one which is thrown together with a
minimum of thought. There are two important aspects to design: individual questions and
the whole questionnaire. For individual questions,
certain types of questionnaire layouts can reduce confusion and contribute to valid
responses. As noted earlier, response options for multiple choice, rank or list for- mats
should be in a single column following the question stem. Fill-in-the-blanks or response
boxes should follow the question rather than precede it. Likert scale responses can follow
the sentence being rated or appear at the right of it.

The second consideration concerns the overall questionnaire format. Years of experience
have proven that a booklet format is best. This format is efficient for several reasons. First,
questionnaires printed on both sides will appear more slender and less onerous to complete
than those which are printed on one side and stapled in the corner. Second, a booklet is
simple to use and easy to produce. Third, booklets decrease the paper requirement and, if
administered through the mail, they also diminish the postage costs; and finally, it can be
produced in a range of sizes, depending on its purpose, but will need to be 4, 8, 12 or
16pages. If there are more than four pages, it should be stapled in two places along the fold
in the manner of a magazine. Because it has to be in four-page multiples, the page size
needs to be chosen so that the information will fit into a four page combination. Some
tolerance regarding this restriction is afforded by the back cover, front cover and even the
back and front inside cover pages.

Unless the questionnaire is short (four pages or less), it is advisable to use the first page as a
cover. Whether or not a cover is used, the questionnaire should have a suitable title, and
there should be an introductory paragraph. The paragraph not only introduces the purpose
of the questionnaire, but it should mention whether the responses are confidential, indicate
that all questions are to be completed and, in the case of mailed questionnaires, identify
where to return the completed questionnaire.

The various sections and questions should be organized to make efficient use of the
available space. Where possible, sections should be self-contained and begin at the top of a
page. Care must be taken to ensure that individual questions are not split but are intact on a
single page. As with any layout, there should be a visually pleasing amount of white space.
Questions must not appear crowded. Open-ended comment questions can serve to open
spaces and number of response lines can be expanded to fill the page. Generally, questions
are numbered sequentially through- out the questionnaire.

In some situations, a questionnaire may need to be sent in two languages.

For example, in Canada, French and English are often required. In such cases one
questionnaire should be in two languages, one on the flip side of the other and both
working into the center. Some people may be inclined to send one copy of each unilingual
questionnaire; however, some respondents find this offensive, and this can bias the results.
Respondents, regardless of linguistic preference, should be treated identically and should be
able to respond in their language of choice.

Occasionally, questionnaires are precoded. That is, they include numbers in

small print adjacent to each possible answer to aid the data entry operator (Appen-
dix 17.2). Personally, I try and avoid the use of precoding as it depersonalizes an
instrument which is already somewhat cold. The procedure is not all that necessary when
there are only 100 or 200 questionnaires. Once numbers get into the 1000's
however, precoding is probably necessary. Exhibit 17.8 summarizes the fundamental
steps in questionnaire design.

Step 5: Pilot-test the Questionnaire

It is always difficult to criticize your own written work and in developing questionnaires it
is essential to obtain comments from at least a small group of the intended respondents.
Pilot-testing will identify ambiguities in the instructions; it will help clarify the wording of
questions, and it may alert you to omissions or unanticipated answers in multiple choice or
ranking questions. Normally, individual questions will be vetted before the draft
questionnaire is assembled. The pilot-testing permits overall reactions including comments
on the length of the questionnaire.
An effective way to pilot-test a questionnaire is to assemble a group of6 to 12 volunteers
and have them complete the questionnaire individually. Encourage to write marginal
comments on the actual questionnaire, then follow-up with a group discussion. After the
pilot-test, review the verbal and written comments, the questionnaire responses and
evaluate its effectiveness. Then if required, revise the instrument. If major changes are
necessary, a second pilot test is advisable.

Step 6: Develop a Strategy for Data Collection and Analysis

Now you have a questionnaire ready to go! You will need to work out a strategy for how
and where to send it. The first part of your strategy is to select a representative sample that
represents all the subjects you want to reach. Then prepare a list. The second part of your
strategy is to decide on the technology you will use to send out your questionnaire. There
are three basic options:

1. Standard:
Questionnaires can be printed, in your office or by a printer, and mailed to respondents.
Respondents fill them out and mail them back. Alternatively, sometimes it is possible to
administer them in a group meeting with the targeted respondents. Results are manually
entered into a database or statistical program for analysis.

2. Optical Scanning
It is possible to print questionnaires so that they can be read by an optical scanner that picks
up the responses automatically. This approach is only beneficial when you have hundreds
of questionnaires. Smaller samples probably do not justify the printing and set-up costs.

3. Electronic Questionnaires
The advent of electronic communication has created yet another way to administer a
questionnaire. Sending questionnaires via electronic mail (e-mail) is growing in popularity
for professionals, organizations and individuals networked through the Internet. This form
of questionnaire is de- signed on a computer and sent as a computer file to the subject via e-
mail. The respondent receives the file, completes the questionnaire on his or her computer,
and e-mails the file back to you. This is most useful for in-house surveys, such as within a
school system.

Develop Cover Letter and Send Questionnaire

Every successful questionnaire comes with a cover letter (Appendix 17.1). The letter should
contain six pieces of information:

• the purpose of the questionnaire.

• who is sending it
• why the respondent was selected.
• where, how, and when to return it.
 • whom to contact if there are further questions; and
• whether and how the results will be shared.

When contacting respondents ensure each subject receives a cover letter, a professionally
developed questionnaire, and a self-addressed return envelope, unless you
Are transmitting electronically.

Monitor Returns and Follow-up Distribution Strategies

You should count on four or six weeks to receive responses to your questionnaires. If your
returns are slow, consider one or more follow-up strategies. First start by tracking the
number of questionnaires returned daily (e-mail lets you know who has not yet replied).
Then send a reminder two weeks after the first mailing. If this reminder does not stimulate
an adequate return, consider an alternate distribution system such as developing a pyramid
network. Pyramid networks use key people to pass out and collect questionnaires on your
behalf. Generally, personal networks are the best way to get a high level of returns.

Data Entry

As soon as the first questionnaires are received you can begin entering the data into a data
base. It is useful to number each questionnaire with a code number that can be used for
such purposes as labeling qualitative comments or to verify a response fi required.
Precoded questionnaires can be entered directly. Questionnaires that have not been
precoded should first be coded. To code a questionnaire, you simply write code numbers
that correspond with the data entry template or code-book on each questionnaire. Code
numbers can be used for quantitative or qualitative data.

A vital part of the data entry process is to check the validity of data entry. Many
professionals enter the data twice and compare the two entries for accuracy. If you do not
do this, it is essential to employ an expert to enter the data. My preference is to use an
expert then a series of statistical analyzes which check for outliers (scores that fall beyond
the normal expected range).

Data Analysis

There are many approaches to questionnaire data analysis. When using quantitative data,
start your analysis when responses dry up. I like to begin with computing the basic statistics
for all items on the questionnaire: frequency distributions, means, standard deviations and
measures of skewness. This permits me to see that each item has variance (spread in
responses) and that the patterns are reasonable. For example, if everyone checked the same
answer category, there may be a coding error or the item may have been misunderstood.
Sometimes respondents write comments on their questionnaire (e.g. 'I don't understand this
question!') that help you figure out what is going on. This phases identifies the background
characteristics of the respondents so that they can be compared to the target population. If
there is a good match (assuming you know something about the target population), then
you have some assurance that your sampling strategy worked.
The next phase probes the data in more statistically dynamic ways. My strategy is to begin
by comparing groups (e.g. males vs. females) using cross-tabulations or graphs. I also
examine interrelationships among variables using correlations and related statistics. This
phase of analysis is the most interesting and soon highlights what is going on. Generally,
you can tell pretty quickly whether you have findings that support your research agenda or
not.