EDUCATORS GUIDE

TO SERVICE-LEARNING
PROGRAM EVALUATION

INTRODUCTION
Service-learning has been demonstrated to be an effective educational strategy and can have
powerful effects on students academic performance, personal/social development, civic
engagement, and career knowledge. Any individual service-learning program, however, may or
may not be successful in achieving its desired outcomes. While much is known about which
strategies are effective, it is only in implementing service-learning activities that you can know
with assurance the degree to which any individual program will succeed.
The way you ensure that service-learning programs are effective in achieving their promise long
term, is by evaluating the programs. With the results of these evaluations, you will know which
programs to continue, how to strengthen them, and which ones to end.
This booklet will provide you with the tools to:
_
_
_

Evaluate how effectively a program meets its objectives;

Diagnose which factors contribute to successful or unsuccessful outcomes; and
Provide guidance for improvement.

This booklet is organized into three sections.

1. Section 1 provides a general overview of program evaluation and suggests specific steps to
follow in planning an evaluation.
2. Section 2 provides worksheets to use in conjunction with Section 1.
3. Section 3 provides detailed guidance for conducting the research.
The best way to use this guide is to first read through Section 1 to get a broad sense of the task at
hand. When you are ready, you can then reread Section 1, using Sections 2 and 3 as guides for
planning and conducting a program evaluation.

SECTION 1
OVERVIEW AND STEP-BY-STEP PLAN FOR PROGRAM EVALUATION
What is Program Evaluation?
Program evaluation, in the most general sense, is a process of gathering information about a
program to measure and understand the programs results. Think of it as an experiment, the kind
students learn about in high school science class.
_

You develop hypotheses that some set of actions (the program) will result in some set of
outcomes. These hypothesized outcomes are usually stated as program goals or objectives.
For example: A program in which students work with senior citizens at a senior center to
develop an oral history of their community (the program) will strengthen students planning
skills, interview skills, writing skills, and presentation skills, and help students develop
empathy for seniors; understand the effects of aging, learn historical perspective and gain
respect for older citizens (the goals).

You then run the experiment (implement the program) and gather data to test your
hypotheses.

You then analyze the data, draw conclusions regarding the impact of the program, and
generate insights into why the program performed as it did.

The purpose of program evaluation is not to conduct the experiment, but to support decisions
regarding how to proceed. While it is not formally part of the evaluation process, the final step
in any evaluation is to use the findings to make decisions regarding program retention and/or
program improvement.
Should a Program be Formally Evaluated?
Not every program needs to be formally evaluated and not every evaluation needs to be a fullblown evaluation.
A formal evaluation should be conducted if:
_
_

there are significant resources or student consequences riding on the success of the program;
or
this is a trial program, and the results of the trial will be used to make decisions regarding
whether to continue or how to improve the program.

There is little value in formally evaluating a program if:

_
_

it is a one time program that will not be repeated; or

the intended impact of the program is very limited.

Use worksheet #1 to work through this process.

General Design Principles for Program Evaluation

Program evaluation primarily focuses attention on the outcomes of a program. The heart of any
program evaluation is answering the basic question: Were the objectives of the program met?
That is, was there a positive change in the measures the program was designed to improve?
There are two general ways of measuring change.
1. Before and after design; and
2. Comparative design.
Before and after design: In a before and after design, you gather information about key
outcome items both before the program and after the program to determine whether the
hypothesized change has occurred.
The before and after design is usually the preferred method to measure change. It sidesteps the
difficulty in comparative designs of having well-matched groups. It also avoids the complication
that, during the program period, the program group or a comparison group could be exposed to
other things that would confuse the analysis. The before and after method, however, requires
advanced planning. Before measures need to be taken before the start of the program. If no
comparison group is available, then a before and after method is the only alternative.
This method, however, measures only the impact of an individual program. Without similar
measures taken for other programs or for students who experience no program, there is no way to
assess the comparative value of a program.
Comparative design: In a comparative design, you identify or create two or more groups that
are similar to each other prior to the start of the program and that will continue to have similar
experiences, other than the program to be evaluated. In the experiment, some groups receive the
experimental treatment (the program). Others may experience other programs, variations of the
program, or no program at all (the control groups). You evaluate the effectiveness of the
program by comparing each group on the outcome measures.
A comparative design is used when comparison of the program to other programs or several
variations of the program is important, or when no before measure is possible, such as when
the evaluation was decided upon after the program was started.
Tool Kit for Conducting a Program Evaluation
Deciding whether to evaluate a program and how to evaluate it should be integral to the planning
process. When program evaluation is managed as an afterthought, options may be limited and
the value of the evaluation often suffers. Following are the steps you need to take in evaluating a
service-learning program.

STEP 1:

Define the program objectives in such a way that they can be

objectively measured.

If you want to know whether and how well a program met its objectives, the objectives need to
be stated in such a way that facilitates evaluation. You need to decide specifically what you need
to know in order to assess the accomplishment of the objectives. Here are some examples.
_

If improving academic achievement, social skills, or career readiness are program objectives,
you will need to define these objectives in such a way that you will be able to measure them,
either using standard measures or creating measures of your own. For example, you may
specify:
_
_
_

academic achievement as the ability to recognize and solve math problems in the areas of
budgeting and scheduling.
social skills performance as respect for others or facilitating consensus building.
career readiness as the level of insight into various life choices, or the presence of skills
required to enter a chosen field.

The key here is to refine an objective to the extent that you will be able to measure it with
confidence. (This refinement also helps in focusing your resources and attention in your
program.)
The Compendium of Assessment and Research Tools (CART) Internet resource may be a useful
tool for this step. See resources for more information about CART.
Use worksheet #2 to work through this process.
STEP 2:

Decide on the kind of analysis you want to conduct.

All program evaluations need to measure the programs success at meeting its objectives, that is,
its outcomes. This is fairly straightforward. If the objective of the program is to improve a
particular factor (e.g., leadership skills), you simply need to measure that factor before and after
the program, or compare that factor for the group that experienced the program with the factor
for a group that did not experience the program.
In addition, it is important to establish why the program improved or failed to improve. This
requires you to link what was done or what was experienced with the resultant outcomes
Examples of factors that might be viewed as correlates or predictors of outcomes include:
_
_
_
_
_
_

type of reflection activities;

voluntary or compulsory participation;
intentional linkage of activities to outcomes;
integration with curriculum;
student choice and voice in planning;
rewards or incentives;

_
_
_

school grade;
socioeconomic status; and/or
family involvement.

For example, if you hypothesized that length of time in a program was important to a successful
outcome, you could measure each participants time in the program and each participants
outcome results and see if there was an association. For this type of analysis to work, there must
be some variation. If all students spent the same amount of time in the program, you would not
be able to test time as a factor in the outcome in the program.
Even if the time factor were the same for all students in a program, you could test the time factor
using a comparative approach. You could run two or more programs using differing program
lengths (or find other comparable programs using a different program length) and compare
outcomes for each program.
You will need to decide whether your evaluation will examine outcomes exclusively or also
examine factors that might affect the outcomes.
Use worksheet #3 to work through this process.
STEP 3:

Select a general evaluation research design: before and after or

comparison groups.

As discussed previously, there are two general research designs used in program evaluation:
1. Before and after design
2. Comparison design
You will need to select one or the other. The general guidelines are:
Select a before and after design if:
_
_

the focus is on evaluating or improving program performance, not comparing various

programs or program variations; and/or
the evaluation will begin before the program has started.

Select a comparison design if:

_
_

the focus is comparing various programs or program variations; and/or

comparison groups are available.

Use worksheet #4 to work through this process.

STEP 4:

Select the sources of information.

After you have decided upon the types of information you will need to evaluate your program,
you will need to select the sources for that information. There are a variety of potential sources
for the kinds of information people need to evaluate a service-learning program. All the
potential sources of information should be considered.
_

Sources of primary information (information that you collect directly from people)
_
_
_
_
_
_
_

students;
teachers;
administrators;
parents;
recipients of the service;
community representatives; and/or
others.

Sources of secondary information (information that is gathered independent of the

program)
_
_
_

school data such as attendance reports;

results of standardized tests; and/or
others.

To decide which sources to use in your evaluation, first identify the information you need in
order to evaluate the program (based on the objectives and diagnostics you have specified); then
determine the most effective and practical sources of that information. Multiple voices or
perspectives are often desirable.
Use worksheet #5 to work through this process. Note: Step 5 below also uses this same
worksheet.
STEP 5:

Decide what measurement methods to use.

There are a variety of ways of measuring performance or capturing data. You will need to
choose which method(s) best fit(s) your needs and resources for each of your information
sources. The choices are listed below.
_

Quantitative methods: Quantitative methods are for quantifying; that is, for measurement.
This is usually required for program evaluation. Questions like: Was there a change? How
big was the change? What proportion of students achieved the objectives are best addressed
using quantitative methods. This can include surveys; tests; or other objective measures of
outcomes such as grades, graduation rates, or attendance records. Surveys can be selfadministered (paper questionnaires) or personal interviews (face-to-face or phone).

In most cases, quantitative methods are the most useful method of program evaluation.
These methods typically gather information on each participant (students, parents,
community members) or a sample (subset) of participants.
_

Qualitative methods: Qualitative methods are best for gaining a deep understanding of
program participants experiences, motivation, opinions, concerns, and so on; that is, those
matters for which you need a conversation to really understand. The primary tools for
qualitative research are:
_
_
_

focus groups (moderated group discussions);

personal interviews (one-on-one interactive discussions, not simple question and answer
sessions); and
observation (in-class or field observations are used for capturing information on what is
actually done and generating a qualitative sense of the experience).

Decisions about the types of measurement methods you will use needs to be addressed in the
context of the specific information you need. For example:
_
_
_

You may choose to measure academic performance using a standardized test.

You may choose to measure social skills using direct observation or feedback from
personal interviews with program clients.
You may choose to measure career readiness using a personal interview.

A variety of methods may be needed.

Use worksheet #5 to work through this process.
STEP 6:

Conduct the research for the evaluation.

At this point in the process, you have completed the overall structure and design for the
evaluation program. Next comes the detailed design and execution of each of the studies you
have specified. The general steps in conducting research include:
1. Write data collection instruments, which include questionnaires, focus group discussion
guides, interview guides, and observation instructions.
2. Develop and execute a sampling plan. Decide how many respondents and what kind of
respondents to include in the research, and then select those respondents.
3. Train data collectors.
4. Collect the data (conduct the survey, personal interview, focus group, or observation)
5. Analyze the results:
_
_

Tabulate and organize the data into a form that is manageable for analysis; and
Examine the data to test hypotheses and derive conclusions.

The specific methods for conducting each of the various forms of research are presented in
Section 3 of this booklet.

SECTION 2
WORKSHEETS
WORKSHEET 1:

SHOULD A PROGRAM BE FORMALLY EVALUATED?

Link to Section 1, page 2: Should a program be formally evaluated

It would be appropriate to evaluate a program if either of the items below are checked.
r There are significant resources or student consequences riding on the success of the program
r This is a program trial. The results of the trial will be used to make decisions regarding
whether to continue the program or how to improve the program.
It may not be worthwhile evaluating a program if either of the items below are checked.
r It is a one-time program that will not be repeated.
r The intended impact of the program is very limited.
Decision:
r Evaluate
r Do not evaluate

WORKSHEET 2:

DEFINE PROGRAM OBJECTIVES

Link to Section 1, page 4: Step 1

Define all program objectives to a level of detail that facilitates evaluation.
_
_

List all program objectives. These are the expected outcomes resulting from the program.
Then define the objectives specifically enough that you will be able to measure or assess
whether or how well they were achieved.

Example:
Objective:
First level of refinement:
Final level of refinement:

Improve academic performance.

Improve reading and language arts skills.
Improve students ability to read with fluency, write expository
essays, provide well-reasoned, well-organized presentations.

Objective:
Refinement
Refinement

________________________________________________
________________________________________________
________________________________________________

Objective:
Refinement
Refinement

________________________________________________
________________________________________________
________________________________________________

Objective:
Refinement
Refinement

________________________________________________
________________________________________________
________________________________________________

Objective:
Refinement
Refinement

________________________________________________
________________________________________________
________________________________________________

Objective:
Refinement
Refinement

________________________________________________
________________________________________________
________________________________________________

Objective:
Refinement
Refinement

________________________________________________
________________________________________________
________________________________________________

When completed, these refined objectives become the foundation for further evaluation planning.

WORKSHEET 3:

DECIDE THE KIND OF ANALYSIS YOU WANT TO CONDUCT:

OUTCOMES OR DIAGNOSTICS

Link to Section 1, page 4: Decide on the kind of analysis you want to conduct.
The information you gather and the complexity of analyzing that information is affected by
whether the program evaluation measures outcomes only or both outcomes and correlates or
predictors. Use this worksheet to help decide which kind of analysis you want to conduct.
r Outcomes only:
Conduct an outcomes only analysis if the only objective of the evaluation is to measure
whether and/or to what extent the objectives (intended outcomes) were achieved.
r Outcomes and correlates or predictors of outcomes:
Outcomes most likely need to be determined in all program evaluations. If there is also a
desire to understand what program features are associated with (cause) stronger or weaker
outcomes, diagnostic measures will be needed.
If you decide that understanding correlates or predictors are needed, these extra steps will be
needed.
1. Identify those program features that are hypothesized to affect outcomes. These will be
the features on which you will need to gather information for the diagnostic analysis.
2. Be sure the program design produces appropriate levels of each feature you listed in Step
1 above. As noted before, you can only do analysis on a feature if there are various levels
of that feature. Stated in research language: variables must vary. For example:
r The length of time a student spends in the program varies from one to four semesters
or between one to five days a week.
r Two versions of leadership are conducted; one with student leaders and one with
adult leaders.
3. Define the features to the same depth as the outcomes defined on Worksheet 2. Record
the feature definitions below.
Example:
Feature:
Refinement
Refinement

Length of time in the program

Number of semesters
________________________________________________

Feature:
Refinement
Refinement

________________________________________________
________________________________________________
________________________________________________

10

Feature:
Refinement
Refinement

________________________________________________
________________________________________________
________________________________________________

Feature:
Refinement
Refinement

________________________________________________
________________________________________________
________________________________________________

11

WORKSHEET 4:

DECIDE ON THE GENERAL DESIGN OF THE STUDY

Link to Section 1, page 5: Select a general evaluation research design.

There are two general designs for program evaluation:
1. Before and after design
2. Comparative design
Use the form below to help decide which design to choose.
Program characteristics appropriate for a before and after design:
r The focus of the evaluation is on measuring the effectiveness of the program, not on
comparing the program to other programs.
r It will be feasible to measure the program objectives both before and after the program.
Program characteristics appropriate for a comparative design:
r A key objective of the evaluation is comparison of the program to other programs or several
variations of the program.
r Well matched comparison groups exist or can be created, and, you will have access to these
comparison groups for assessment.
Decision:
r Before and after design
r Comparative design
r Combined before and after and comparative design

12

WORKSHEET 5:

SOURCES OF INFORMATION AND METHODS OF DATA

COLLECTION

This worksheet brings together three important elements of the evaluation design:
1. Program objectives;
2. Sources of information; and
3. Methods of data collection.
Link to Section 1, page 5: Select the sources of information.
You will need to have first completed Worksheet 2. Record the objectives from Worksheet 2
onto the form below. Then, for each of the objectives, check off which source(s) will be needed
to assess that objective. Each of the cells you have checked will represent a separate study
design, data collection and analysis effort. If you have checked many cells, you may need to
prioritize to keep the effort to a manageable level.
SOURCE

Objective 1

OBJECTIVES
Objective 2
Objective 3

Objective 4

Students
Teachers
Administrators
Parents
Recipients of the
service
Community
representatives
Other

Link to Section 1, page 6: Decide what measurement methods to use.

Once you have completed the task above, you will then need to decide what form of data
collection to employ for each study (cell you have checked on the table above). For each study
(column on the table below) you should check one method (cell).
DATA COLLECTION METHOD

Study 1

Quantitative
Existing or independent source
Survey
Personal interview
Self-administered paper questionnaire
Qualitative
Class or field observation
Personal interview
Focus group

13

STUDY
Study 2
Study 3

Study 4

SECTION 3
DETAILED GUIDANCE FOR CONDUCTING THE RESEARCH
Once you have completed the worksheets, you will have made all the major design decisions for
your program evaluation. The next step is to design and carry out each of the studies that you
planned. In this section, there are three separate guides that provide instruction for carrying out
each of the three types of studies.
1. Quantitative research: surveys and analysis of existing data
2. Qualitative research: focus groups and personal interviews
3. Observation

14

GUIDE FOR DESIGNING AND CONDUCTING QUANTITATIVE RESEARCH:

SURVEYS AND EXISTING RECORDS
Conducting survey research (creating and analyzing new information) and conducting research
using existing records (e.g., attendance records, grades, or scores from standardized tests) share
the same foundation, but there are also variations in the activities you need to conduct. In the
scenario below, survey research is presented first, followed by a discussion of variations
appropriate when working with existing records.
STEP 1:

Designing the Research.

There are several elements to designing a survey project.

Choose who to interview. There are two key terms relevant to this stage of the design:
population and sample. The population is all the people who make up the group in which you
are interested. Examples may be:
_
_

Students participating in the project

Recipients of the service

The population needs to be precisely defined so it is clear who is included and who is not.
When populations are small and manageable, it is feasible to include the entire population in a
study. When populations are large, it becomes less feasible and unnecessary to study the entire
population. In these cases, a sub-group of the population (called a sample) can be interviewed.
It is expected that programs you will be asked to evaluate will be small in scale so that sampling
will not be necessary. If populations are large in any of your studies, you should seek
professional assistance in sample design.
For each of your studies, state in detail who is included in the population.
Method of data collection. There are a variety of survey methods at your disposal. The most
basic methods are:
_

interviewer assisted
_
_

telephone interviewing
personal interviewing

self-administered
_

mailed questionnaire or other distribution method

15

Interviewer assisted interviewing is more expensive, but provides several benefits.

_
_
_

The cooperation rate is higher than for self-administered methods since it is easier for people
to discard a questionnaire than to tell the interviewer that they decline to participate.
The quality of the information can be better since the interviewer can detect and clarify any
confusion that may exist on the part of the respondent.
The sample control is better.
_
_

You can be sure that only qualified people are completing the questionnaire.
You have control over the sample size, rather than waiting for an undertermined amount
of returned questionnaires.

If the flow of the questionnaire is at all complicated, having an interviewer manage the flow
can reduce errors.

Self-administered interviewing is less expensive and also provides a unique advantage as well.
_

When the content of the questionnaire is sensitive or when anonymity is important,

respondents are more comfortable with self-administered questionnaires.

Select a method of data collection: interviewer assisted or self-administered.

Writing the questionnaire.
At this point, you know what you need to learn (the objectives and feature definitions from
Worksheets 2 and 3) and you have decided on a sampling and data collection method. Now it is
time to write your questionnaire, which has four parts:
1.
2.
3.
4.

Introduction
Screener
Body
Close

The introduction: The introduction is extremely important. In todays environment, many

people view interviews as a bother at best, or an intrusion at worst. You need to get your foot in
the door, and the introduction is how you do it. In the introduction the interviewer needs to
introduce him/herself, identify the sponsor of the research, indicate what the research is about,
and why the respondent should care enough to take the time. This needs to be done quickly.
People will not read or listen to long stories. Be sure the tone is respectful. Remember, you are
asking people to give you their time, attention, and personal information, all valuable
commodities.
The screener: Once you have your foot in the door, you need to be sure the person is a qualified
respondent. Either state who this interview is for We are talking to parents and guardians who
have children attending Smith Middle School this year, or ask questions that qualify the
respondent: Do you have any children in this household who currently attend Smith Middle

16

School? For people who do not qualify, explain to whom the survey is directed and politely end
the interview.
The body: The body of the questionnaire is where the questions are asked. Your guidance for
writing the questions comes from the objectives and features you listed on Worksheets 2 and 3.
Working from this list, you need to convert each information objective into a question or series
of questions that will yield the information you need. Only ask what you need to know. Do not
succumb to the temptation to add interesting and nice to know questions that do not derive
from the list of objectives and features. These unnecessary questions add length and time to the
interview, increasing your non-response rate and your costs.
There are a number of question styles you can choose from. They generally fall into one of two
generic categories.
1. Closed-ended questions: These are tightly controlled questions in which you provide a list
of answers from which the respondent must choose or questions for which the possible
answers are pre-coded on the questionnaire but not revealed to the respondent. Types of
closed-ended questions include:
_
_

_
_
_

Yes/no questions
Verbal or numeric scale questions: very satisfied, somewhat satisfied, somewhat
dissatisfied, very dissatisfied; definitely, probably, might or might not, probably not,
definitely not; strongly agree, somewhat agree, somewhat disagree, strongly disagree;
always, often, sometimes, usually not, rarely or never; 5 or 10 point rating scales with
labeled endpoints, such as 10 equals completely satisfied and 1 equals completely
dissatisfied, and so on.
List questions: Which of the following topics would you like to see on the program?
Un-aided questions: What topics would you like to see on the program?
Narrative responses: What do you plan to do when you graduate? Go to a four-year
college, go to a trade or vocational school,.

When writing the response categories (answers) for a close-ended question, do your best to
anticipate all the likely responses. As a catchall, it is often useful to leave a fill-in response
of Other _____________ for respondents whose responses you have not anticipated.
Depending on the nature of the question, including the responses Dont know, and
Refused may also be appropriate.
There are several benefits of closed-ended questions.
_
_

They capture information in a form you design to be most easily analyzed.

They are efficient in capturing information and in processing information.

Formatting: On self-administered questionnaires, you should provide a number or letter for

the respondent to circle, or a box to check next to the response category.

17

2. Open-ended questions: These are questions in which respondents fill in the blank. There
are several general situations in which open-ended questions are used.
_
_
_

When not enough is known to anticipate the likely responses. These are exploratory
questions.
When you do not want to influence or bias a respondent by offering a list of answers to
choose from.
When you want to hear the respondents language, to see what words they use.

Formatting: On self-administered questionnaires, you usually provide a number of lines

under the question where respondents or interviewers can write in their response.
There are a number of additional issues to consider in writing a questionnaire.
Flow: Go from general to specific, and from interesting to boring. Other than questions needed
to screen and qualify respondents, place demographic and other classification questions at the
end of a questionnaire.
Sometimes one question can provide information or otherwise influence how a respondent
answers another question. Put questions in an order that will minimize this kind of effect.
Language: It is amazing how easy it is to write a question that is confusing, ambiguous or
otherwise open to misinterpretation. Since the validity of the research is predicated on all
respondents understanding a question to mean the same thing, you need to put a lot of attention
into how you word questions. Here are some general rules.
_
_
_

Keep the language simple and direct. Avoid double negatives. Avoid professional
jargon unless the survey is directed to a professional audience.
Avoid using words with high emotional content.
Write questions in a neutral way so that all responses are equally acceptable. Even
though you, as an individual, may have a point of view, as a researcher your interest is in
learning other peoples points of view. Tipping your hand as to your personal position in
the way questions are worded will often impact the way people answer questions. Some
people want to please or are motivated to provide socially acceptable answers, even when
those answers do not reflect their positions. You want to write questions in a way that
makes all opinions and positions acceptable. Sometimes the best way to do that is to
state both sides of a position rather than only one. Here is an example using reading to
children as the subject.
Based on personal beliefs and family situations, families in our
community differ in how often they read to their preschool children.
Some read to their children often, some occasionally, and some rarely
or never. On average how often is your preschool child read to by
family members each week?

18

r
r
r
r
r

Rarely
Less than one day a week
1 to 2 days a week
3 to 4 days a week
5 or more days a week

Compare this approach with a one-sided approach to the same issue.

To what extent do you agree or disagree with the following statement:
Preschool children should be read to at least five days a week?
r
r
r
r

Strongly agree
Somewhat agree
Somewhat disagree
Strongly disagree

Since there is a bias that good parents read to their children, getting honest information
from people who do not read to their children requires us to acknowledge to the respondent
that there is a range of choices and that we recognize that people have reasons for their
choices.
Double-barreled questions: A double-barreled question is when two issues are embedded into a
single questions such that you cannot be sure which question is being answered.
How satisfied are you with the math and reading instruction at this
school?
Unless respondents are equally satisfied with math and reading instruction, they dont have a
very good way of answering the question, and you will not know how to interpret their response.
Skip patterns: On many questionnaires, some questions need to be answered or skipped, based
on the response to an earlier question. For example, one question might ask community
members if they had attended any collaboration meetings. A block of questions asking further
information about the collaboration meeting experience should be answered only by people who
attended meetings. Use instructions such as, If No, skip to Question 8, to guide the
interviewer or self-administered respondent. For self-administered surveys, keep skips to a
minimum.
Formatting: There are two general approaches to tabulating survey findings: hand-tabbing and
data processing. Either way, you will want to lay out the questionnaire in a way that will
facilitate the tabulation process. Several standards help.
_
_
_

Number your questions.

Line your response categories up near the right hand margin of the page to make
scanning the answers efficient.
Number or letter the response categories.

19

If you will be tabulating the results by computer, you will want to label each question
(usually in the right hand margin) to indicate where on the data record the response to the
question should be placed. (A 20 next to the response categories of Question 10
indicates to the data entry people that the coded answer to Question 10 (yes equals 1)
should be entered in position 20 of the data record). You may want to request
experienced support if you plan to use computerized data tabulation.

The Close: When all the questions have been asked and answered, you need to end the interview
with an appropriate statement of appreciation.
Confidentiality: It is standard for respondents comments to be confidential. Unless there is an
overriding need to be able to track or identify individual respondents, names or other identifying
information should not be recorded on the questionnaire. When the identity of the respondent
needs to be retained (e.g., when you need to link the results of a survey with information from
another source of data), replacing names with respondent ID numbers is a useful way of
protecting confidentiality. If the information will not be confidential, respondents must be
advised of this. Further, special care must be taken to control and protect the raw data.
The outcome of this step is a written, formatted questionnaire.
STEP 2:

Conducting the Survey.

Interviewer training: If you are conducting an interviewer-assisted survey (telephone or faceto-face), you will probably be conducting the survey using volunteer interviewers. You need to
take the time to train the interviewers. The key elements of interviewer training are:
_

Brief interviewers on the background and objectives of the survey. They may be called upon
to exercise some judgement in the course of an interview, and you want them to have a
strong basis for making those judgements.

In general, you want interviewers to read the question exactly as written. You know how
much effort you put into the detailed wording of each question. You do not want
interviewers paraphrasing questions.

Interviewers should not coach respondents. If a respondent does not understand the intent of
a question or answers a question in a way that implies he/she misunderstood, the interviewer
should clarify the question. Otherwise the interviewer should not discuss the question or
provide guidance. Under no circumstances should an interviewer discuss his/her views on
the subject of the interview.

It is strongly recommended that interviewers practice the interview with other interviewers
several times prior to conducting their first interview with a real respondent. This practice
helps interviewers get a sense of the flow of the interview. And interviewers are less likely
to stumble or become confused if they have actually gotten the words out beforehand and are
familiar with the questions.

20

At this point, you are getting close to being almost ready to go ahead with the study. But first,
you want to give the questionnaire and the overall methodology a thorough test.
Pilot test: Before you actually conduct a survey, you will want to try it out in a pilot test. Pilot
testing a questionnaire is different from the kind of practice sessions described above. In a pilot
test, the respondent is a member of the target audience who believes the interview counts. The
purpose of a pilot is to test out the questionnaire exactly as it will be presented in the survey to be
sure the questions are clear and correctly understood by the respondent, to be sure the flow
works well, and all instructions are clear. It is difficult to anticipate some of the problems that
may emerge when real respondents get hold of a questionnaire. You certainly do not want to
discover problems after interviewing has started. In a self-administered questionnaire, the pilot
respondent needs to be interviewed after completing the questionnaire on his/her own to discuss
each of the issues noted above.
Administering the survey: The specifics of conducting a survey will be different depending on
the specific methodology.
_

Telephone interviews: Interviewers will be given a list of names/numbers to call.

Depending on the type of respondent, they may have to focus their calling during daytime
hours or during the evenings. Interviewers should expect difficulties in reaching people such
as getting no answer, having no qualified respondent at home, getting an answering machine,
and so forth. If someone answers the phone but the interview cannot be done at that time,
interviewers can try to schedule a time for the interview. Interviewers should call at different
times of the day, and weekday and weekend times to give themselves the best chance of
reaching a person. You should set a standard for the number of callbacks, such as four or
six attempts, before an interviewer abandons a number as unreachable.
Interviewers should maintain a record of their efforts. For each number, they should record
the time and result of each call attempt. They should also keep a running tally of progress:
number of completed interviews, number of refusals, number of people not available or
cant be reached after the specified number of attempts, number of people who did not pass
the screener. These tallies should be gathered and compiled daily in order to track progress.

Personal interviews: Personal interviews can be conducted with a list of specifically

targeted individuals, such as parents of students in a class, or using an availability sample,
similar to interviews at shopping malls where potential respondents are approached
(intercepted) and invited to participate in an interview. The latter approach might be
appropriate at an event such as a school fair.
In studies where personal interviews are to be conducted and respondents will be recruited
using sample lists, you typically will need to make arrangements to schedule the interviews.
Interviews can be conducted at a school, at the respondents home, or at another location.
The location should be selected based on some balance between the respondents
convenience and the timing, resources, and efficiency needed in conducting the survey. It is
best if the interview can be conducted in a quiet comfortable place where the participants will
not be disturbed.

21

For intercept interviews, where you simply stop people and ask them questions, you will
need to set up a location where the interviewer and respondent can sit, unless the interview is
very short and can be done standing up using a clipboard.
In intercept surveys, it is a challenge to get a random or representative sample since you do
not randomly select respondents off of a list. It is human nature for interviewers to approach
people with whom they would feel most comfortable. This, however, will produce a skewed
sample. A simple approach to take the human choice factor out of respondent selection is for
the interviewer to pick a number (say five) and approach the fifth person he/she passes.
_

Self administered questionnaires: The key activity in self-administered questionnaires is to

develop a method for distribution and retrieval. For distribution, the most common methods
are mail, student take-home, and meeting/event handout. Retrieval methods include mail and
return with student. For questionnaires distributed at onsite events, the most effective
method is to ask people to complete the questionnaire onsite and to return it to a collection
point while still at the event. Return rates plummet once people leave the premises.

Administration: Research managers need a system for monitoring and tallying progress of the
survey. Questionnaires need to be checked in, counted, and stored appropriately.
STEP 3:

How to Analyze Survey Results.

Once the data collection is completed, it is time to figure out what you have learned. As with
any kind of research analysis, you begin the analysis process by revisiting the research
objectives, the decisions you need to make, and the information you decided you needed. Using
this information, you build an outline of your report, specifying the sections you will write and
the information that will go into each section. You enter the analysis process with a clear idea of
the issues you need to address and the type of information you have to work with.
Once you have reviewed the purpose of the research, it is time to begin tabulating. Tabulating
the closed-ended questions is straightforward. Tabulating open-ended questions takes more
work. Since quantitative research is all about counting and measuring, you need to convert the
narrative responses of open-ended questions into numbers. This is done in three steps.
_

Step 1:

Make a list of all, or a sample of the responses to an open-ended question.

Step 2:

Examine the list and figure out what the themes are. For example, in response to
a question about reasons for participating in an after school service-learning
program, you may receive comments about meeting people, doing better in
school, having someplace fun to go, and so forth. Even though the language in
each comment may be unique, you can gather them together into common themes.
Sometimes themes will have sub-themes. For example, a higher level theme may
be student centered benefits, and lower level themes might include specific
student benefits, such as better scores, improved self esteem, and so forth. Once
these themes are identified, you need to assign each theme a unique numeric code.

22

Step 3:

In this step, you go through each individual questionnaire and write down the
numeric code that corresponds to the open-ended response. Now the open-ended
response is in a format that can be counted and categorized just like the closedended responses.

Tabulation: Tabulation can be done either by hand (hand-tabbing) or with a computer. The
advantage of using a computer is that it is easier to break out data in different ways. For
example, you could look separately at the responses of different age groups or genders. Doing
this by hand becomes very time consuming and tedious. If a questionnaire is short and the
analysis is simple, hand-tabbing may be the most efficient way to proceed.
The downside of computer tabulation is that it requires specialized equipment (software) and
skills that may or may not be available. It can also be more time consuming to enter the data,
program the questionnaire, and run the program than to hand-tab a simple questionnaire.
The way to hand-tab a questionnaire is to set up a recording sheet modeled after the
questionnaire, with a place to put a tic mark for each response. One by one, go through each
questionnaire and place a tic mark signifying the response to each question. For example, if a
respondent said Yes to Question 1, place a tic mark next to the Yes on Question 1 on the
tally sheet. When you are finished, add up the tic marks and calculate percentages for each
question.
Analysis: With the data now in a form you can use, it is time to start making sense of it. Rather
than looking at each question and reporting how many respondents said what, center your
analysis on the decisions you need to make, which findings shed light on those decisions, and
what the implications of those findings are.
The mistake often made by inexperienced analysts is to treat all information equally and report
out everything that was learned. This makes for boring reading and is not a useful tool for
helping people make decisions. A function of analysis is to sift through all the data and decide
what is important. Focus the analysis on what matters, the information and insights that will help
readers learn what is important and support the decisions they need to make.
For reports that will be read outside the study team, it is useful to follow a simple report outline.
_
_
_
_
_
_
_

Cover page;
Table of Contents;
Executive Summary, a brief summary of each section of the report that someone can quickly
read and understand the objectives, methodology, findings, and conclusions;
Introduction, including a discussion of the background, objectives and methodology
employed in the research;
Findings, main body of the report;
Conclusions or recommendations, if appropriate; and
Appendix, any important documents, such as the questionnaire, that would be useful in
understanding the findings or methodology.

23

Variations Appropriate for Research Using Existing Records

Population and sample: When using existing data, there is still a need to define the population
and consider issues of sampling. For example, if school attendance is the measure, you will need
to define specifically who should be included in the population.
Method of data collection: While there is no direct data collection using existing records as
there is with a survey, you will need to extract the information from its source. This may be a
manual or electronic process. In either case, you will need to create a form and layout for
capturing and recording the information that facilitates analysis. In some cases, this may involve
coding narrative information, especially if you plan to analyze the data electronically. For
example, you may want to recode an attendance check mark indicating a students presence
into a 1 if present and 2 if absent.
Analysis: Analysis of existing data is conducted in the same way as survey data. In some cases
the existing data are analyzed separately and independently. In other cases, they are folded into
other data (e.g., attendance records added to a database generated from a survey of the same
students) and analyzed together.

24

GUIDE FOR DESIGNING AND CONDUCTING QUALITATIVE RESEARCH:

FOCUS GROUPS AND PERSONAL INTERVIEWS
Conducting focus groups (moderated group discussions) and conducting qualitative personal
interviews share the same foundation, with slight variations. In the paragraphs that follow, focus
group research is presented first, followed by a discussion of variations appropriate when
conducting personal interviews.
STEP 1:

Designing the Research. Focus groups are most effective when the members of
a session are similar to each other (homogeneous) on key characteristics. For
example, if the program you want to study has been implemented in K-5, it is
likely that the needs and experiences of kindergarten teachers will be very
different from those experienced by fifth grade teachers. Putting both in the same
focus group makes for weak dynamics. Similarly, putting together teachers and
administrators, teachers and students, or teachers and parents does not work well.
The best focus groups are populated by people who come from the same
perspective. In the focus group session, you explore for differences within that
common perspective. Sometimes you even hold separate focus groups among
people you know like a program and people who you know hate a program,
although mixing them up and watching the differences emerge is also appropriate,
as long as the participants are similar on the basic characteristics (e.g., all teachers
involved in the program).

This means you need to decide what the key differences are in your populations that you do not
want to mix. This multiplies the number of focus groups you need to conduct. It also improves
immeasurably the quality of the information.
Focus groups are discussions among a small number of people. Whenever you select a small
number of people from a larger population, it is always possible to get an odd bunch; one that is
not representative of the range of opinions. For this reason, whenever possible, always at least
double up, conducting a minimum of two focus groups among each of the key groups.
There also may be constituency issues that need to be addressed in order to gain credibility. It
may be necessary to include various schools, districts, regions, states, or other constituents in a
study in order to earn credibility and gain buy-in of the various sponsoring or interested
constituencies. Sometimes it is methodologically important to do so. Other times it is needed
for reasons of inclusion or support. Regardless of which, it is a design consideration that needs
to be taken into account.
The product of this step is a description of the type of participants to invite and the number of
focus groups to conduct.

25

STEP 2:

Normally, two activities are carried out simultaneously in Step 2.

1. Planning and logistics

2. Preparing the discussion guide
Planning and logistics: You need to decide whom to invite (also called recruiting), when and
where to hold the sessions, and then invite participants.
Selecting and recruiting participants does not need to be as rigorously random as with
quantitative research. Still, you want to recruit as random and representative a group of people
as possible. Ideally, selection and recruiting is done randomly from lists. Be careful not to
delegate the selection of participants to someone with a point of view who might load the group
with people who share a common opinion. Recruiting can be done on the phone or via
messages.
The ideal focus group size is between eight to ten participants. You typically recruit 12,
assuming some people will cancel at the last minute or simply not show.
Be sure to confirm the recruitment of all participants by sending them in writing the information
they need. They will need the following information.
_
_
_
_

The topic and purpose of the study;

The location of the focus group sessions (directions if appropriate);
Date and time, including when the session will conclude; and
Information about travel compensation if appropriate.

Emphasize that the meeting will start promptly at the time specified, so people should arrive five
minutes ahead of time. Reconfirm all participants one to two days before their session.
Facilities: You will need to select a facility that is appropriate for conducting focus groups. Key
features are:
_
_
_
_

a room large enough to hold 12 people comfortably;

appropriate furniture (conference-like table);
a quiet undisturbed environment; and
a convenient location.

Conference-like table arrangements are best for focus groups. You want a single table (or
smaller tables pushed together) at which there is ample room and everyone can see each other.
Square or rectangular arrangements work well. In a pinch, people can sit in a circle without a
table. The ability to close doors to seal out outside noise is also important.
Length: A focus group session usually lasts one to two hours, depending on the amount of
material needed to be covered. Two hours is usually the maximum you can hold peoples
attention.

26

Time line: Typically you want to allocate two weeks for recruiting, that is, start recruiting two
weeks before the focus groups are scheduled.
Preparing the discussion guide: The role of a moderator, which will be discussed in detail
below, is to facilitate the discussion among the participants on topics of interest to the
researchers. The discussion guide is an outline the moderator uses to guide the discussion.
Typically a discussion guide has three main sections.
1. Introductions;
2. Warm-up and general discussion; and
3. Discussing the details.
Introductions: There are two parts to the introduction section:
1. the moderator introducing him/herself and providing information on the discussion to come
and
2. participant introductions
Presented below is a script of a typical introduction.
Moderators introduction:
_

Introduce yourself (name, role) and thank participants for coming.

As you know, you have all been invited here to participate in a discussion
about(be general). This is a form of research we call a focus group. Is there
anyone here who has never been in a focus group?

Give quick overview regardless.

_

Key points:
Group discussion.
Looking for all opinions, expect diversity, differences of opinion.
Not looking for consensus or seeking a majority.
Your job: To let me know what you think. All opinions are equally valid. Speak only for
yourself and from your own experience. It is best not to speak for the group.
My job: My job is to keep us on task and on time and to be sure everyone has their fair
share of air time. Works best when Im not the center of attention. I will start things off.
Then you talk among yourselves.

_
_

Assure confidentiality.
Explain about audio taping (to help in preparing a report).

27

_
_
_

Tell them the length of the session. Plans for break or no break, location of restrooms,
any necessary fire or evacuation procedures.
Restate the purpose and value of the session.
Ask for and answer any questions.

Participant introductions:
Ask participants to introduce themselves by providing the basic information you will need to
understand who they are in the context of the research (e.g., Mary, fifth grade teacher, Adams
Elementary school, been teaching ten years, three at Adams. This is my first year with the
program). If you can arrange for an easel in the room, it is useful to write the list of information
you want on the easel beforehand.
_

Warm-up and general discussion: The flow of the second and third sections of the
discussion guide goes from general to specific. A typical question to ask in the second
section is: As you know, each of you in this room [is a teacher who taught the new
curriculum, is a principal at a school where you are piloting Program Y]. Tell me what it was
like to [teach the new curriculum, introduce the new program]. The purpose of this section is
to get the energy flowing, to get people engaged in the discussion, and to get a high level, yet
personal understanding of the range of experiences regarding the subject you are studying.

Discussing the details: The third section is where you get to the details. Use the list of
information objectives you prepared earlier as the basis for this section. Organize the issues
into a logical order, thinking about how the discussion will flow most naturally. Then using
short phrases or sentences, write down the questions you will ask that will elicit the
discussion you want. Sometimes you will literally ask a question: What was it like?,
What did you like best about?, How did you approach? When possible, you will
suggest a discussion: Tell me about the students reaction to. Questions should almost
always be open questions; that is, questions that elicit a story, an explanation, or a point of
view. Yes/no questions or ones that elicit a short answer do not produce insights, which is
what focus groups do best.
In addition to straightforward discussion, there are a number of tools and techniques that a
moderator can employ to elicit participation and insights. A useful tool for getting juices
flowing is group brainstorming with the moderator posting ideas on an easel. Some tools are
useful to prevent group think such as assigning participants the task of writing down as
many ideas as they can in a few minutes, then debriefing participants with an open
discussion.
Some tools stimulate creativity or engage right brain thinking.
_
_

collage building
projective techniques such as: If this program were a vehicle, what kind of vehicle
would it be. Explain why?

28

Sometimes it is useful to use handouts or other displays (e.g., a list of the features of a new
program) if you need participants to be well informed for a discussion.
Timing: It is always a challenge to fit the discussion to the time you have available. Once you
have the discussion guide drafted, estimate how long each section will take. If it runs too long,
you will have to pare it down. There are two ways to pare down a discussion guide: drop issues
or questions that have a lower priority or shorten the amount of time you will allow discussion
on each subject. It is useful to write target start and finish times for each section in the margin of
the guide to help the moderator manage time well.
The physical guide: The moderator will use the guide in the course of the focus group session;
therefore, it must be easy and efficient to use. Some people write out each question or
instruction verbatim. Others write phrases, key words or short sentences as reminders, and then
ask the questions in their own words. In contrast to surveys in which each question must be
asked identically to each respondent, the moderator has more leeway in a focus group, just as
long as the point gets across. You should share the moderators guide with others on your team
for feedback, since it specifies exactly what will be discussed and makes clear what will not be
discussed. Do not distribute the guide to focus group participants.
Conducting the Focus Group
Managing Logistics

Arranging the room: As noted above, you want the room to be set up conference style. If it is
not already set up that way, you will need to do some rearranging of furniture. Always arrive at
a new facility with enough time to adjust furniture if necessary. Set up the moderators seat
where it will be easiest to have eye contact with all the participants, usually at the long end of a
rectangular table or in the middle of a side of a square table. Put materials (papers, coat) at that
position before participants enter the room so others will not take it. Do what you can to make
the room a quiet, comfortable place to have a discussion.
Directing people to the location: Often, participants will not be familiar with the facilities
where the focus group will be held. You want to make it easy for participants to find the location
of the focus group session. Place signs as appropriate.
When focus groups are scheduled consecutively, there should be a designated waiting area where
participants for the next group can wait. It is best for there to be a greeter to welcome
participants as they arrive, answer questions, and provide instructions if needed. Snacks and
beverages in the waiting area are appreciated. If there is a waiting area, bring people into the
focus group room several minutes before you want to start to allow them to settle in.
It is useful to have name cards for participants and the moderator. They help quickly put people
on a first name basis which greatly enhances group dynamics. It is best to not include last names
to enhance the feeling of confidentiality.

29

Beginning the Session

As the moderator, you are the host. Greet people. Introduce yourself. Make them comfortable.
Make small talk. Create a friendly, informal, welcoming environment.
You may want to wait up to five minutes for latecomers, but assuming you have a full agenda
and out of respect for those people who arrived on time, it is best to start promptly. Latecomers
can join as they arrive. (In commercial focus groups, latecomers are often paid their incentive
and sent home.) Once the small talk is finished and you have enough people in attendance, it is
time to formally start the focus group.
You start by re-introducing yourself, thanking participants for coming, and launching into the
discussion guide.
Becoming an Effective Moderator
Effective moderators come in all sorts of packages. They are not all the same, nor do they all
moderate in the same ways. There are a number of attributes, however, that are common to the
best ones. Effective moderators have:
_
_
_
_
_
_
_
_

a genuine interest in people, their experiences, and what they think and feel;
appreciation for the differences in people;
the ability to suspend ones own views, feelings, and beliefs, and search for the truth as
viewed by the participants;
good listening skills including the ability to recognize what is not being said;
an inquisitive mind, wanting to understand, to know why;
good observation skills (body language, facial expressions);
the ability to ask questions and express oneself clearly;
self-confidence
_
_

feeling comfortable in a leadership position

feeling comfortable with a group of strangers; and

flexibility and ability to quickly deal with the unexpected.

Presented below are suggestions for successful moderating.

Introductions are usually done in order around the table. Afterward, discussion is random, with
whoever wants to speak doing so. During participant introductions, make eye contact and thank
each participant when they are done. A little small talk and interaction during these
introductions helps establish rapport and informality.
During participant introductions, an aid for the moderator is to draw a participant map on a sheet
of paper with names and key information.

30

Know what you want to learn. Do not just ask questions. It may take several shots at a question
and a lot of probing to get what you need. By having the objectives clearly in your mind, it gives
you the flexibility to innovate and change approaches as the unexpected problems and
opportunities pop up.
Practice asking questions. The words will come out more smoothly if it is not the first time you
say them. Be prepared to ask questions in different ways if the first way is not working. It is
best not read directly from the moderators guide (unless it is a formal statement that needs to be
presented the same way each time). Make what you are saying feel conversational.
Listen! Do not get caught up in asking questions, planning for your next question, or taking
notes. Your job is to actively listen and respond to what you hear.
Keep in the background. The objective of the focus groups is to learn what the participants have
to say. You want to maximize participants air time. If you subtract your introduction time and
the time you need to ask questions, each participant is already down to about ten minutes of talk
time in a two-hour session.
Spend enough time framing questions so participants understand the context and intent of the
question. Be sure framing is objective and neutral.
Assume you do not know the answer to what you are asking. That way, you will be open to
hearing the unexpected.
Watch for all forms of communication by speakers and by others (e.g., words, body language,
tone, facial expressions). Ask people to translate observed non-verbals [Mary, I noticed that
you smiled/looked at Jane when Joe said . Was that a smile? What were you thinking?]
People are different. Some talk a lot, some are quiet. Some are interesting. Some are tedious.
Respect all styles. Encourage the quiet ones to participate by asking directed questions and a lot
of eye contact. But do not overdo it. It is often easier for quiet ones to compare or contrast with
others who have already spoken, rather than asking them to go first. Often the quiet ones are
good at reflecting once everything else is said. Sometimes they simply have nothing much to
say.
It is very dangerous to cut people off. You can shut them down and possibly, lose the whole
group. Wait for them to take a breath, then quickly grab the floor, often asking others to
comment on the same subject. Selma, whats your experience on this issue?
An even bigger challenge is the self described expert or spokesperson. People often will
defer to the expert, resulting in much of the group shutting down. If this occurs, say to the
expert, Id like to ask a special favor of you. Because of your special knowledge on this
subject, Id like to save your comments for last. I definitely want to hear what you have to say,
but lets use your knowledge to help sum things up. Would you do that for me? They will
usually comply, but may require a reminder.

31

When there is a spokesperson in a group, rather than speaking directly to that person, say to
the whole group, Its important in this type of research that each individual speaks only for
him/herself. That way we get all the possible views that are out there. So when each of you
offers a comment, please make the comment from your own personal experience and opinion
and allow others to do the same.
The most common problem is a dead group (i.e., quiet, disinterested, tired). Antidotes are: the
moderators enthusiasm and energy; directing questions to individuals (Ellen, tell me about
your first day with the new students); and facilitating interaction among the participants (Juan,
what do you think about what Ayesha just said?). Doing some type of activity also helps raise
the energy level.
Some people will tune out, check out, even go to sleep. When this occurs, leave them alone.
Rarely, people will be belligerent or come in drunk. If they are disruptive to the group, you
should politely ask them to meet with you outside of the focus group room, and while there,
firmly dismiss them.
Be an active listener, probing to get deeper, to find out why, to clarify ambiguities. Do not be
satisfied with surface answers. You want more than answers; you want understanding. Be
active in exploring inconsistencies; not to resolve them, but to understand them. Surface answers
include: It is more convenient [find out what convenience means to them]. I like that better
[What is better about it?] Id never do that [Why not?] We tried that and we didnt like it.
[Describe what you tried and what you didnt like about it?]
The kinds of responses that should be probed or followed up include:
_
_
_
_

those that are very interesting and that you want to understand more deeply;
those that are ambiguous, and you are not sure whether they mean this or that;
when an individual contradicts him/herself. They are not wrong for doing that, but it begs for
deeper understanding; and
when logic would have taken you in a different direction (I thought it was useful. It was
well designed. I didnt do service-learning. Probe: Gee, after hearing you talk about it, I
was sure that you would do service-learning. Tell me about the reasons you didnt.)

Let silence work for you. Be patient for a good 15 or 20 seconds to see what emerges. Do not
be too anxious to fill in dead spots (unless you have just asked a question and they have no clue
what you are asking).
Watch out for leading questions, such as those that imply a preferred response. Dont you think
that...? As you know, most experts in the field agree that.... It is okay to play devils
advocate if you want to probe the alternative of what appears to be a bandwagon position. It is
also okay to introduce a concept that you think is relevant but that has not been raised by the
group. Your purpose is not to sell it, but to determine whether it is not relevant (as would be
implied by it not being mentioned) or whether it is relevant but just got missed.

32

Open vs. closed questions. Closed questions are those that can be satisfied with a short definitive
answer (i.e., yes/no, never/sometimes/often). Open questions require a story. Tell me about
your experience with.... How did you feel when...? What were the students reactions when...?
Closed questions are unavoidable, but they are not what focus groups are best at. If overused,
they turn a focus group into eight one-on-one interviews.
If you are not absolutely clear about a persons point, it is often useful to restate their position in
your own words and ask them whether you have it right. If a conclusion seems to be emerging
from the discussion, it is okay to sum it up in your own words and ask participants if you have it
right. From everything Ive heard, it seems like the real reason you are upset about the change
in policy isnt because of ABC, but rather because of XYZ. Do I have it right? The
participants will either confirm your analysis or will straighten you out. Either way, this
technique improves the quality and reliability of your analysis when it comes time to write your
report.
In-session note-taking: the most important thing is paying attention to what participants are
saying and what you need to do next. If you can do that and take notes too, great. If not, do not
take notes. Rely on a tape recording.
Avoid having observers in the room. It is not disastrous, but they can be distracting. Strongly
avoid observers who have a supervisory or evaluative relationship with the participants. The
argument to use in these cases is to tell the requesting observer that their presence will reduce
participants willingness to be fully candid and, as a result, weaken the research and the quality
of the information. Since the people requesting to be observers will typically be consumers of
the findings, they will not want to be the cause of weakening the research. They will usually
comply with your request. If observers must be present, they have to keep quiet. It is okay,
though, to invite observers to join in and ask questions after the moderators guide has been
completed.
A moderator should appear competent, but not be positioned as an expert in the field.
Participants do not feel comfortable or see the value in explaining things to an expert. (Whats
the point? They already know this better than I do!)
If a participant asks the moderator what his/her opinion is on the topic of discussion, the
moderator should defer answering. Inserting your own opinion can have a profound effect on the
ensuing discussion. Your job is to find out what the participants think. In order to do that well,
you need to be completely objective and to suspend your own opinions. If you reveal your
personal opinions, you run the risk of planting the seed in some peoples minds that you are not
truly objective. In a contentious or politicized environment, that can undermine the credibility of
the research conclusions.
Stay on task, be efficient, maintain control. You are in charge. Do not let people ramble. Do
not let the subject get off track. Once the information is in, move on.

33

Stay objective and independent. It can be useful to praise good focus group behavior, but not
participants programs. Otherwise you are judging and giving participants cues for what the
moderator deems good.
Do not consult during focus group. You are the moderator. You are there to learn from
participants. If you change your role to consultant, the relationship changes.
How to Analyze Focus Group Results
Once the focus groups have been completed, it is time to figure out what you have learned. As
with any kind of research analysis, you begin the analysis process by revisiting the research
objectives, the decisions you need to make, and the information you decided you needed. Using
this information, you build an outline of your report, specifying the sections you will write and
the information that will go into each section. You enter the analysis process with a clear idea of
the issues you need to address and the type of information you have to work with.
The data you have to work with comes in four possible forms.
1.
2.
3.
4.

Audio or video tapes of the focus groups that you can watch or listen to;
Transcripts of the focus groups;
Notes you or others may have taken during the focus groups; and
Recollections and insights from the sessions you may have written down or remember.

Once the outline is prepared, the process of analysis begins with reviewing the tapes, transcripts
and/or notes, and organizing the ideas into common areas. For example, write down together all
the likes and dislikes mentioned about a program, or write down all the comments about reasons
for participation. It is by looking at the lists like these that you gain some insights. If you
conducted focus groups with several populations, such as teachers and administrators, make
different lists so you can determine if the likes, dislikes and reasons are different. That would be
a useful finding. Once these lists are complete, you are ready to start the analysis. For each
section of your outline review aggregated data, figure out what it all means, and begin to write
the story.
There are two dangers to watch out for while analyzing focus group results.
1. The first is preconceived notions or agendas that may creep into your analysis. Since the
data you have to work with are much softer than survey results, your personal views can
insidiously sneak into your thinking and your analysis. The antidote for this is to surface, in
advance, your personal views, and constantly ask yourself during the analysis if your
conclusions are colored by your personal views. A simpler solution is to have the results
analyzed by someone with no preconceived notions.
2. The second danger is letting recollections from the focus groups lead your analysis. Many
focus groups have individuals who are particularly articulate, and some things learned in
focus groups may resonate with you more than others. These are the ones you remember.
They may not, however, be as important as you think they are. It is quite common for

34

conclusions that come from a thorough review of the data to differ substantially from your
recollections and instant analyses.
Since focus groups are discussions, you often get terrific quotes that illustrate a point, or
competing quotes that demonstrate differences. Quotes bring a report to life and build credibility
in the conclusions. You are showing that a point is not simply the analyzers conclusion;
someone actually said this. Sometimes including several quotes that all make the same point is a
useful way to reinforce the prevalence of a position.
Remember that this is not quantitative research. While you can draw some conclusions about
which views are dominant and which are infrequent, if you do enough focus groups, your
primary objective in focus group research is to identify the range of views and gain insights into
them. Avoid giving counts of participants with a particular view.
The mistake often made by inexperienced analysts is to treat all information equally and report
out everything that was learned. This makes for boring reading and is not a useful tool for
helping people make decisions. A function of analysis is to sift through all the data and decide
what is important. Focus the analysis on what matters, the information and insights that will help
readers learn what is important and support the decisions they need to make.
For reports that will be read outside the study team, it is useful to follow a simple report outline.
_
_
_
_
_
_
_

Cover page;
Table of Contents;
Executive Summary, a brief summary of each section of the report that someone can quickly
read and understand the objectives, methodology, findings, and conclusions;
Introduction, including a discussion of the background, objectives and methodology
employed in the research;
Findings, main body of the report;
Conclusions or recommendations, if appropriate; and
Appendix, any important documents, such as the questionnaire, that would be useful in
understanding the findings or methodology.

Variations Appropriate for Personal Interviews

Sample selection: Qualitative research is not intended to be statistically representative. Its key
benefit is its ability to generate indepth insights. Still, interviewing a broadly representative
sample of respondents is important. Capturing insights from only one segment of a population
would yield misleading results.
When a population to be interviewed is small, (e.g., all teachers involved in a program), a census
of all members of the population should be attempted. When the population is too large for a
census to be conducted, then a representative sample should be interviewed. Typically ten
interviews are adequate for gaining a good sense of the range of experiences and opinions.
Respondents can be selected randomly from the population. Alternatively, individuals known to
represent various positions or experiences can be selected.

35

Planning and logistics: Personal interviews are typically arranged by contacting the respondents
then inviting them to participate, and scheduling a time convenient for them. You will need to
set up a location where the interviewer and respondent can sit. Interviews can be conducted at a
school, at the respondents home or at another location. The location should be selected based
on some balance between the respondents convenience and the timing, resources, and efficiency
needed in conducting the interview. It is best if the interview can be conducted in a quiet,
comfortable place where the interview will not be disturbed.
Preparing the interview guide: Much like in a focus group, the interviewer uses an interview
guide to direct the flow of the conversation. Qualitative personal interviews are not rigid
question-and-answer sessions, but rather, exploratory conversations in which the interviewer
endeavors to gain insight into the respondents experience and views. Similar to a focus group
discussion guide, a personal interview guide should have an introduction, a warm-up and general
discussion section followed by the body of the interview in which the major information
objectives of the interview are addressed. In the introduction, the interviewer should introduce
him/herself, explain the purpose of the interview, explain how the findings will be used, and if or
how the respondents confidentiality will be protected.
Conducting the interview: It is important for the interviewer to establish a positive, trusting
rapport with the respondent. The interviewer should present a positive, friendly, and interested
demeanor. Key to successful interviewing is an inquisitive mind. Probe and follow up
frequently to understand the reasons or motivation behind actions, beliefs, and opinions. It is
crucial for the interviewer to be non-judgmental and to not express personal opinions during an
interview.
Recording the information: The interview guide should have generous space for recording
responses. The interviewer should take as much time as is needed to record, in detail, the
respondents responses. In some cases, key words may be adequate. Other times, complete
verbatim transcriptions that capture the essence of a respondents views are needed and will help
bring the analysis to life.
Analysis: Analysis is conducted in the same way as for focus group research.

36

OBSERVATION
Observing classroom or field activities can be a useful source of insight and is highly
recommended as part of a service-learning program evaluation. It can provide a context or
framework for an evaluation. Observation must be thought of as a formal analytic tool.
Informally sitting in on a field activity is not adequate. Presented below is guidance for
designing a plan for observation as an element of program evaluation.
_

Sampling: Sampling, in the research context, typically refers to the selection of the
individuals you interview. In the case of observation, sampling refers to choosing the
activities you will observe. You want the activities to be representative of the activities of
the program. That would include observing a representative set of activities and a
representative set of participants. You should typically not rely on a single observation.

You should have a general sense of the session you plan to observe. You should know what
activities are planned, the approximate schedule, and so forth. This will allow you to plan
your observation.

You must specify what you hope to learn, in the same way you would when planning a
survey or focus group. You should prepare a list of specific elements or issues you plan to
examine, such as style of instruction or level of student participation. These elements should
be recorded on the form used by each observer.

Observation: The observer must be as unobtrusive as possible and avoid influencing the
activity. The objective is to observe the activity in its natural state. In most cases, an
observer will sit or stand outside the sphere of activity. If the observers presence is apparent
to the participants, it is wise for the activity leader to introduce the observer, explain the
purpose of his/her presence, and then go on as if the observer were not present. In most
cases, the observer must not participate in the activity or in any way interfere with the normal
flow of the activity. If appropriate, the observer can take notes on the form during the
observation. If note taking would be obtrusive, the observer needs to be familiar with the
form and record his or her observations on the form as soon after the observation as possible.

Permission: You must gain permission from program coordinators to observe their programs
activities. Further, those leading an activity (e.g., teachers) should be informed ahead of
time about the observers presence, purpose, and plans.

37

RESOURCES
_

CART: CART is an Internet resource that may be useful for several purposes. CART is a
Compendium of Assessment and Research Tools for measuring education and youth
development outcomes. The heart of CART is a searchable database of test instruments
(questionnaires). The search engine is organized into categories and subcategories that list
well over 100 factors that have been used as outcomes or design (diagnostic) factors.
_

The list can be used during the planning phase of an evaluation program to stimulate
thinking about what outcomes and diagnostics to study in your evaluation.

The database can also be used for questionnaire development. You can search on a factor
to find questionnaires that have been developed to measure that factor. You can use the
questionnaire or ideas from the questionnaire in your own research design. (Be sure to
get appropriate permission from the questionnaires authors, as described on the CART
Web site. Go to http://cart.rmcdenver.com.)

38