DEFINITIONS, USES AND
TYPES OF EVALUATION
1
Lesson Objectives
By the end of the class session, students should:
Be able to define key terms used in program
evaluation
Understand the uses of program evaluation
Understand the 7 main types of program evaluation
2
Key definitions
Evaluation
Evaluation is the collection and analysis of information
using various methodological strategies to determine the
relevance, progress, efficiency, effectiveness, and
impact of activities of a program/activity/project.
Evaluation research
Evaluation research-is a more rigorous effort than
evaluation because it involves the scientific research
methodology. 3
Key definitions
Assessment
Assessment is commonly interchanged with evaluation.
Assessment is a more limited process with a specific
purpose. It could be part of the evaluation.
Needs assessment
Needs assessment is the first stage in program
implementation.
It is the process of determining the nature and extent of the
4
problem(s) that a program is designed to address.
Why program Evaluation / Uses?
Development programs need to prove they have
made a strong and positive impact.
Different types of evaluations are important in
showing the benefits of the program to – backers,
sponsors and beneficiaries
5
Historical perspective
In the past, measuring development programs
involved pen-&-paper surveys and many people
This was inefficient, expensive and time consuming, at
times generating inaccurate data.
Results would also be available after several months
Today – due to technology, M&E can be done more
effectively & various types of evaluation can be
conduced at various program stages. 6
TYPES OF EVALUATION
7
Choosing types of evaluations
Depends on the program phase (7 types)
Stage of the Purpose Type of Evaluation
project
Conceptualization • Prevent wastage
Phase • Identify potential areas of 1. Formative
(Identification & concern Evaluation
Design) • Increase chances of success
• Optimize the project 2. Process evaluation
• Measure ability to meet 3. Outcome
Implementation targets evaluation
phase • Suggest ways to improve 4. Economic
efficiency evaluation
• Obtain insights into project 5. Impact evaluation
success and impact 6. Summative
Project Closure • Highlight potential evaluation
phase improvements for 7. Goals – based
subsequent projects evaluation 8
Evaluation Type 1 - Formative
also known as “evaluative or baseline assessment”
Generates data on the need & purpose for the
program
Develops the baselines to benchmark during
program monitoring
When
During the development of a new program (design)
During program expansion
What - Gives insights on program’s priorities by
Helping managers to determine areas of concern and
focus
Increasing awareness of the program among the
target population 9
Evaluation Type 1 - Formative
also known as “evaluative or baseline assessment”
What - Gives insights on program’s priorities by
Helping identify key area that need improvement so that
the improvements are done earlier in the program
Allows project managers to refine or improve the
program
How
Conduct sample surveys and focus group discussions
among the population, focusing on whether they are
likely to need, understand and accept program elements
by asking the following questions
Is there need for the program?
What can be done to improve the program? 10
Evaluation Type 2 – Process
also known as “program monitoring”
When
Program implementation begins
During operation of an existing program
What – data generated is useful for
Identifying inefficiencies and streamlining the processes
Portraying the program’s status to external parties
Measuring to understand whether program
activities/strategies are working as they should
(Effectiveness)
Understanding whether the program is reaching its
target population and what the population thinks
11
Evaluation Type 2 – Process
also known as “program monitoring”
Why
To provide an opportunity to avoid problems by spotting
them early
To allow program managers to determine how well the
program is working.
How
Aim to measure number of beneficiaries, how long they
wait to receive services and their experiences.
Therefore, review internal reports and conduct a survey
of program managers and a sample of target population
by asking the following questions.
Who is being reached by the program?
How is the program being implemented?
What are the gaps? 12
Are targets being met?
Evaluation Type 3 – Outcome
also known as “objective – based evaluation”
Generates data on program’s outcomes and to what
extent the outcomes are attributable to the program
itself;
When
After the program has run for some period of time
At any appropriate time, so as to measure outcomes
against set targets
What
Measure the effect of the program on the target
population (understanding of how effective the program
has been)
Ascertain the degree of benefit provided by the
program
13
Evaluation Type 3 – Outcome
also known as “objective – based evaluation”
Why
Helps program managers tell whether program is
meeting its objectives
Gives insights from outcome-focused feedback to help
increase effectiveness.
How
Conduct randomized trials (survey or focus group
discussions), comparing before and during program or
compare with beneficiaries to similar people outside the
program by asking the following questions.
Did participants report the desired change after program
implementation?
What are short or long term results reported by
participants? 14
Evaluation Type 4 – Economic
also known as “cost analysis, cost-effectiveness evaluation, cost
benefit analysis and cost-utility analysis”
Generates quantitative data for
Sponsors and backers so as to see benefits their money
is bringing to beneficiaries.
When
At the beginning of a program, to remove potential
leakages.
During implementation of the program to find and remove
inefficiencies
What
Measures benefits of programs against costs.
Ascertain what resources are being spent and where
Understand how these costs are translating into
outcomes. 15
Evaluation Type 4 – Economic
also known as “cost analysis, cost-effectiveness evaluation, cost
benefit analysis and cost-utility analysis”
Why
Program managers can use the information to report to
funders and justify or streamline costs
Program can use the information to modify program
implementation so as to deliver better results at lower
costs.
How
Conduct systematic analysis of program by collecting
data on program costs including capital and man-hours
of work.
Could also require survey of program officers and
target population to determine potential areas of waste.
Questions to ask include.
Where is the program spending its resources ?
What are the resulting outcomes? 16
Evaluation Type 5 – Impact
Studies the entire program from beginning to end (or
whatever stage the program is at).
The focus is on long term impact, and it is useful for
measuring sustained changes due to the program,
making policy changes and/or modifying the program
When
At pre-selected intervals in the program
At the end of the program
What
Looks to quantify whether or not the program has been
successful
Assesses the change in the target population’s well-being
Accounts for what would have happened if there had
been no program
17
Evaluation Type 5 – Impact
Why
Show proof of impact by comparing beneficiaries
Provide insights to help in making policy and funding
decisions
How
A macroscopic view of the program, coupled with
extensive survey of program participants to determine
effort involved and impact achieved.
Insights and suggestions from program participants and
officers and control group from non-participants is crucial.
Questions to ask include.
What changes in program participant’s lives are
attributable to the program?
What would those not participating in the program have
missed out on? 18
Evaluation Type 6 – Summative
Useful for program managers to justify the project
and show what they achieved
Also helps to lobby for project continuation or
expansion
When
At end of the program
At end of the program cycle
What
Generates data about how well the project
delivered benefits to the target population
Helps to understand how effectively the program
made the desired change happen
Helps to understand how the program changed the
lives of program participants 19
Evaluation Type 6 – Summative
Why
Justify continuing the program
Get insights into effectiveness and efficiency of the
program
How
Conduct a review of internal reports and a survey for
program managers and target populations. The aim
should be to measure the change the project brought
about and compare the change to the costs
Questions to ask include.
Should the program continue to be funded?
Should the program be expanded? If so, where? What
factors worked in its favor and what worked against it?
20
Evaluation Type 7 – Goals-Based
also known as “objectively set evaluation”
Programs often set SMART objectives
So goals-based evaluation measures progress towards
achievement of these targets
It is useful for presenting reports to program managers and
funders as it provides information on what was agreed on at the
start of the program
When
After the end of the program
At pre-determined milestones / time intervals
What
Measures how the program has performed on initial metrics
Ascertains whether the program has achieved its goals
21
Evaluation Type 7 – Goals-Based
also known as “objectively set evaluation”
Why
To show that the program is meeting its initial benchmarks
To review the program and its progress
How
Depends on the goals agreed upon.
Usually it will involve some survey of participants to
measure impact as well as review input costs and
efficiency.
Questions to ask include?
Has the program met its goals?
Were the goals and objectives achieved due to the
program or externalities ? 22
Evaluation Criteria
OECD (Organization for Economic Co-operation and
Development)
The criteria for evaluating a program/project or
activity, should make sure it addresses the
following 5 fundamental things
1. Relevance –
• Are we doing the right thing?
• How important is the relevance or significance
of the intervention regarding local and national
requirements and priorities? .
23
Evaluation Criteria ctd
2. Effectiveness –
• Are the objectives of the development interventions
being achieved?
• How big is the effectiveness or impact of the
project compared to the objectives planned
(Comparison: result – planning)?.
3. Efficiency –
• Are the objectives being achieved economically by
the development intervention?
• How big is the efficiency or utilization ratio of the
resources used (Comparison: resources applied –
results)? 24
Evaluation Criteria ctd
4. Impact –
• Does the development intervention contribute to
reaching higher level development objectives
(preferably, overall objective)?
• What is the impact or effect of the intervention in
proportion to the overall situation of the target
group or those effected?
5. Sustainability –
• Are the positive effects or impacts sustainable?
• How is the sustainability or permanence of the
intervention and its effects to be assessed? 25
Stages In The Evaluation Process
1. Set the goals and objectives of the evaluation (what
do you want to achieve)
2. Design activities to be conducted during the
evaluation (what will you do to achieve them)
3. Agree on survey methods ( sample size, selection
of sample, data analysis)
4. Data collection - (collect the data)
5. Data analysis – (analyze the data)
6. Explanation of success level (present findings)
7. Recommendations (on what needs to improve,
maintain or policy considerations) 26
Basic Elements of an Evaluation
Report
1. Introduction
2. Objectives (main and specific)
3. Program description
4. Measurable indicators
5. Methodology
6. Results
7. Discussion
8. Conclusions and recommendations
27
