betterevaluation.
org
May 2013
Synthesize data from one or more evaluations
Combine data to form an overall assessment of the merit or worth of the intervention, or to summarize
evidence across several evaluations.
1. Synthesize data from a single evaluation
How will you synthesize data from a single evaluation?
Processes Multi-Criteria Analysis: a systematic process to address
multiple criteria and perspectives.
Consensus Conference: a process where a selected
group of lay people (non-experts) representing the Numeric Weighting: developing numeric scales to rate
community are briefed, consider the evidence and performance against each evaluation criterion and then
prepare a joint finding and recommendation. add them up for a total score.
Expert Panel: a process where a selected group of Qualitative Weight and Sum: using qualitative ratings
experts consider the evidence and prepare a joint (such as symbols) to identify performance in terms of
finding. essential, important and unimportant criteria
Techniques Rubrics: using a descriptive scale for rating
performance that incorporates performance across a
Cost Benefit Analysis: compares costs to benefits, both
number of criteria.
expressed in monetary units.
Value for Money: a term used in different ways,
Cost-Effectiveness Analysis: compares costs to the
including as a synonym for cost-effectiveness, and
outcomes expressed in terms of a standardized unit (eg
as systematic approach to considering these issues
additional years of schooling).
throughout planning and implementation, not only in
Cost Utility Analysis: a particular type of cost- evaluation.
effectiveness analysis that expresses benefits in terms
of a standard unit such as Quality Adjusted Life Years.
Approaches
Social Return on Investment: a systematic way of incorporating social, environmental, economic and other values
into decision-making processes
Examples of overall judgement as success or failure:
Achieved Achieved Significant
Overall
all intended some intended negative
performance
outcomes outcomes outcomes
Scenario 1 ✓ - SUCCESS
Scenario 2 ✓ FAIL
Scenario 3 ✓ ?
Scenario 4 ✓ - ✓ ?
You may use this document under the terms of the Creative Commons Attribution-Non Commercial Unported licence available at
[Link]
2. Synthesize data across evaluations
Do you need to synthesize data across evaluations? If so, how should this be done?
Best evidence synthesis: a synthesis that, like a realist Realist synthesis: synthesizing all relevant existing
synthesis, draws on a wide range of evidence (including research in order to make evidence-based policy
single case studies) and explores the impact of context, recommendations.
and also builds in an iterative, participatory approach to
Systematic review: a synthesis that takes a systematic
building and using a knowledge base.
approach to searching, assessing, extracting and
Meta-analysis: a statistical method for combining synthesizing evidence from multiple studies. Meta-
numeric evidence from experimental (and sometimes analysis, meta-ethnography and realist synthesis are
quasi-experimental studies) to produce a weighted different types of systematic review.
average effect size.
Vote counting: comparing the number of positive
Meta-ethnography: a method for combining data studies (studies showing benefit) with the number of
from qualitative evaluation and research, especially negative studies (studies showing harm).
ethnographic data, by translating concepts and
metaphors across studies.
Rapid evidence assessment: a process that is faster
and less rigorous than a full systematic review but more
rigorous than ad hoc searching, it uses a combination
of key informant interviews and targeted literature
searches to produce a report in a few days or a few
weeks.
3. Generalize findings
How can the findings from this evaluation be generalized to the future, to other sites and to other programs?
Analytical generalisation: making projections about Statistical generalisation: statistically calculating the
the likely transferability of findings from an evaluation, likely parameters of a population using data from a
based on a theoretical analysis of the factors producing random sample of that population.
outcomes and the effect of context. Realist evaluation
can be particularly important for this.
Approaches
Positive Deviance: Involves intended evaluation users in identifying ‘outliers’ – those with exceptionally good
outcomes - and understanding how they have achieved these.
Horizontal Evaluation: An approach that combines self-assessment by local participants and external review by
peers
Find options (methods), resources and more information on these tasks and approaches online at
[Link]
BetterEvaluation - Synthesize data from one or more evaluations (May 2013)
[Link]
