BOOK REVIEWS

Title: Evidence-based Policy: A Realist Perspective

Author: Ray Pawson
Publisher: Sage, London
Publication date: 2006
Extent/type:196 pages, paperback
Price: A$66 from Footprint Books which offers a 15% discount to AES members, phone (02) 9997 3973
ISBN: 1-4129-1060-9

This book is an excellent ■ purposefully drawing However, in a book that contests

resource for evaluators who from ostensibly disparate the hegemony of meta-analytic
wish to inform policy decisions literature to identify systematic reviews, I would have
with collective insights underlying mechanisms and liked more acknowledgement
gained from diverse threads theories that could impact of the rich literature on diverse
of relevant literature. In the upon the implementation forms of research synthesis
current international scenario, and outcomes of a given methods. It would have been
systematic reviews with a intervention valuable to see Pawson’s views
priori protocols are attracting on how his method draws
■ systematic involvement of key
large funding (Suri 2004). on, and differs from, various
stakeholders throughout the
Often, reductionist systematic formally proposed qualitative
synthesis
reviews oversimplify the role of research synthesis methods.
evidence in policy formulation ■ regarding a fragment of Helpful descriptions of these
by assuming that policy evidence or a program theory, diverse research synthesis
decisions should be informed by rather than a study, as a unit methods are available in the
aggregated relationships between of analysis published literature (e.g. Dixon-
individual contextual variables Woods, Agarwal, Jones, Young
■ evaluating any fragment of
and outcome variables. This & Sutton 2005; Mays, Pope
evidence in the context of
book is timely in its emphasis & Popay 2005). Even though
how it contributes towards
on the complex, multifaceted Pawson is inclusive of qualitative
constructing a synthetic
nexus between evidence and and quantitative research in
understanding rather than
policy. Contesting the notion general, he has shied away from
employing a priori hierarchy
of social interventions as clarifying the role of postmodern
of evidence
‘treatments’ aimed at ‘subjects’, and emancipatory research
Pawson emphasises that social ■ learning from, rather than reports. Given that openly
interventions are ‘complex controlling for, ‘diversity, ideological research is gaining
systems thrust amidst complex change, idiosyncrasy, popularity in many circles of
systems’ (p. 168). Recognising adaptation, cross- social sciences, this is a serious
that most policy decisions are contamination and program omission. At times I found an
made in politically charged “failure”’ (p. 178) excessive use of metaphors and
environments that are constantly analogies distracting, and this
changing, Pawson makes ■ tentative findings that tell also made some sections difficult
sensible suggestions on how decision-makers of likely to read.
to realistically inform policy implications of different Despite these caveats,
decisions with the relevant decisions in different Pawson’s realist synthesis offers
evidence by deliberately straying situations rather than telling useful guidelines to synthesise
‘outside the policy domain in what works evidence for:
question’ (p. 173). ■ acknowledging inherent ■ examining the integrity of a
Salient features of Pawson’s selectivity in any synthesis program theory
realist synthesis include: process.
■ adjudicating between rival
■ focusing on individual Evaluators with varied program theories
‘mechanisms, contexts and levels of expertise in research
outcomes’ to investigate synthesis methods can draw on ■ investigating the suitability of
‘for whom, in what this book. Experienced research the same theory in different
circumstances, and in synthesists can find some useful settings
what respects a family of strategies for improving their
■ comparing the official version
programmes work’ (p. 2) research syntheses. Novices
of a theory with its actual
■ learning from successful would find this book a good
implementation in practice.
as well as unsuccessful introduction to meta-analytic
systematic reviews and ways in I would urge prospective
implementations of a readers to skim through this
program which realist syntheses move
beyond systematic reviews. book’s last chapter published

56 Evaluation Journal of Australasia, Vol. 7, No. 1, 2007

 BOOK REVIEWS

at <www.leeds.ac.uk/ Mays, N, Pope, C & Popay, J

realistsynthesis> to get a flavour 2005, ‘Systematically reviewing Reviewed by:
of the book. This companion qualitative and quantitative
evidence to inform management
site also includes useful articles and policy making in the health Harsh Suri
that augment the main thesis field’, Journal of Health Services
presented in the book. On Research and Policy, vol. 10, no.
Centre for the Advancement of
the whole, the book and its 1, pp. 6-20. Learning and Teaching,
companion website are valuable Suri, H 2004, ‘Synthesising research Monash University, Melbourne.
methodological resources. for diverse purposes: moving Email:
beyond “What works?”’, <[email protected]>
References paper presented at the annual
conference of the Australian
Dixon-Woods, M, Agarwal, S, Jones, Association for Research in
D, Young, B & Sutton, AJ 2005, Education (AARE), Melbourne,
‘Synthesising qualitative and 28 November – 2 December,
quantitative evidence: a review <www.aare.edu.au/04pap/
of possible methods’, Journal of sur04790.pdf>.
Health Services Research and
Policy, vol. 10, no. 1, pp. 45–53.

Title: The Sage Handbook of Evaluation: Policies, Programs and Practices

Editors: Ian Shaw, Jennifer Greene and Melvin Mark
Publisher: Sage, London
Publication date: 2006
Extent/type: 608 pages, hardback
Price: A$241 from Footprint Books which offers a 15% discount for AES members, phone (02) 9997 3973
ISBN: 0-7619-7305-2

I remember feeling very confused I am not sure why it is called point, and exploring them
when I first went to evaluation a handbook. After carrying philosophically and empirically.
conferences. So many different around this 600-page tome for a The focus is on the evaluation
fields of interests, so many points couple of days, I can assure you of programs, policies and
of view, so many seemingly that it is too heavy, both literally practices in ‘people-related
unrelated methods, such different and metaphorically, to keep in areas’, with the contributors
expectations and experiences. your handbag or briefcase ready largely drawn from the
After a couple of years, I realised to haul out when an evaluation domains of education, health,
that it was never all going to gets a bit tricky and you need social policy and international
fall neatly into place, that this some guidance. The title is development. While the
was the reality of contemporary misleading in another sense too; Handbook does not address
evaluation, and I began to enjoy it’s a ‘mind book’ rather than a other fields such as, in the
the complexity and diversity, ‘handbook’. It offers 26 essays words of the editors, ‘energy
and found myself reassuring that demonstrate different ways policies, space exploration
newcomers. As an evaluation of thinking about evaluation. programs, defence policy, or
practitioner, I found that this For newcomers looking for animal husbandry practices’,
book takes my dilemma as its some simple approaches to much of the discussion is
starting point. The introductory doing evaluation, it may be equally applicable to rapidly
essay by the editors spends many disappointing. For those of growing domains of evaluation
pages discussing the themes and us with more experience, practice such as natural resource
boundaries of evaluation in the the Handbook is likely to be management.
21st century, and the subsequent confirming and enlightening, The Handbook’s stated
chapters explore this in different taking the dilemmas and aim is to produce a picture of
directions in detail and with confusions that we experience evaluation that is definitive,
enthusiasm. with evaluation as its starting diverse, comprehensive and

Book reviews 57