Service Evaluation Toolkit

for Health and Social Care Commissioners

Avon Primary Care Research Collaborative

2015
 Evaluation Toolkit
Introduction How to use this toolkit
Improving the quality of health and care is a priority The toolkit is set out in four steps: Assess, Plan, Do and
for the NHS. Evaluation an integral part of policy Review. Each provides you with a set of templates and
tools to support your evaluation. Used as a whole, this
making, strategic planning and commissioning. It toolkit will help you formulate your evaluation plan,
can help you make judgements about the policy or collect and analyse your data, and then use and share
service as well as understanding what works and your findings.
what needs improving. The toolkit is part of a series of documents, including a
complimentary guide and a plain English summary so
It is likely that you are reading this because you that you can evaluate with confidence.
have been tasked with considering how to evaluate
It is important to note that while the steps do not
a service or intervention or you have been asked to necessarily follow a linear process, by completing each
participate in an evaluation. step you will have addressed the main points required
to carry out a well-planned and successful evaluation.
This toolkit has been developed as a practical
resource to help you to:
• Frame the important questions about your
services
• Plan a pragmatic evaluation that will answer Tools Guided
those questions templates
• Share and use the findings from the evaluation
Top Tips Resources
This is a simplified approach, but by going through
each section, you will gather the information you
need to be able to evaluate your service effectively.
Note: Service is used through out but relates to projects, programmes, interventions, pilots etc
 What is Evaluation?
What is Evaluation? What is Service Evaluation?
There are multiple definitions for evaluation, many of which have There are multiple facets to evaluation:
common themes around judging the merit, worth, value or quality of
something. Our definition we have adopted locally was produced in • from research to a service evaluation;
collaboration with stakeholders by the West of England Evaluation • from identifying areas for improvement to providing
Strategy Group is as follows: accountability;

“A study in which research procedures are used in a • from the simple (such as a single survey) to the complex
systematic way to judge the quality or value of a service (using multiple methods);
or intervention, providing evidence that can be used to and it can be used in many different contexts. A central
improve it” (West of England Evaluation Strategy Groups, component to evaluation is one of comparison, either
2013) comparing before, with another service or with itself.
In health and care evaluation tends to be distinguished as either
research or service evaluation. The key distinguishing feature
Monitoring and Evaluation between the two is dependent on the purpose of the study:
We routinely monitor the performance of services however Research aiming to derive “generalisable” new knowledge
monitoring alone can only tell us so much – in that we are doing beyond the setting it took place; where as
things – but it unable to tell us if we are doing the right things, what is
working and what needs improving. However, monitoring data is a A service evaluation is performed to meet a specific local need
useful source of information to help you to evaluate your service. In and is focused on what is (or will form) current care with the
turn planning your evaluation alongside your monitoring framework evidence collected to inform recommendations not intended to
can help to reduce duplication and unnecessary data collection, shape be used beyond the setting it took place (although may be
and improve the monitoring data you plan to collect and identify transferable to other areas).
appropriate baseline and benchmarking data.
This toolkit is focused on helping you to plan and conduct an
appropriate evaluation.

Health Research Authorities - Defining Research Leaflet - http://www.nres.nhs.uk/applications/is-your-project-research/ provides

accepted definitions for research, service evaluation and clinical audit
Charities Evaluation Service http://www.ces-vol.org.uk/about-performance-improvement/about-monitoring-evaluation guide to
monitoring and evaluation
Top Tips
These top-tips have been developed from years of experience
of supporting those working in the NHS and Public Health
with evaluation. They are useful practical tips to support you
to start to develop your evaluation plans.
Remember evaluations do require adequate resourcing, skills
and time so it is important to ensure that your evaluation is
feasible, proportionate and appropriate to the service being
evaluated and that it will provide you with useful and valuable
information.

A useful question to ask yourself is “can you

afford not to do an evaluation”?1

E.valua.tion
Extremely
Valuable
Information2

Ref 1: HM Treasury (2011) The Magenta Book. Guidance for Evaluation

Ref 2: Carolyn Cumpsty Fowler, John Hopkins University (2011)
http://www.odh.ohio.gov/~/media/HealthyOhio/ASSETS/Files/injury
%20prevention/evaltrainingfowlerpresentation.ashx
 Assess Plan Do Review

Navigation
This toolkit provides you with a suite of resources (tips, tools, templates, signposting) to enable you conduct a service evaluation.
You can use the tools and templates as a package or in isolation if you just need support with one aspect.
The questions below will help you to navigate through the toolkit or signpost you to specific areas you would like help with.

Question Toolkit Guide Page

Resource No.
Assess • Do you need to do an evaluation? 1
• Do you have an understanding of how your service will deliver the planned 2
outcomes including evidence to support these assumptions?
• Are you confident you have all the right people involved and engaged? 3
• Have you identified appropriate resources to conduct the evaluation? 4

Plan • Have you developed your evaluation plan including identifying your: 5
• aims and objectives of the evaluation? 6
• evaluation types and approach 7
• data to be collected? baseline data? 8, 9 and 10
• ethics and governance implications? 11

Do • 13. Do you need to develop your data collection tools?

• 14. Have you analysed your data?
Review • Do you need to write an evaluation report? 12
• Who should you be sharing your findings with? 13
• Do you need to develop an action plan based on the findings and 14
recommendations?
• Am I sure I have covered everything - Checklist
 Assess Plan Do Review

Resource 1. Do I need to be doing an evaluation?

Evaluation requires an investment of time, money and resources and so before you begin you need to assess whether you
should be conducting an evaluation before making this investment. The following questions will help you to start to consider
whether you need to and can do an evaluation, we recommend that you consider these in partnership with your
stakeholders. You may not be able to answer them all straight away but useful to keep in mind when planning your
evaluation.
An Evaluability Assessment may be useful in determining whether the service is ready to be evaluated and what
type of evaluation is required.

Question Yes No ?
Do I know what I want to evaluate and why? Consider drivers and audience
Do I already know the answer to my evaluation question? Consider evidence
Will I be able to use the results of the evaluation? Consider context
Will these evaluation results provide me with useful information that will be considered “value for
money”? Do benefits of conducting the evaluation out way costs and consequences.
Do I have the resources and/or skills available to undertake this evaluation? Consider how

Will the information I collect be able to answer my evaluation question (the evaluations aims and
objectives)? Consider quality and accessibility of data
Is it the right time to conduct an evaluation of the service? Consider:
• Stage of development and complexity of the service
• Timescales of the service
• Context the service is operating in
Are there other options to doing an evaluation? Consider:
• Are current project processes such as monitoring enough?
• Should I consider other options such as clinical audit, quality improvement or research?

For more information on “Evaluability Assessment” the following resources will help:
http://betterevaluation.org/themes/evaluability_assessment
 Assess Plan Do Review

Resource 2. Understanding your service – Context

When considering your evaluation you first need to understand your service in terms of the outcomes you want the service to
achieve and for whom (what need are you trying to address?), what will deliver these changes and in what context. The following are
aspects of the service that are useful to consider when planning your evaluation.
Aspect Description Implication

Purpose What is the purpose of the service? It is important to understand the purpose of your service (i.e. what
What need is it addressing? What changes it intends to make – outcomes – and how it intends to do this)
outcomes will it achieve? and the evaluation (i.e. what answer does it need to address as this will
What is the purpose of the impact on the type of evaluation you need).
evaluation? How will it be used?
Who is your audience?

Evidence What is the evidence base for the Understanding the evidence base for your planned service can help
base planned service and associated inform your evaluation approach and methods: from understanding the
outcomes? strength of the evidence available i.e. if well evidenced then you might
What is the strength of evidence? focus on the implementation (process) evaluation; to identifying
How have similar services previously appropriate outcomes and outcome measures; to looking at how others
been evaluated? have evaluated similar schemes before.

Stage of Is it new? Has it been in place for a Understanding the stage of development of your service will also impact
development while? on the type of evaluation you need to conduct. If it is new and being
developed you will want to take a more formative (improvement)
approach.

Level of Is it a single change? Are there Understanding how complex your service is and the context in which it is
complexity multiple components? What is the operating is also important. How many services do you know trying to
and context context/environment in which it is reduce unplanned admissions?
working?

Timescales What are the timescales? Is it a one Timescales will impact on what you are able to evaluate and how, i.e. if a
year or five year pilot? long term investment you are likely to be able to not only look at
implementation (formative/process evaluation) but also outcomes.
 Assess Plan Do Review

Resource 2: Understanding your service – Theory of change

A useful process for planning your evaluation is to develop a “theory of change” for your service. This can be useful way of
articulating and providing a visual representation of the links between the various activities of service and how this will lead to the
long term outcomes it is trying to achieve.

Intermediate
Inputs Activities Outputs Impact
Outcomes

To develop your “theory of change” working with your key stakeholders, some flip chart paper, pens and post-it
notes consider the following questions1:
1. Who the service is for? i.e. your case for change will usually set out the population group, their needs and
characteristics, the problem it is trying to address
2. What are the long – term outcomes you want to achieve?
Then working backwards
3. What are the intermediate outcomes (short and medium term) that will lead to these long term outcomes?
4. What are the activities that the service will undertake to deliver these intermediate outcomes?
5. What evidence (from research and local learning) is available to inform and support the links between
activities and outcomes? i.e. what are your assumptions based on the evidence, expertise and learning?
6. What other factors need to be in place to enable this service to work? i.e. what are your enablers?

NPC guide to developing Theory of Change 1 - http://www.thinknpc.org/publications/theory-of-change/

Kellogg Logic Model Development Guide: http://

www.wkkf.org/resource-directory/resource/2006/02/wk-kellogg-foundation-logic-model-development-guide
 Assess Plan Do Review

Resource 2: Understanding your service – Accessing and reviewing the evidence base
It is important to consider what evidence is available to inform your planning and decision making in terms of the service as well as the
evaluation. You will be used to using a broad range of evidence from multiple sources including needs assessments, public health and
performance data, evidence from research and best practice as well as expertise and local learning. This evidence can be used to
inform your evaluation, help identify the outcomes you hope to achieve and the activities (processes) and outputs that will deliver
these outcomes.

Understand what evidence is already available to inform not only your service design, but the type and level of
evaluation you need.

Our evidence toolkit http://evidence.apcrc.nhs.uk can help you with accessing the best
available published evidence from research, evaluation and the grey literature.

How does this help me?

Understanding your service, the context in which it is operating, providing a visual representation of your service and understanding
the evidence and theory behind it can help you to identify what your should focus your evaluation on and the type of evaluation you
might undertake. The diagram below tries to outline how this helps.
 Assess Plan Do Review

Resource 3. Involving and engaging the right stakeholders.

It is important to consider who will be interested in this service, evaluation or its findings. Engaging and involving all the right
stakeholders right at the beginning is an important step not only in developing your service but also in ensuring a good evaluation.
Stakeholders will be any one affected by the service or service change and this will include the users, providers and commissioners of
the planned services.

Engage all key stakeholders, including the patients and their carers, in the design, delivery and dissemination of the
evaluation

Who are my stakeholders?

You may have already completed a stakeholder analysis as a part of your service planning, however if you have not completed one
already then we recommend that you conduct a stakeholder analysis. This is ideally done as a group and could use existing forums
such as the project team, steering group or advisory group. Don’t forget to involve your local and organisational experts in patient and
public involvement, and your equalities and communications leads to help you.

Once you have conducted the stakeholder analysis this can then be used to inform:
• Who to involve in the evaluation
• What expertise and people are available to support the evaluation (including expertise in data collection, data analysis,
communication, patient and public involvement)
• Your project, evaluation and communication plans

N.B. Make sure you consider the needs of those that are seldom heard and are vulnerable. If you have not already, consider
completing an equalities impact assessment for the planned service or service change.

The NHS Institutes tool, part of its quality improvement tools, http
://www.institute.nhs.uk/quality_and_service_improvement_tools/quality_and_service_improvement_tools/stakehol
der_analysis.html
willmore
For help information
you to conduct a stakeholder
about analysis.
involving patients and the public then INVOLVE www.invo.org.uk, who are funded by
the National Institute for Health Research (NIHR) to support public involvement in NHS, public health and social care
research, are a useful resource.
 Assess Plan Do Review

Resource 4. Resources
It is important to consider what resources are required for the evaluation from the outset. This may need to be reviewed throughout
the planning process. The resource requirements will be informed by the type of evaluation you need (and quite often impact on the
design of your evaluation). This is an important step and should be considered early when planning and developing your service (i.e.
business case, QIPP scheme, service specification, intervention, programme or project).

Plan your evaluation early in the commissioning cycle. This will enable you to ensure that you use the most appropriate
methodology, collect the right data, allocate adequate resources and set appropriate timescales.

Question Evaluation Resources

Who is responsible for overseeing the There will usually be an identified lead for the evaluation who can often be the
delivery of the Evaluation? project/service manager or the commissioner. You will also want to consider your reporting
(governance) arrangements for the evaluation, this may be an established project group or
committee.

Who should be involved in the evaluation? Who are your key stakeholders? Are there local experts that can help? Discuss lay
Who can help you to deliver this evaluation? involvement with your PPI lead - service users and their families are often well placed to
help evaluate the services.

What resources are available to support the This could be money and/or staff time depending on the evaluation. If you are conducting
evaluation? Include financial resources an internal evaluation then you will need input from staff. If you need to commission an
external evaluation then you will need to also identify a budget (approximately 10% of the
project value). Utilise existing data and project resources where possible and appropriate.

Who will conduct the evaluation? Evaluations can be conducted internally, externally by an independent organisation or a
mixture of the two. What you decide will depend on the purpose of your evaluation (i.e. if it
is to inform and improve then an internal evaluation may be adequate; if it is to demonstrate
a level of accountability then you need to consider an external evaluation), the level of risk
involved to the organisation and the participants and the skills and resources available.

Our service evaluation guide and website tbc provides you with guidance on commissioning an independent external
Evaluation and template to set out your invitation to tender.
 Assess Plan Do Review
Resource 5. Evaluation Planning Template
This template sets out the key elements to consider when planning your evaluation.

Don’t forget to build the evaluation into your project plans and processes as much as possible. Make sure it is feasible
and utilises appropriate existing data that is aligned to, and informs, your monitoring plans.

Title Something that explains what you will be doing.

Project team Outline who makes up the team and what their roles will be.
Background Outline the context or rationale for undertaking the evaluation. Bring in here information from your business case,
service specifications and evidence reviews i.e. what is your case for change, what outcomes do you hope the service
will achieve.
Aim and Ensure that your evaluation has a clear purpose and focus. To do this set your aims and objectives at the beginning
objectives with your key stakeholders. It is important to get them right at the outset as this will guide your evaluation design. The
overall aim will be a statement of what the evaluation will do. The objectives are narrower statements of what the
Resource 6 evaluation will try to achieve in order to meet the overall aim.

Design and There are multiple approaches to evaluation which can be complex and confusing. A simple approach to overcome this
methods is to consider the types of information and data you need to answer your evaluations aims and objectives. Consider
both qualitative (i.e. narrative data from interviews, focus groups) and quantitative (i.e. numerical data from surveys,
Resource 7 - 10 monitoring forms), the sources of that data (does it already exist or do you need to collect it). Ensure that you baseline
your information where possible and consider utilising benchmarking data.

Ethics and Ensure you have considered what the ethical implications of the evaluation are and how they will be mitigated and
Governance reviewed throughout the project lifecycle.
Resource 9
Resources Outline the resource requirements and include here internal project resources to support the evaluation as well as any
ring-fenced financial resources (and funding available). You have already started to identify these when assessing
Resource 4 whether to do an evaluation.
Outputs and This is a description of what will be produced (e.g. the evaluation report) from the evaluation as well as the intended
impact use and impact. Remember to build this into the services communication plans.
References All sources quoted in the proposal should be acknowledged and correctly referenced.

Health Foundations Evaluation: What to Consider Guide http://

www.health.org.uk/publication/evaluation-what-consider is a very useful overview of what to consider when planning
an evaluation
 Assess Plan Do Review

Resource 6: Setting Aims and Objectives

It is important to get your aims and objectives right at the outset as this will guide your evaluation design. Involving key stakeholders in
setting the aims and objectives of the evaluation helps elicit buy-in; promotes cooperation; ensures all interests are accounted for; and
that expectations are managed.
The overall aim will be a statement of what the evaluation will do. The objectives are narrower statements of what the evaluation will try
to achieve in order to meet the overall aim. The case study below summarises a real-life process of co-producing evaluation objectives
with a stakeholder group conducting a training pilot in partnership with volunteers to increase the skills and knowledge of people with
Fibromyalgia. The group met to discuss what was important for each of them and considered how the objectives linked together. Note
how some of the objectives meet the needs of more than one stakeholder.
Stakeholder What they want from the evaluation Agreed Objective

The Lay To see how the project is improving experience of living with To evaluate the impact of the training on patient knowledge and
the condition in order to influence NICE guideline review skills for enhanced self-care of fibromyalgia
Representative To measure stakeholder satisfaction with the approach

The Is the intervention cost-effective? To understand the potential of the project to effect cost-savings
through a reduction in unnecessary hospital admissions and
Commissioner Does it fit in with the CCG’s strategic priorities? unnecessary GP attendances.

The Service Are the aims of the project met? (ie in this case, do patients’
knowledge and skills to self-manage increase?)
To evaluate the impact of the training on patient knowledge and
skills for enhanced self-care of fibromyalgia .
Provider To measure stakeholder satisfaction with the approach
What do service-users think of the project?

The Voluntary Did volunteers help meet the aims of the project?
What was the experience of Volunteers in the project?
To evaluate the impact of the training on patient knowledge and
skills for enhanced self-care of fibromyalgia .
Sector Partner Do volunteers have the potential to deliver cost-savings? To measure stakeholder satisfaction with the approach
To understand the potential of the project to effect cost-savings
through a reduction in unnecessary hospital admissions and
unnecessary GP attendances.

AIM: to assess the impact of the pilot to make recommendations to commissioners about future funding
 Assess Plan Do Review

Resource 7: Types of Evaluation – which type of approach should I use?

There are a number of different types of evaluation, the most widely used are formative, Process and summative (outcome)
evaluation. The following information provides you with a guide to the types pf evaluation, when they are used and the types
of approaches and data collection methods that are associated with them. (Roberts, Cavill and Rutter 2009)

• Assesses implementation and delivery of the initiative, and whether this was carried out as planned
• Takes place throughout the project implementation and delivery phase.
• Demonstrates to stakeholders that project Objectives are being met
Process • Qualitative methods: Interviews; Focus Groups; Questionnaires
• Quantitative methods: Metrics, Benchmarking, quantifiable Performance, Financial, Service Usage data
• Service Improvement methods: PDSA Cycles; Rapid Evaluation Cycles

•A type of process evaluation that assesses how the intervention / programme can be improved as it is being
implemented
•Takes place before the project starts and throughout pilot phase to inform implementation.
•Shows what worked well and why, and identify challenges to capture lessons learned (continually improving)
Formative •Utilises evaluation theory e.g. Theory-of-Change; Realist Evaluation, Experience-Based Design, Critical Path etc.
•Helps to assess needs; clarify theory-of-change; to identify areas for shared learning and improvement
•Qualitative methods: Literature Review; Interviews; Focus Groups; Observation; Diaries; Questionnaires
•Service Improvement methods: Stakeholder Analysis; Logic Modelling; PDSA Cycles; Rapid Evaluation Cycles

• Was the project successful? Has the met its predefined aims and objectives?
• Looks at the Outcomes (i.e. measures the changes that have occurred as a result of the programme) and/or Impact (i.e.
the longer, deeper, systemic changes).
• Demonstrates to stakeholders if project aims were met.
Summative • Qualitative methods: Interviews; Focus Groups; Questionnaires
• Quantitative methods: Metrics, Benchmarking, quantifiable Performance, Financial, Service Usage data
• Can be Experimental (e.g. randomised controlled trial), Quasi-Experimental e.g. comparing before and after or use
service Improvement methods: PDSA Cycles; Rapid Evaluation Cycles

http://www.ihi.org/resources/Pages/HowtoImprove/default.aspx
http://www.civicpartnerships.org/#!quantitative--qualitative-eval-methods/c1bel
http://www.institute.nhs.uk/research_and_evaluation/general/useful_resources.html
 Assess Plan Do Review
Resource 8. Data Collection Planning Template
During the initial planning phase you would have identified the purpose of your evaluation (aims and objectives), the type of evaluation
you might need as well as starting to consider the types of data and information you need to be able to answer your aims and objectives.
Use the information you have identified so far as well as resources 9 and 10 to complete the following table, setting out your objectives
against the data you plan to collect, the sources of that data and who is responsible for collecting and analysing the data.

Identify and utilise existing data and data collection methods, where it is available and of good quality, to avoid duplication
and unnecessary data collection. Make sure you collect baseline data, consider benchmarking and utilise validated tools
where they exist. Consider using multiple (audit, quality improvement) and mixed (qualitative and quantitative) methods.

Evaluation Objectives Evaluation questions Data Sources / Tools Responsibility and

Timescale

What are your What are the questions you need to answer to What data do you need to be able to answer your Who will be doing
evaluation objectives? enable you to demonstrate whether you have specific questions / measures? what and when?
achieved your objectives? Where can you source that data from? Or What
data collection tools do you need?
What sample size will you need?
What baseline and benchmarking data is available?

Objective 1

Objective 2

Explore your evidence review to see how others have evaluated similar initiatives including whether there are any validated
tools (for example surveys, patient reported outcome measures, patient reported experience measures) available that you
can use?
 Assess Plan Do Review

Resource 9. Data Collection - Decision Tree Tool

To help you decide how to collect the information or data you need to support your evaluation the following decision tree can
help. Think about the types of information you need and then use this as the starting point to navigate the tree to identify a
suggested method for collecting the information.

This decision tree was developed in 2005 by Marsh and Glendenning so remember that with advances in technology there are also other ways
to collect the information such as online surveys, text messaging, video messaging, social media, apps and other mobile technologies.
 Assess Plan Do Review
Resource 10: Types of data and information
Having an adequate baseline for your key metrics is extremely important as a key aspect of evaluation is comparison,
whether it is comparing with it self, before or after the service change or with another service. Benchmarking is also a useful
tool to compare with other similar services or with national trends.

• This is additional
• Uses narrative or descriptive data new data collected
QUALITATIVE
rather than numbers. For example, a PRIMARY for the specific
description of the views and attitudes purpose of the
of those receiving/providing a service. evaluation.

QUANTITATIVE • This is existing data and

SECONDARY
• Gives numerical results. For example, will include information
the percentage of participants you or others already
completing a programme. collect. For example,
monitoring data, local
and national audits,
• This is data for the census data.
BASELINE DATA
period before the
service change, to SAMPLING
compare with the
period after
implementation of • It is important to consider sample size, i.e. how
the change. many people from the population you need for your
evaluation. It is important that the sample size is
feasible, practical and reasonable/proportionate for
• This is performance data on your the evaluation planned.
BENCHMARKING service that can be compared with • There are different approaches to identifying your
DATA similar data collected on other sample. These include ‘convenience sampling,
services locally and nationally. purposive sampling, snowballing sampling.
 Assess Plan Do Review

Resource 11. Am I doing an evaluation?

Once you have identified the need to do an evaluation and planned your approach it is important that you check that you are
doing a service evaluation and not research. To do this we recommend you use the Health Research Authorities tool to help you
to assess this or contact your local experts such as your research and development department.

http://www.hra-decisiontools.org.uk/research/

We suggest that you keep a record of the result as a part of your evaluation plan as this acts as an audit trail. We also
recommend that you check with your own organisation to ensure that you comply with local arrangements when it
comes to the governance and ethics of your evaluation.

Ethics and Governance

If you are doing a service evaluation then you will need to comply with your local and/or organisational ethics and governance
processes. These local and organisational processes may vary and so it is important to ensure that you obtain the right advice and
seek the right permissions. If you are unsure check with your local experts, these could be a Research and Development lead, Clinical
Governance lead, an Evaluation lead or a University Ethics department.

Engage with your local research and information governance experts

As a service evaluation you do not need any formal ethical approval from an NHS Ethics Committee, however every service evaluation
should ensure they apply best practice when it comes to ethics ensuring that adequate safeguards are put in place and that the
benefits of conducting an evaluation outweigh the risks. This will involve reviewing your evaluation during the design, delivery and
dissemination to ensure any ethical issues are identified and actions are put in place to address them. This will include ensuring
appropriate informed consent is obtained from participants, confidentiality and anonymity are maintained and the study complies
with your organisation‘s information governance and data protection policies and procedures. Particular consideration needs to made
for any participants that may be considered vulnerable such as children and people who lack capacity.
 Assess Plan Do Review
12. Reporting Template
The template below outlines the key information you need to provide within your evaluation report. Note: use this as a guide as
your own organisation may already have a reporting template which incorporates your organisations house style.
Heading What to include
Executive summary Provide a high level summary of your evaluation on a page (max 2 pages)
Introduction Outline the purpose of the evaluation, project/evaluation team (who is involved in the evaluation),
structure of the report
Background Provide the National and Local context for the service and evaluation (why it is important, case for
change, evidence base) including information from your business case and evidence review
Aims and objectives Clearly set out the aims and objectives of the evaluation and how they were developed / identified
Methods Using the information in your evaluation plan set out the overall approach to the evaluation,
specific data collection methods/sources (include who, where, when, how and how many) and
approach to data analysis. Note here any limitations of the methods.

Results Present the key results and analysis from the data collection here (you may decide to include some
results in the Appendix if you have lots of data) .
Findings Present an interpretation of your findings here, bring together different data that work together
(triangulation). Include here any limitations of the data and findings and be careful with the
language you use (i.e. you are unlikely to be able to show cause and effect)

Discussion Relate how the key findings fit with the local and national context, what the key learning is from the
evaluation.
Conclusion and Provide evidence based conclusions to make evidence informed recommendations from the
Recommendations evaluation
Appendices Include here your outputs from your worksheets plus any additional data or information to help the
reader understand your evaluation.

Charity Evaluation Service guide to report writing

http://www.ces-vol.org.uk/Resources/CharitiesEvaluationServices/Documents/Writing%20better%20evaluation%20reports%20Final.pdf
 Assess Plan Do Review
13. Action Planning template
Once you have reviewed the results of your evaluation (and taken account of limitations of these findings) you can use these to
make your recommendations for going forward based on the evidence. Ideally these would be reviewed by and agreed with your
project team and tested out with your key stakeholders. The next step is to develop an action plan for implementing these
recommendations. Your organisation may already have a standard template for this, however the following is a simple tool.

To maximise the impact of your evaluation make sure you use your findings to inform your decision making going forward
And identify areas for improvement

Recommendation Action Why Who When Risk level /

Issues

Recommendation specific action to help explanation of why Lead Timescale Use your
from the address the the action will lead to responsible for organisations
evaluation recommendation the desired change delivery risk matrix
Note any risks
or issues
associated
with this
activity

NHS Scotlands Quality Improvement Hub Guide to Action planning

http://www.qihub.scot.nhs.uk/knowledge-centre/quality-improvement-tools/action-plan.aspx
 Assess Plan Do Review

14. Share your findings (Dissemination/Communication) planning template

Working with your organisations communications experts and project team draw up a dissemination plan for your evaluation to
maximise its impact. Consider who your audience is, what the different approaches are to communicating the messages and the
different formats these could be communicated in and by. Where available, build into your project/service communications
strategy.

Use multiple formats to communicate your findings for the greatest impact and where possible utilise existing forms
and channels of communication (engage your communications experts!)

Audience Approach Format Who When Risk level / Issues

who you are trying to how you are what your Lead responsible Timescale Use your
reach e.g. Participants going to reach message will look for delivery organisations risk
in the study your like) e.g. Report, matrix
audience) e.g. leaflet, poster etc Note any risks or
Website, Staff issues associated
meeting, with this activity
newsletter

Better Evaluation blog on infographics

http://betterevaluation.org/blog/infographics_to_make_your_eval_results_go_viral

NHS Wirral’s guide to dissemination

http://info.wirral.nhs.uk/document_uploads/evidence-factsheets/19disseminatethefindings11052011.pdf
 Evaluation Planning Checklist (1 of 2) Front
Planning Checklist
Planning is a crucial part of the evaluation process. The following checklist (based on the original APCRC checklist) has been designed
to help with the process of carefully planning your evaluation, outlining the key steps going forward.
Identify Questions to ask
Study type 1. Is this a service evaluation or research?
Project Governance 2. Who will take responsibility for getting the evaluation off the ground and oversee its
delivery?
3. Who will write the evaluation plan, identify the required resources and engage
stakeholders?
4. Do you need an advisory or steering group?

Project resources 4. What resources are available to support this evaluation? What resources do you
think you might need / how much might it cost? Who might fund the evaluation?
5. What level or type of evaluation do you need?
Support for the project: All Stakeholders 9. Who are your key stakeholders?
9. Who needs to be informed?
10. Who do you need to involve in the evaluation planning, delivery and
dissemination?
10. Who has skills, experience and expertise to support you with your project? i.e.
Patient and public involvement, equalities, communications and engagement,
evaluation leads in your own or partner organisations

Support for the project: Service User 14. How will you involve service users, patients, carers and the public in your evaluation?
Involvement Consider this in terms of the design, delivery (data collection) and dissemination
(communicating your findings).
Context: Evidence Base 19. What is the evidence base for the planned service, service change, pilot?
20. How have similar services been evaluated in the past?
Context: Understanding the Service 19. Is it clear who the service is for? (population group, needs and characteristics)
20. Is it clear what the desired intermediate and long term outcomes are and how the activities
of the service or intervention will lead to these?

Scope of the project 19. Have you agreed with your stakeholders the purpose of the evaluation?
20. Are you clear what the evaluation will focus on?
21. Is it clear why you are conducting an evaluation?
 Evaluation Planning Checklist (2 of 2) Back
Identify Questions to ask
Aims and Objectives of the 23. Have you engaged your stakeholders to help you identify your evaluations aims (why you are doing
evaluation this evaluation) and objectives (what you are trying to achieve)?
24. Are your aims and objectives SMART?
Evaluation approach 25. What evaluation approach or method are you planning to take?
26. Do you need to commission an external evaluation?
Data requirements 26. What information and data do you already have available to support your evaluation?
27. What additional data collection do you need to undertake to be able to answer the aims and
objectives of your evaluation?
Data Collection, Analysis 26. What data do you need to collect? Will your data collection tools work? Are there any validated
and Reporting tools that can help? Who will collect the data?
27. How will you analyse the data? Who will analyse the data?
28. How and who will write up the findings?
29. Have you identified any training needs to support these activities?

Step 5: Evaluation Plan

Timescales, 28. What are the timescales for the evaluation and the data collection?
responsibilities and 29. Who will be responsible for each of these?
resources 30. Do you need any additional resources or funding?
Information Governance 30. Have you reviewed your organisational policies on Information Governance including data
protection, storage and use to ensure that your evaluation plan complies or sort advice from your
information governance lead?
31. Have you sought relevant permissions to undertake the service evaluation?

Ethical implications 32. Have you considered the impact of your evaluation on the participants and the service?
33. Have you put adequate safeguards in place to protect the participants in your study? Including
gaining consent and feeding back findings.
Recommendations and 34. Have you agreed your recommendations and how you are going to implement them?
action planning 35. Have you developed your action plan?
Sharing the findings 35. How are you going to feedback the findings from your evaluation to your stakeholders?
36. Have you developed a communication plan to share your findings and recommendations?
Are you still doing an 29. Once you have planned your evaluation recheck whether you are doing a service evaluation or
evaluation? research to ensure that you have the appropriate permissions and approvals for starting the project.
 Additional Resources
There are a number of resources available to support your evaluation, some are included within the toolkit to support specific aspects of
your evaluation, this section provides you with other toolkits that may be of use to you.
Evaluation Toolkits
• CLAHRC Evaluation Guide: Developed by NIHR CLAHRC Leicestershire, Northamptonshire and Rutland in 2012 for clinicians and NHS
Managers to help guide them through the process of evaluation
– http://www.clahrc-cp.nihr.ac.uk/wp-content/uploads/2012/07/Evaluation_GUIDE.pdf
• NHS Cambridge Full Evaluation Toolkit: Adapted from the PRIMARY CARE SERVICE EVALUATION TOOLKIT Version 1.5 Peter Marsh and
Robert Glendenning to support NHS Cambridge CCG with Evaluation
– http://clahrc-cp.nihr.ac.uk/wp-content/uploads/2012/07/Full_Evaluation_Toolkit.pdf
Evaluation Guides
• Magenta Book: Developed by HM Treasury to support evaluation of policy
– http://www.hm-treasury.gov.uk/d/magenta_book_combined.pdf
• MRC Framework: Developed by the MRC to help with evaluating complex health interventions
– http://www.mrc.ac.uk/Utilities/Documentrecord/index.htm?d=MRC004871
• Health Foundations Evaluation: What to Consider Guide provides insight into the things you need to consider when planning an
evaluation, the different types of evaluation and different methodological approaches
– http://www.health.org.uk/publication/evaluation-what-consider
Other useful resources
• Charities Evaluation Service Evaluation Tools and Resources
– http://www.ces-vol.org.uk/tools-and-resources
• NPC Evaluation Tools and Resources
– http://www.thinknpc.org/
• Social Value UK (SROI network) guide to Social Return on Investment
– http://socialvalueuk.org/what-is-sroi/the-sroi-guide