Policy Design and Bounded Rationality: Understanding Herbert A.

Simon’s Legacy

Introduction

Herbert A. Simon was a towering figure in the development of decision theory, public
administration, and artificial intelligence, whose ideas on bounded rationality fundamentally
reshaped the way we think about human decision-making in public policy contexts. Simon
challenged the assumptions of the "rational actor" model that had dominated economics and
political science for decades, offering instead a deeply empirical, psychologically grounded, and
behaviorally realistic model of how individuals and organizations make choices. His concept of
policy design was inextricably linked to his theory of bounded rationality, which posited that
real-world decision-makers operate under constraints of cognitive limitations, incomplete
information, and emotional pressures.

Bryan D. Jones, in his extensive essay on Simon's contribution to policy sciences,

reemphasizes Simon’s theoretical revolution and the implications it continues to have for public
policy research and practice. This essay synthesizes those arguments, breaking down key
concepts such as bounded rationality, thin and thick rationality, and the Fernandes–Simon
approach to illustrate Simon’s enduring relevance in the field of policy design.

I. Herbert Simon and the Foundations of Policy Design

Simon’s path-breaking work began with the 1947 publication of Administrative Behavior, where
he laid the foundational critique of comprehensive rationality. His key argument was that human
beings are not fully rational calculators but are intendedly rational, seeking to make good
decisions but doing so under constraints. This recognition of human limitations led to the
concept of bounded rationality, which became central to his model of decision-making.

Jones (2002) points out that Simon’s approach aimed at three goals in public policy modeling:
(1) it must not mislead (i.e., it should reflect real-world behavior), (2) it should bridge individual
and organizational levels of decision-making seamlessly, and (3) it must be efficient by
excluding unnecessary cognitive detail. Simon’s work with James March, Allen Newell, and
others produced a comprehensive behavioral model of choice by the late 1950s—one that
Jones believes still outperforms the rational actor model in explaining policy outcomes.

Simon's approach to policy design was not limited to abstract theorization; it included a
practical emphasis on how institutions, rules, and organizational structures shape and limit
human decisions. Rather than assuming that policymakers maximize utility, Simon showed that
they often satisfice—they search for options that are “good enough,” given the constraints they
face.
II. Bounded Rationality: The Cornerstone of Simon’s Model
Simon’s theory of bounded rationality comprises four core principles that distinguish it from the
idealized rational choice model:

1. Principle of Intended Rationality

Human beings are goal-oriented and try to behave rationally, but their efforts are constrained by
cognitive and environmental limitations. Simon adopted what Jones refers to as the
Pareto–Simon Principle, noting that rationality “does not determine behavior” but is bounded
by irrational and non-rational elements such as emotion, habit, and memory.

As Jones explains, this principle challenges the view that decision-making failure is due to lack
of computational capacity. Simon, instead, emphasizes the importance of attention, memory,
and emotional commitment in guiding behavior. Decision-makers often operate under immense
uncertainty, and they rely on heuristics and mental shortcuts to process information and arrive at
decisions.

2. Principle of Adaptation

Humans adapt their thinking based on the “task environment.” Given time and motivation, their
mental models approximate the structure of problems. This principle helps explain why
organizations, too, develop standard operating procedures—they are cognitive aids designed to
help actors cope with complexity.

3. Principle of Uncertainty

Unlike the rational actor model that reduces uncertainty to probabilities, Simon highlighted
fundamental uncertainty—where actors may not even know what outcomes are possible, let
alone their likelihood. This kind of uncertainty influences how problems are framed and how
solutions are sought.

4. Principle of Trade-offs

Simon’s concept of satisficing arises from the difficulty of making trade-offs between competing
goals. People don’t always compare and maximize across a utility curve. Instead, they set
aspiration levels, and when those are met, they stop searching. This behavior is psychologically
more realistic than the optimization required in rational actor models.

In policy design, these principles underscore the importance of considering how actual
humans—not theoretical actors—respond to institutional incentives and constraints.
Policymakers face limited time, information, and cognitive bandwidth, which deeply affects how
policies are crafted, implemented, and evaluated.
III. Behavioral Theory of Choice and Policy Outcomes
Jones elaborates on the behavioral theory of choice, which Simon developed in collaboration
with Newell and March. It includes mechanisms such as:

●​ Long-term memory encoding​

●​ Short-term attention filtering​

●​ Emotion-driven prioritization​

●​ Prepared vs. searched solutions​

●​ Identification with means over ends​

These mechanisms not only guide individual decision-making but also explain how
organizations encode behavior into rules, routines, and norms. For example, bureaucracies
develop decision procedures that simplify complex choices and enable delegation. However,
they also resist change and can become entrenched, reacting episodically to signals from the
environment—a pattern that explains punctuated equilibrium in policy processes.

IV. Thick and Thin Rationality: Critiquing the Rational Actor Model
Simon’s criticism of the rational actor model is further sharpened in Jones’ discussion of thick
and thin rationality:

Thick Rationality

This assumes that individuals pursue self-interest in a consistent, utility-maximizing way. It

allows for prediction but fails empirically. Experimental findings, as cited by Jones (e.g., Frolich
and Oppenheimer), show that people often act in altruistic or norm-driven ways, violating thick
rationality assumptions.

Thin Rationality

Here, rationality is divorced from content: it means acting based on reasons, regardless of what
those reasons are. As Jones notes, this version is “not capable of scientific predictions without
empirical study of the formation of reasons.” It becomes tautological—any action can be
rationalized post hoc.

Bounded rationality, by contrast, offers a middle path. It acknowledges that human beings act
purposefully but within cognitive limits. It makes empirically testable predictions, not by
assuming optimal behavior but by analyzing how people actually solve problems under
pressure.
V. The Fernandes–Simon Approach: Problem Solving and Cognitive
Identification
A particularly compelling application of Simon’s ideas to policy design is found in his later work
with Ronald Fernandes. Their 1999 study applied process-tracing techniques, originally
developed for laboratory experiments, to real-world policy problem-solving.

Fernandes and Simon examined how professionals solve complex, ill-structured

problems—the kind commonly encountered in policymaking. One striking finding was that
many participants used a “KNOW → RECOMMEND” heuristic. This means they relied on
pre-formed solution sets rather than engaging in fresh analysis. This phenomenon was linked to
what Simon called “identification with means”—an emotional and cognitive attachment to
preferred strategies or ideologies, regardless of their relevance.

As Jones interprets it, this insight has deep implications for public administration. It means that
policy inertia and rigidity often stem not from institutional rules but from the way individuals
cling to known strategies. The Fernandes–Simon study thus provides a window into how policy
design can be improved: by creating environments that encourage reflection, disconfirmation,
and adaptive learning.

VI. Implications for Policy Design and Institutional Reform

Simon’s model demands a rethinking of how we approach policy formulation, implementation,
and evaluation. Key implications include:

1.​ Avoid Overreliance on Economic Rationality​

Policy models that assume optimization will misfire in complex settings. For example,
projecting budgets based on past trends (as in Wildavsky’s incrementalism) can fail
when organizations abruptly react to ignored signals, as shown by True, Jones, and
Baumgartner.​

2.​ Design Institutions for Adaptability​

Organizations should be structured to allow parallel processing for routine decisions
and serial, central processing for novel challenges. This means empowering frontline
workers, enabling feedback loops, and building redundancy.​

3.​ Incorporate Emotional and Cognitive Factors​

Policies succeed not just on technical merit but on how they align with stakeholders’
identities, emotions, and moral commitments. Simon’s notion of “identification with
the means” reminds us to assess how professional, ideological, or bureaucratic cultures
affect policy uptake.​

4.​ Promote Experimentation and Learning​

Simon’s advocacy for a behavioral, empirical approach to decision-making means policy
 design should include pilot programs, feedback mechanisms, and adaptive
adjustments.​

Conclusion
Herbert Simon’s contribution to policy design, as reinterpreted by Bryan Jones, remains
profoundly relevant. By grounding public policy in a realistic model of human cognition,
Simon dismantled the rational actor paradigm and offered an alternative rooted in empirical
observation and behavioral science. The concepts of bounded rationality, satisficing,
selective attention, and emotional identification provide a rich framework for understanding
how individuals and institutions actually make decisions.

The Fernandes–Simon approach extends this legacy by showing how entrenched cognitive
patterns affect problem-solving in policy contexts. Together with Jones’ elaboration, Simon’s
ideas offer not just a critique of flawed models but a constructive vision for improving public
policy through deeper insight into human decision-making.

As contemporary challenges grow in complexity—from climate change to AI

governance—Simon’s behavioral foundation becomes all the more essential for designing
policies that are not just ideal but implementable.
Research Methods for Policy Evaluation: Comprehensive
Analysis

Introduction

Policy evaluations aim to determine whether and how an intervention works by answering
questions about a program’s implementation and its outcomes. Broadly, evaluations split into
two main types: process evaluation (also called implementation evaluation) and impact
evaluation. Process evaluation examines how a policy or program is delivered on the ground,
while impact evaluation assesses the program’s effects on desired outcomes, usually by
estimating the counterfactual (what would have happened in the absence of the program).
Often both types are needed, as they provide complementary insights. For example, a process
study might reveal why a program was implemented unevenly, while an impact study measures
its causal effect. In practice, choosing an evaluation design depends on the evaluation
questions, program scale, and practical constraints. Below, we explain each major approach in
detail – process evaluation, various impact evaluation designs (randomized trials, matched
comparisons, before-after studies, difference-in-differences), and cost-benefit analysis –
highlighting key principles, strengths and weaknesses, and real-world applications in health,
education, labor and other policy fields.

Process Evaluation

Process evaluation focuses on how a policy or program is implemented, verifying “what the
program is and whether or not it is delivered as intended to the targeted recipients”. It typically
operates independently of day-to-day service delivery (often by external evaluators) and
documents the program’s operations in practice. This type of evaluation asks questions such as:

●​ Reach and awareness: Are eligible people aware of the program? How did they hear
about it, and do they understand its key components?​

●​ Uptake and coverage: Do all eligible individuals receive the program? Who participates
and why, and conversely, who does not participate and why?​

●​ Fidelity and quality: Is the service being delivered as intended according to program
design? Are all components of the program provided adequately and consistently, and
are standards (e.g. quality or legal standards) being met?​

●​ Delivery variation: Are there differences in how the program is administered across
locations or sub-groups (for example, different regions or providers)? If so, what models
seem more effective, and what changes occurred since implementation compared to the
status quo ante?​
Process evaluations use a mix of methods to gather information: analyzing administrative data
and monitoring reports, conducting social research such as surveys, interviews, focus groups,
case studies, observations, and reviewing documents. This can involve, for instance,
large-scale surveys of participants and non-participants to gauge satisfaction and awareness, or
in-depth interviews with staff to understand operational challenges. While adding such research
increases cost, it is justified when existing monitoring data are incomplete or insufficient (for
example, administrative records might track enrollment numbers but not explain why some
eligible people didn’t enroll).

When to use: In some cases, a process evaluation alone may suffice – especially if measuring
impact is not yet feasible. Scenarios include a new pilot program that will evolve before a full
impact study (here process evaluation plays a formative role to improve the model); an
established program that is underperforming or facing management issues, where the priority is
to diagnose delivery problems; a program believed to be effective (or whose outcomes are
already known through prior research) but where implementation is the main concern; or
situations where an impact evaluation is desired but not possible due to a small scale, minimal
expected effects, or insufficient time for outcomes to materialize. In such contexts,
understanding the process can be more practical than attempting an underpowered impact
study.

More commonly, process evaluation is paired with impact evaluation. When done alongside
an outcome study, process data provide vital context to interpret the results. For example, an
impact evaluation might find that overall a job-training program increased employment by a
certain percentage, but process findings can explain variations — perhaps certain regions had
better staff training or outreach, correlating with higher impacts. Process insights can also
identify if the program was implemented with fidelity; if not, a modest impact might be attributed
to implementation failures rather than the policy idea itself. In sum, process evaluation helps
policymakers understand how and why a program worked or failed, informing program
improvements and future scaling.

Strengths of Process Evaluation: It provides detailed insight into program operations and
participant experiences, which is crucial for improving program design and management. It
can uncover implementation barriers (e.g. low staff capacity, poor outreach) and best practices.
Process studies are generally free of the ethical issues that plague experimental designs since
they do not withhold services – they observe and analyze actual delivery. They are also flexible,
using qualitative and quantitative data to answer a wide range of “how” and “why” questions.
Because of this, process evaluation often directly informs policy adjustments and helps ensure
that outcome evaluations are interpreting results in the right context.

Weaknesses of Process Evaluation: By design, process evaluation does not measure the
program’s causal impact on outcomes. It cannot tell whether the program caused change; it
can only indicate whether the program was implemented as intended and how participants
reacted. Thus, process data alone might show high satisfaction and enrollment, but we still
would not know if the program actually solved the problem it targets. Additionally, process
evaluations can be time-consuming and potentially costly, as they often involve extensive data
collection (e.g. nationwide surveys or many field interviews). There’s also a risk of focusing too
much on implementation details and losing sight of outcomes – hence why process and impact
evaluations together give a fuller picture. Finally, process findings may be somewhat subjective
(especially qualitative insights) and need careful interpretation to translate into actionable
program changes.

Real-world application: Virtually every field uses process evaluations. In public health, for
example, when rolling out a new vaccination campaign, a process evaluation might track
whether clinics received vaccines on schedule, if healthcare workers followed proper protocols,
and reasons some people didn’t show up for their shots. In education, if a new curriculum is
introduced, a process study might observe classrooms to see if teachers are using the new
materials and identify training needs. In the labor policy context, consider the rollout of a job
training program: a process evaluation would check if the local employment offices delivered the
training as designed, how many eligible jobseekers actually participated, and any operational
issues (such as delays in funding or participant drop-out reasons). Such information is
invaluable – for instance, the UK’s early pilot of the New Deal for Lone Parents (NDLP) not
only measured outcomes but also evaluated implementation, finding issues like outreach
methods and staff guidance that needed refinement before national expansion. Overall, process
evaluations ensure that policymakers know what was delivered and can improve program
delivery, which is a prerequisite for achieving desired outcomes.

Impact Evaluation and Causal Designs

Impact evaluation is about determining the causal effect of a policy – did the intervention
actually produce better outcomes than would have occurred otherwise? This requires estimating
the counterfactual: what outcomes would the target population have experienced in the absence
of the program. Because we can’t directly observe the counterfactual, impact evaluations use
various designs to approximate it by creating or identifying a control group that did not receive
the intervention. The difference in outcomes between the intervention group (those exposed to
the policy) and the control group (those not exposed) serves as an estimate of the program’s
impact. Below we describe the major impact evaluation designs:

Randomized Controlled Trials (RCTs)

A Randomized Controlled Trial (RCT), or randomized impact evaluation, is often considered

the gold standard for causal evaluation. In an RCT, eligible units (individuals, communities,
schools, etc.) are assigned by chance to either the intervention group or a control group. The
control group is denied the program (or receives the standard services as if the new program
didn’t exist). Because the assignment is random, on average the two groups are statistically
equivalent in both observed and unobserved characteristics before the intervention. Thus, apart
from random fluctuations, the only systematic difference between them is the presence or
absence of the program. This means any difference in outcomes after the intervention can be
attributed to the program itself, providing an internally valid impact estimate. In other words,
RCTs use randomization to limit bias and ensure the counterfactual is credible.
In practice, an RCT can be implemented in different ways. Often it involves individual
randomization – for example, in a job training program, a lottery might determine which eligible
applicants receive the training and which do not. In some cases, cluster or area
randomization is used: for instance, randomly selecting certain communities or schools to roll
out a new policy while others serve as controls. (The choice depends on practical
considerations and to avoid contamination; if individuals within one community were split, those
denied might still be indirectly affected, so instead the entire community is treated or not.)

Strengths: RCTs, when properly conducted, yield the most credible impact estimates. The
random assignment ensures no systematic bias – differences like motivation, ability, or context
are equally distributed, so they cancel out between groups. This eliminates selection bias and
confounding factors that plague non-experimental studies. An RCT doesn’t even require
baseline (pre-program) data to ensure group comparability, since randomization guarantees
equivalence in expectation. They are straightforward to analyze (often a simple difference in
means between groups is an unbiased impact estimator). Because of this rigor, RCT findings
are highly trusted by policymakers and researchers; indeed, results from well-done trials have a
credibility that other designs usually cannot match. RCTs are widely used in medicine (clinical
trials) and have increasingly been used in social policy to test innovations. For example,
education programs have been evaluated via RCT by randomly assigning some schools or
students to receive an intervention (like a new tutoring program) and others not, to measure test
score impacts. In the labor domain, the UK’s Intensive Gateway “Trailblazer” pilot was
evaluated as an RCT by using applicants’ National Insurance Number digits to randomly assign
them to the new intensive job assistance versus normal services. Likewise, many developing
countries have tested antipoverty programs with RCTs – a famous example is Mexico’s
PROGRESA conditional cash transfer program which randomly phased in benefits to some
poor villages first and not others, allowing for a rigorous impact assessment on outcomes like
school attendance and child health.

Weaknesses: Despite being ideal in theory, RCTs face practical, ethical, and logistical
challenges. It is not always feasible or ethical to randomize access to a program. For instance,
if a program is believed to be highly beneficial or is mandated for all, denying it to a control
group raises ethical concerns (particularly for voluntary programs where advertising a service
but then randomly denying some applicants is considered unfair). Practically, running an
experiment often requires significant planning, coordination, and sometimes running parallel
systems: within one area, you may have to manage two groups separately (one getting the new
program, one not). This can strain staff and resources. There are also sample size
considerations – to detect meaningful effects, RCTs need enough units; randomizing at the
individual level is statistically efficient, but randomizing at the area level (e.g. different regions)
might require a large number of areas to have adequate power, which is rarely possible in policy
settings. Moreover, RCTs often happen in controlled pilot conditions and results may lack
generalizability if the full-scale rollout or other contexts differ (“Will it work elsewhere or at
scale?” is a common question). Additionally, issues like non-compliance (some in the treatment
group don’t actually get the treatment, or control units somehow receive it) or attrition (people
dropping out of the study) can bias results if not handled carefully. An RCT’s estimate is
internally valid but could be biased if those implementation issues occur. Finally, political and
timing constraints sometimes make randomization impossible – e.g. a program rolled out
nationwide due to urgency can’t randomly withhold benefits. In summary, while no other design
is more powerful in eliminating bias, RCTs are not always usable, and one must weigh ethical
obligations and practicalities.

Use cases: RCTs are ideal when a program is in pilot stage or resources are limited so that
random allocation is inherently fair (e.g. a scholarship program with more applicants than slots
might use a lottery). In health policy, many interventions (like new care models or prevention
programs) have been tested via community trials. In education, RCTs evaluate things like
teacher training or scholarship impacts by random assignment. In labor and welfare, there
have been prominent trials: for example, the U.S. evaluated a job training program via a large
RCT in the 1980s, and the UK tested new welfare-to-work initiatives with randomized pilot
groups. When well executed, these studies provided clear evidence of what works, guiding
policy expansion or redesign. Increasingly, policymakers embrace RCTs for evidence-based
decisions, but always with careful design to address ethical concerns (such as providing the
control group with some alternative or eventually phasing them into the program).

Matched Comparison Designs

When randomization is not feasible, evaluators often turn to quasi-experimental designs to

create a comparison group. A key approach is the matched comparison design, where
participants (those who got the program) are compared to non-participants that did not get the
program but are selected to be as similar as possible to the participants. The goal is to mimic
a control group by matching on characteristics so that the two groups differ only in program
participation. There are two common variants: (1) matched area comparisons, which operate
at the area or community level, and (2) matched individual (group) comparisons, which
operate at the person (or firm/school) level.

Matched Area Comparison: In a matched area design, the program is implemented in a limited
set of pilot areas (say, a handful of regions or districts), and those areas are then paired with
comparable areas that did not get the program. The matching of areas is based on
characteristics like demographic and labor market profiles, so that each pilot area has a
counterpart control area that is similar in context. Then outcomes (e.g. employment rates, or
whatever the program aims to affect) are measured after the program period in both pilot and
control areas. Any difference in outcomes between the pilot and matched control areas is
attributed to the program, after adjusting for any remaining observable differences between the
areas. For example, suppose a new employment initiative is trialed in 10 counties; in a matched
area evaluation, one would select another 10 counties with similar unemployment trends and
industries (that did not implement the initiative) to serve as a comparison. After one year, if pilot
counties saw, say, a 5% higher job placement rate than the controls (controlling for small initial
differences), this difference would be taken as the program’s effect.

Strengths: Matched area comparisons are often easier and more politically acceptable to
implement than individual randomization. All eligible people in the pilot areas can receive the
program (no one is deliberately denied within those areas), so there is less ethical concern
locally. It avoids running dual systems in one location – each area either fully has the program or
doesn’t – which simplifies administration (only one system per area). This design was used, for
instance, in early UK welfare-to-work pilots: the New Deal for Lone Parents (NDLP) prototype
was launched in 8 areas and evaluated by comparing outcomes in those areas to 6 carefully
matched areas without NDLP. Because no individuals were excluded in pilot areas, and other
areas were untouched, this was seen as more acceptable than an RCT where neighbors might
be treated differently. In short, matched areas allow evaluation when political or practical
realities preclude denying services within communities.

Weaknesses: The biggest challenge is ensuring the areas are truly comparable. Interpretation
of results can be very difficult because any outcome difference could be due not only to the
program but also to pre-existing area differences or divergent trends. Even with careful
matching on observable factors (unemployment rate, demographics, etc.), there may be
unobserved differences between areas – for example, one area’s economy might have an
upswing due to unrelated factors. Residual differences are almost inevitable. Analysts must
statistically control for observable differences between pilot and control areas when estimating
impacts, but with only a small number of areas this control is crude and may not fully account for
all confounders. Another issue is that areas might change differently over time. Even if they
looked the same at the start, by the end of the pilot period some local shock (a factory closure, a
policy change, etc.) might hit one region and not the other, biasing the comparison. In other
words, without randomization, we can’t be certain that pilot vs. control differences are due to the
program rather than other forces. This is especially problematic if the program’s true effect is
modest; a small impact can be easily masked or mimicked by natural variation or unobserved
differences. For example, in the NDLP prototype evaluation, the outcome of interest (lone
parents leaving welfare for work) was only about 2 percentage points higher in the prototype
areas than in control areas – a difference so small that it was hard to be confident it wasn’t due
to underlying area differences rather than NDLP itself. Thus, matched area designs have lower
internal validity than RCTs; the impact estimates may be biased if matching is imperfect. They
also typically require assuming that any remaining differences can be adjusted for or are
negligible, an assumption that is hard to verify.

Matched Individual (Comparison Group) Design: This design attempts to simulate an RCT
by selecting a comparison group of individuals ex post. Here, the program is often already
implemented (perhaps nationwide or broadly available). We then take the eligible population
and divide it into those who chose to participate and those who did not participate. From
each group, we draw a sample such that each participant is “matched” with one or more
non-participants who have similar characteristics. The matching criteria include factors relevant
to program selection and outcomes (for example, in a job training program, you might match
participants and non-participants on age, education, prior employment history, etc. – anything
that might affect their likelihood of enrolling and their employment outcomes). If done well, the
matched non-participant is virtually identical to the participant in all key respects except
actually receiving the intervention. Both groups are then followed over time, and the difference
in outcomes between participants and their matched counterparts is attributed to the program.
Essentially, each matched pair (or matched set) serves as a mini comparison, and averaging
across them gives the estimated impact.
Strengths: Matched comparison group designs are very useful when a program has already
been rolled out broadly (or universally) and you cannot create a no-program control group by
design. They at least provide a way to estimate impact by utilizing those who did not participate
as a comparison. A big ethical advantage is that no one is denied service – the control group
comprises people who, for their own reasons, didn’t use the program, so you’re not withholding
anything, merely observing outcomes. This design was, for example, applied in evaluating
national programs like training for the unemployed: researchers compared those who attended
the training to similar jobseekers who did not. If implemented carefully, a matched design can be
less biased than a simple before-after analysis and can concentrate on those who received the
intervention, where effects are likely strongest. Another practical benefit is efficiency: Unlike
area designs which include entire eligible populations (many of whom might not participate,
diluting average effects), the matched group design focuses on participants vs non-participants.
Because the effect is concentrated among participants, you often need a smaller sample to
detect impacts. This can reduce evaluation costs, especially if primary survey data is needed –
you’re surveying maybe a few thousand participants and a matched set of non-participants,
rather than an entire population across areas. It’s also adaptable to various data sources: one
can use administrative data (e.g. earnings records of those who did vs didn’t join a program) or
do custom surveys. Overall, matched individual designs are a cornerstone of observational
impact evaluation when randomization isn’t an option – methods like propensity score
matching are built on this idea of constructing a quasi-control group from non-participants.

Weaknesses: The validity of this design hinges entirely on the quality of the matching. Any
inadequacies in the matching procedure can introduce bias in the impact estimates. The
fundamental problem is that participants might differ from non-participants in ways that are hard
to observe. For example, those who opt into a voluntary program might be more motivated or
have better support systems than those who don’t – factors which also help them succeed
independent of the program. If such differences exist and are not fully captured in the matching
variables, the impact estimate will be biased (usually overstating the program’s effect, since
participants might have done better anyway). The Purdon et al. paper gives an illustration:
suppose we match disabled jobseekers who joined a new employment program with disabled
jobseekers who didn’t, matching on age, gender, type of disability, and time on benefits. If the
program attracts the more proactive individuals (those actively seeking work), they may find jobs
at a higher rate than the non-participants even without the program. In that case, the program’s
effect would be exaggerated because the comparison group wasn’t truly equivalent in
motivation. This challenge – often called selection bias on unobservables – is the Achilles’
heel of matching designs. Analysts do their best to include all relevant characteristics in
matches, but one can never be sure that unobservable differences (like motivation, social
support, innate ability) aren’t driving the results. Additionally, matched designs can be complex
to implement: one needs a good dataset on both participants and non-participants and a
method to find close matches for each participant. If some participants are very unique (no
comparable non-participant), you might have to drop them or accept a bad match, which affects
the estimate. There’s also the risk of “matching on the wrong factors” if the evaluator doesn’t
correctly identify which characteristics matter for both selection and outcome. Modern statistical
matching methods help but do not eliminate these issues. In summary, while matched
comparison designs improve over naive comparisons, they rely on the assumption that you’ve
accounted for all outcome-relevant differences between participants and non-participants – an
assumption that is untestable and often questionable. Therefore, the results, though useful, are
considered less definitive than an RCT’s. They may, however, be the best available evidence
for nationwide programs where an experiment wasn’t done.

Applications: Many policy evaluations use matching. In job training and labor programs, a
common approach is to use administrative data to match training participants with similar
non-participants to estimate employment impacts (the famous LaLonde (1986) study examined
how well such methods replicate experimental results, highlighting their limitations). In
education, if a new tutoring program is offered but not required, evaluators might compare
students who used the tutoring with those who didn’t, matching on prior grades, to gauge impact
on test scores. In health, suppose a new screening is offered in a region but uptake is
voluntary; analysts could match patients who took the screening with those who didn’t on
demographics and health history to see if outcomes (e.g. disease detection rates) differ.
Policymakers often use matched group designs when implementing a program broadly but still
wanting an evaluation: for example, the UK’s New Deal for Disabled People (NDDP) was
evaluated by comparing participants to non-participants with similar profiles. The key is to be
cautious with interpretation: positive findings from a matched comparison are suggestive, but
one must consider that some of the effect could be due to underlying differences.

Before-After (Pre-Post) Studies

A before-after study (also called pre-post study) is the simplest form of impact evaluation: it
looks at the outcomes of the target population before a program and after a program, and
attributes any change to the program. Essentially, the pre-program data serve as a baseline
“control” for the post-program outcomes. For example, if the unemployment rate among youth
was 15% before a new training program and 10% after the program, one might conclude the
program reduced unemployment by 5 percentage points. This design does not use an external
control group – the comparison is internal, over time.

In practice, outcomes are measured at one or more points prior to implementation and then
again at one or more points after implementation. The basic version uses just one before and
one after measurement, but collecting multiple observations (a time series) before and after can
strengthen the analysis. Before-after designs are most commonly used when a policy is rolled
out nationwide or to an entire population without a pilot or control. In that case, evaluators
rely on historical baseline data as the only point of reference.

Strengths: The primary appeal of before-after studies is that they can be done even if a
program is universal or mandatory, where no contemporaneous control group exists. In
theory, this allows one to gauge impact even for nationwide policies – you essentially ask, “did
the key metrics improve after the policy was introduced?” Also, they are straightforward and
inexpensive since they often use existing data (administrative records or surveys over time) and
don’t require complex sampling or randomization. If the measured change is very large and
sudden, and coincides exactly with the program, a before-after study might provide convincing
evidence of impact (because it’s unlikely something else caused such a sharp change at that
exact time). For example, if a law requiring seatbelt use is implemented and immediately
fatalities drop by 20% the next year, one might reasonably infer a causal effect. Thus, in
situations of dramatic change or when alternative factors can be ruled out, before-after can
reveal meaningful insights. It’s essentially an “interrupted time series” approach if one has
long-term data: one looks for a clear break in the trend at the moment the intervention began.
With enough pre- and post-data points, this can be a powerful approach to identify an
intervention’s effect by checking if the post-intervention trend deviates from the established
baseline trend. However, doing this well (often called time-series analysis or interrupted time
series design) requires strong data and analytic techniques and is more complex than a simple
pre-post comparison. Still, the fact remains that a basic before-after is sometimes the only
feasible evaluation if no control group can be obtained. Policymakers may use it as a rough
initial indication of impact.

Weaknesses: The big problem is attribution. Any change observed could be due to the
program – or due to other factors that changed concurrently. A before-after study cannot
separate the program’s effect from other changes over time. For instance, economic
conditions, seasonal variations, or other policy changes might be responsible for some or all of
the observed difference. This is especially problematic if the program’s expected effect is
relatively small compared to typical fluctuations. If unemployment normally fluctuates by a few
points year to year, a modest program effect might be indistinguishable from the normal ebb and
flow. The worst-case scenario is when the natural trend is larger than the program’s impact –
you could conclude an initiative had no effect or even a negative effect when in reality a positive
effect was swamped by an unrelated downturn, or vice versa. Without a control group,
counterfactual inference is weak – you essentially assume that the only thing causing change
was the program, which is a strong assumption. Analysts try to mitigate this by extending the
time series (to understand underlying trends) and looking for a distinct “shift” when the program
started. Even then, one must assume no other interventions or events coincided with that shift.
Additionally, this approach often relies on administrative or aggregate data, which limits the
range of outcomes you can examine (you may only have broad indicators, not nuanced
individual outcomes). Another drawback is timing – to be confident in a before-after, you often
need to wait and collect data for some time after implementation to ensure any effect is captured
and sustained. This means results come long after the program start, reducing their usefulness
for quick feedback. In summary, before-after designs have very low internal validity – many
alternative explanations for the observed changes exist, so any conclusions about impact are
tentative. They are a method of last resort or a complement to other evidence.

Applications: Before-after analysis is common in policy areas where experimental or

quasi-experimental designs are infeasible. For example, if a nationwide education reform (like
a new curriculum standard) is implemented in 2024, one might compare test scores from 2023
vs 2025 to see if there’s improvement – albeit with caution that other changes in 2024–2025
(funding, demographic shifts, etc.) could influence scores. In public health, if a new law bans
smoking in public places across the country, researchers might compare health outcomes (e.g.
hospital admissions for asthma or heart attacks) before and after the ban, often using an
interrupted time series approach to strengthen inference. Similarly, macro-level policies (tax
changes, national minimum wage hikes) are sometimes initially evaluated with before-after
trends. For instance, a straightforward analysis might look at employment rates before and after
a minimum wage increase in one state. However, to credibly attribute changes to the law,
researchers often enhance this by adding a control state (which leads to the
difference-in-differences design, discussed next). In short, before-after data is usually presented
but interpreted with skepticism unless the change is unmistakably attributable to the policy.
Policymakers use it for preliminary insights or when nothing better is available, but they
understand its limitations.

Difference-in-Differences (DID)

Difference-in-Differences (DiD) is a powerful evaluation method that combines features of the

before-after and comparison group approaches. In a DiD design, we observe two groups (one
exposed to the intervention, one not) at two or more points in time (before and after the
intervention). We then compare the changes in outcomes over time between the groups.
Essentially, each group has its own before-after change, and the difference between those
changes is the estimated impact of the program. This differences-of-differences removes
common trends, under the key assumption that in the absence of the program the two groups
would have experienced the same change over time (the parallel trends assumption).

The DiD approach is often explained with a 2x2 table: we have outcomes for Group A
(treatment) and Group B (control) at Time 0 (pre) and Time 1 (post). We compute the difference
(Time1 – Time0) for Group A and the difference for Group B. Subtracting Group B’s change
from Group A’s change gives the DiD estimate. Group B’s change is an estimate of the “natural”
or background change that would have happened to Group A as well without the intervention.
By netting that out, we isolate the policy’s effect.

Difference-in-differences is more of an estimation strategy than a standalone design. In fact, it

can be applied on top of other designs. For example, you could have a matched comparison
and still do a DiD if you have baseline and follow-up data. The only case where it doesn’t apply
is a pure single-group before-after (since you need at least two groups for a difference).

Strengths: DiD has a major advantage of controlling for any fixed differences between the
groups. Unlike a simple post-only comparison (which could be biased if groups differed initially)
or a simple pre-post (which could be biased by time trends), DiD differences out both the
baseline difference and the common trend. This makes the estimate more robust. It effectively
uses the control group as a way to subtract the confounding trend that affected both groups.
For instance, if employment rose 2% generally due to economic growth, and the treatment
group rose 5%, DiD would attribute only the extra 3% to the program. DiD is particularly useful
when randomized experiments are not feasible but you have a natural experiment or policy
change affecting one group and not another. A classic example in labor economics is Card and
Krueger’s minimum wage study: New Jersey raised its minimum wage while neighboring
Pennsylvania did not, and researchers surveyed fast-food restaurants in both states before and
after the increase. Using DiD, they found no significant job loss in NJ relative to PA, meaning the
employment trend in NJ (the treatment state) did not worsen compared to PA (control state)
after the wage hike. This approach controlled for any regional trends affecting both states (like a
regional recession or fast-food industry changes) and focused on the divergence at the policy
implementation. In evaluation terms, DiD often provides a reasonable approximation of an
experiment when the parallel trend assumption holds, since it mimics what an RCT would do:
account for initial differences and secular changes. It’s also relatively easy to implement with
panel data or repeated cross-sections, using regression methods to adjust and to test
sensitivity. Additionally, DiD can be extended to multiple time periods and groups, and can
incorporate more complex models (e.g. adding covariates). In summary, DiD strengthens
causal inference in non-experimental settings and is widely regarded as a credible method
when the right conditions are met.

Weaknesses: The validity of DiD rests on the parallel trends assumption – that the control
group provides a true counterfactual for how the treatment group would have changed in the
absence of the intervention. This assumption is not directly testable (since we never observe the
treated group’s no-treatment trajectory), but one can check past trends: if the two groups had
similar trends pre-intervention, it boosts confidence. If trends were diverging pre-program, the
assumption is suspect. Another issue is if something else changes at the same time as the
program but affects the groups differently. For example, if another policy or shock hit the
treatment group when the program started, that violates the assumption and could bias the DiD
estimate. In Card & Krueger’s case, if NJ had other simultaneous changes (e.g. a new tax law
affecting restaurants), that could confound results. DiD also assumes the composition of groups
remains comparable – if there is differential attrition or migration between groups over time, that
complicates analysis. Sometimes spillover effects can occur: if the policy indirectly affects the
control group (maybe workers commute from NJ to PA or vice versa), then the control is no
longer a true control. Moreover, DiD typically provides an average treatment effect on the
treated group relative to control, but if the groups differ or the effect varies over time,
interpretation can be nuanced. Despite these concerns, DiD is generally seen as one of the
more reliable non-random methods. It’s essentially a refinement over a simple before-after or
simple group comparison, combining the strengths of both. One must just carefully justify that
no major differential shocks aside from the program occurred. Graphical checks of trends and
placebo tests (assuming false intervention dates to see if differences appear when nothing
should have changed) are often used to support the assumptions.

Applications: Difference-in-differences is extremely common in policy evaluation, especially for

natural experiments and phased roll-outs. In education, suppose a new policy (like reduced
class size) is implemented in one district but not in another; by comparing test score changes
over time between the districts, one can use DiD. In public health, if one region launches a new
health campaign (e.g. anti-smoking) and another similar region does not, DiD can estimate the
campaign’s impact on smoking rates or health outcomes. We already mentioned labor
economics with minimum wage laws; another example is evaluating job programs introduced in
certain areas: one could compare employment trends in pilot areas vs non-pilot areas before
and after the program. This was done in Britain for the Employment Zones initiative – some
areas had the new program and others didn’t, and evaluators compared unemployment exit
rates over time. Essentially, DiD requires that you have a comparison group that didn’t get the
intervention at the same time, plus data from before the intervention. When those conditions are
met, it becomes a go-to method. Its popularity in economics and social sciences stems from its
intuitive appeal and the fact that policy changes often lend themselves to this setup. As a
concrete case, consider a health policy: one Canadian province introduces free dental care for
children in 2020 while a neighboring province does not; researchers can compare dental health
outcomes from 2018 to 2022 in both provinces – if the treated province improved more, that
difference-in-differences is evidence of the policy’s effect. DiD results, especially from
well-chosen natural experiments, have influenced policy debates by providing evidence akin to
a controlled trial.

Cost-Benefit Analysis

After understanding how a program operates (process) and what its impacts are (via one of the
designs above), policymakers often ask: Is the program worth it? This is where Cost-Benefit
Analysis (CBA) comes in. Cost-benefit analysis is not an impact evaluation design per se, but
rather an analytic exercise that builds on impact findings. It involves identifying all the costs
and all the benefits associated with a program and converting them into monetary terms to
assess the program’s overall value or return on investment. Essentially, CBA asks: Do the
benefits of the policy outweigh its costs, and by how much?

The steps in a cost-benefit analysis include: (1) determining the program’s effects or outcomes
(this is where impact evaluation results feed in), (2) assigning a monetary value to each of those
outcomes (benefits), and (3) summing up the benefits and comparing them to the program’s
costs (both direct implementation costs and any indirect costs). The result could be presented
as a net benefit (total benefits minus total costs) or a benefit-cost ratio (total benefits divided
by total costs). If benefits exceed costs, the program is economically worthwhile in a pure
efficiency sense. CBA can also compare multiple programs or policy alternatives: which gives
the highest net benefit or best return for the money?. This helps in choosing between options.

A simple example: consider a job training program. Benefits might include increased earnings
for participants, higher tax revenues from those earnings, and reduced welfare payments if
participants gain employment. Costs would include the program’s operating expenses and
perhaps opportunity costs of participants’ time. If, when monetized, the program yields (say)
$5,000 in benefits per participant at a cost of $4,000 per participant, that’s a positive net benefit
of $1,000 per person – a good investment. If instead benefits were only $3,000, the net would
be –$1,000, suggesting the program’s costs outweigh its gains.

Key considerations: CBA requires valuing outcomes in monetary terms, which can be
challenging and sometimes controversial. Some outcomes are straightforward (earnings, taxes,
healthcare costs saved), but others are intangible (value of improved health or reduced crime).
Often, analysts stick to easily monetizable outcomes and acknowledge those they left out.
Importantly, perspective matters: costs and benefits can be counted from the perspective of
the government budget, the participants themselves, or society at large. For example, getting
someone off unemployment benefits is a savings to the government (benefit from taxpayer
perspective), but for the individual the benefit is their new earnings. A comprehensive CBA tries
to include all societal costs and benefits, potentially highlighting how they accrue to different
groups. If distribution matters, sometimes costs and benefits are reported separately for
government vs participants vs others. Additionally, timing is considered – costs and benefits
over time may be discounted to present value.

Role in evaluation: Cost-benefit analysis is crucial for decision-making. A program may have a
positive impact, but is it cost-effective? Some interventions achieve only small gains at very high
cost, while others achieve large gains cheaply. CBA helps in resource allocation decisions –
whether to continue, scale up, or cut a program. For instance, if Program A and Program B both
reduce unemployment by 5%, but A costs twice as much as B, a cost-benefit perspective favors
B. It can also set a benchmark: e.g., if an environmental regulation costs $100 million but yields
health benefits valued at $500 million, it’s clearly justified. If it yields only $50 million, perhaps
not.

Challenges: As noted, quantifying certain benefits is difficult. Some social outcomes (improved
quality of life, community cohesion, reduced crime) are hard to price. Analysts may use proxies
(like “value of a statistical life” for saved lives, or willingness-to-pay studies) but these can be
debated. If important benefits are omitted because they can’t be measured, a CBA might
undervalue a program. Moreover, if impact evaluation is uncertain about the effect size, the CBA
inherits that uncertainty – a cost-benefit result is only as reliable as the impact estimates and
cost data behind it.

Examples: The document gives an illustration with the UK’s ONE program evaluation. The
ONE initiative (which aimed to get welfare recipients into work by integrating employment
services) was subject to a cost-benefit analysis examining things like: how much did ONE cost
to operate versus how much it saved in social security payments, how much additional tax
revenue came from clients who found jobs, and any wider economic benefits of increased
employment. They noted some effects (like reduced crime or improved health due to
employment) would not be included because they’re hard to measure or attribute. Another
example from the text is the New Deal for Lone Parents (NDLP) prototype. The cost-benefit
analysis for NDLP found that the economic returns were slightly less than the program’s cost –
essentially, the program almost paid for itself but not quite. It was estimated that about 20% of
the jobs that lone parents got were directly attributable to the program (i.e., “additional” jobs that
wouldn’t have happened otherwise). If that figure had been 23% instead, the benefits (in terms
of welfare savings and added taxes from those additional workers) would have equaled the
program costs. This sensitivity analysis highlights how a small change in measured impact can
tip a program from looking not worthwhile to worthwhile. Such findings guide policymakers: in
this case, NDLP’s initial cost-benefit was borderline, which might justify efforts to improve the
program’s effectiveness (to get that extra impact) or reduce costs.

In fields like health, cost-benefit (or its cousin, cost-effectiveness analysis) is routinely used. For
example, a new medical treatment might be evaluated for cost per quality-adjusted life year
(QALY) gained. In education, an intervention’s cost per increase in test score or graduation rate
may be calculated. These help compare very different programs on a common scale (dollars).
One noteworthy initiative in the U.S. is the Perry Preschool program: decades-long research
calculated the economic return of this early childhood education program (through participants’
higher earnings, reduced crime, etc.) and found a very high benefit-cost ratio, bolstering the
case for preschool investments.

In sum, cost-benefit analysis adds a critical economic perspective to evaluation. A policy might
be effective, but society has limited resources – CBA helps determine if an intervention delivers
enough bang for the buck. It can justify programs with high returns or signal the need to
redesign those with poor returns. When combined with solid impact evaluations, cost-benefit
analysis provides a comprehensive view of a policy’s overall merit, informing strategic policy
decisions beyond mere effectiveness.

Conclusion

Conclusion: Evaluating public policies requires a combination of methods to answer both the
“how” and the “how much” questions. Process evaluations ensure we understand
implementation and can improve delivery, while impact evaluations (through randomized trials
or well-crafted quasi-experiments like matching, before-after with time series, and
difference-in-differences) aim to isolate the causal effects of programs. Each method comes
with trade-offs in feasibility, validity, and ethics. The strongest evidence comes from randomized
trials, but when those are not possible, matched comparisons and DiD approaches can provide
valuable insights if used carefully. Often, multiple approaches are applied together – for
example, a process evaluation alongside an impact study, or using DiD to strengthen a matched
design, etc., to bolster the evidence base. Finally, cost-benefit analysis brings it all together by
translating effects into economic terms, helping policymakers decide which interventions provide
the greatest social value.

In real-world policy fields like health, education, and labor, these tools have been essential.
From evaluating education reforms and job training programs to analyzing healthcare policies,
governments and researchers have built a robust toolkit: they pilot and test interventions
(sometimes randomly), compare outcomes with control groups (matched or otherwise), track
changes over time, and weigh the benefits against the costs. By understanding the principles,
strengths, and weaknesses of each method, an evaluator can design studies that provide
credible and actionable evidence. As this comprehensive overview shows, no single method is
best in all situations – the art of evaluation is choosing the right design or combination for the
policy question at hand. Ultimately, rigorous policy evaluation helps ensure that public programs
achieve desired outcomes and that resources are used effectively to improve society’s
well-being.

Sources: The analysis above is based on Research Methods for Policy Evaluation by Purdon et
al. (2001), complemented by examples from subsequent research and practical evaluations in
various policy domains. Each method and concept is documented with references to the source
text and related evaluation literature to aid further study.