ARTICLE IN PRESS

Social Science & Medicine 65 (2007) 2116–2129

www.elsevier.com/locate/socscimed

Cost-effectiveness analysis and formulary decision making

in England: Findings from research
Iestyn P. Williamsa,, Stirling Bryanb
a
The Health Services Management Centre, University of Birmingham, Park House Birmingham,
40 Edgbaston Park Road, Edgbaston, Birmingham B15 2RT, UK
b
Health Economics Facility, University of Birmingham, Edgbaston, Birmingham B15 2TT, UK
Available online 14 August 2007

Abstract

In a context of rapid technological advances in health care and increasing demand for expensive treatments, local formulary
committees are key players in the management of scarce resources. However, little is known about the information and
processes used when making decisions on the inclusion of new treatments. This paper reports research on the use of economic
evaluations in technology coverage decisions in England, although the findings have a relevance to other health care systems
with devolved responsibility for resource allocation. It reports a study of four local formulary committees in which both
qualitative and quantitative data were collected. Our main research finding is that it is an exception for cost-effectiveness
analysis to inform technology coverage decisions. Barriers to use include access and expertise levels, concerns relating to the
independence of analyses and problems with implementation of study recommendations. Further barriers derive from the
constraints on decision makers, a lack of clarity over functions and aims of local committees, and the challenge of disinvestment
in medical technologies. The relative weakness of the research-practice dynamics in this context suggests the need for a
rethinking of the role of both analysts and decision makers. Our research supports the view that in order to be useful, analysis
needs to better reflect the constraints of the local decision-making environment. We also recommend that local decision-making
committees and bodies in the National Health Service more clearly identify the ‘problems’ which they are charged with solving
and how their outputs contribute to broader finance and commissioning functions. This would help to establish the ways in
which the routine use of cost-effectiveness analysis might become a reality.

Summary of findings

 Local formulary decision-making committees vary in their capacity, functions and scope of responsibility. Their
primary function appears to be to control spending rather than evidence-based technology coverage.
 Most committees routinely request information on clinical effect and costs but few request cost-effectiveness
information.
 Case study committees had only limited capacity to access and interpret economic evaluations. Further barriers
included concerns regarding bias in studies, the inability to implement savings, and ethical objections to underlying
values of health economics.
 A number of features of the decision-making environment appeared to militate against emphasis on cost-
effectiveness analysis including: unclear relationships with resource allocators; an explicitly political decision-
making process, and; poorly specified decision-making criteria.

Corresponding author. Tel.: +44 121 414 7063; fax: +44 121 414 7051.
E-mail addresses: [email protected] (I.P. Williams), [email protected] (S. Bryan).

0277-9536/$ - see front matter r 2007 Elsevier Ltd. All rights reserved.
doi:10.1016/j.socscimed.2007.06.009
 ARTICLE IN PRESS
I.P. Williams, S. Bryan / Social Science & Medicine 65 (2007) 2116–2129 2117

 These factors, combined with constraints on the capacity to generate, access and interpret information, led to a
minor role for cost-effectiveness analysis in the decision-making process.

r 2007 Elsevier Ltd. All rights reserved.

Keywords: England; Cost effectiveness; Technology coverage; Health policy; National health service (NHS); Economic evaluation;
Formulary decision making; UK

Introduction newly licensed treatments nor the majority of

treatments licensed prior to the Institute’s inception.
Decisions to ‘cover’ technologies (implying that Therefore, local arrangements are still required for
their cost will be reimbursed as part of an insurance decisions concerning the majority of health care
package) are taken in health systems where private provision. In a context of rapid technological
insurance is prevalent and in systems dominated by advances in health care and increasing demand for
government health insurance programmes. In order expensive treatments, local formulary committees are
to be ‘evidence based,’ coverage decisions need to be therefore potential key players in the management of
informed by a range of information derived from scarce resources. However, little is known about the
clinical trials. Techniques such as systematic reviews information and processes used by these when
in health technology assessment enable clinical making decisions on the inclusion of new treatments.
evidence to be presented as potential solutions to The focus of this paper is on the use of economic
policy problems (Lehoux, Denis, Tailliez, & Hivon, evaluations in technology coverage decisions in
2005; Mays, Pope, & Popay, 2005). However, England. It reports findings of a study into local
data on effectiveness do not provide all of the committees in which both qualitative and quantita-
information required by those who make resource tive data were collected. We sought to explore how
allocation decisions and the importance of incor- such committees operate, the information they use
porating analysis of cost-effectiveness has become and the extent to which economic evaluation
increasingly accepted. However, successful applica- featured in this.
tion of cost-effectiveness principles has proved to be
a challenge to decision makers across a range of Background
health care systems (Blumstein, 1997; Duthie,
Trueman, Chancellor, & Diez, 1999; Hoffman & In an earlier paper the authors reviewed previous
von der Schulenburg, 2000; Neumann, 2005; Ross, research in this area and discussed issues of
1995; Walley, Barton, Cooke, & Drummond, 1997). ‘accessibility’ and ‘acceptability’ in the application
Perhaps the most significant example of the of health economics analysis to local formulary
routine use of formal cost-effectiveness analysis at a decision making (Williams & Bryan, 2007). We use
national policy level is the National Institute for this framework here to analyse the emerging findings.
Health and Clinical Excellence (NICE). NICE makes As such, the study adds to a growing literature on the
recommendations on technology coverage for the role and impact of cost-effectiveness analysis across a
National Health Service (NHS) in England and range of health care settings (for example: Duthie
implementation of this guidance is mandatory for et al., 1999; Fattore & Torbica, 2006; Iglesias,
local health care purchasers and providers. NICE Drummond, & Rovira, 2005; Ross, 1995; Sintonen,
appraisals are each preceded by an ‘assessment’ 2000; Spath, Charavel, Morelle, & Carrere, 2003;
phase which is carried out by independent academic Walley et al., 1997) and is designed to build on the
review teams. A recent study by the authors found insights developed previously. There are two strands
that techniques such as systematic reviewing of to this literature: the first includes a number of
clinical evidence and economic evaluation are at the studies focussing specifically on the use of economic
heart of the Technology Appraisals Programme evaluation and employing self-complete question-
albeit with caveats relating to the nature of the naires (Drummond, Cooke, & Walley, 1997; Motheral,
latter’s incorporation into decision making (Bryan, Grizzle, Armstrong, Cox, & Fairman, 2000;
Williams, & McIver, 2007; Williams, Bryan, & Sintonen, 2000) and/or interviews (including group
McIver, 2007). However, the existing body of NICE discussions) (Duthie et al., 1999; Hoffmann et al.,
recommendations exhausts neither the full gamut of 2002; Rijkom & Rutten, 2000; Ross, 1995). These
 ARTICLE IN PRESS
2118 I.P. Williams, S. Bryan / Social Science & Medicine 65 (2007) 2116–2129

studies are useful in highlighting a number of specific arenas of technology coverage decision
barriers to the adoption of health economics— making difficult. However, most NHS hospitals
particularly by meso or local level decision ma- have both a Drug and Therapeutics Committee
kers—many of which appear to apply across health (DTC) that controls the introduction of new drugs,
care systems (see Table 1). The second strand and a formulary that acts as a key implementation
includes case studies of formulary or priority setting mechanism in this process (Haslé-Pham et al., 2005).
committees, typically involving observation and The extent to which such committees determine drug
interview ( Haslé-Pham, Arnould, Späth, Follet, policy for the hospital is highly influenced by the
Duru, & Marquis, 2005; Jenkings & Barber, 2004; clinical directorate structure and pharmacists work-
Madden, Martin, Downey, & Singer, 2005; Martin, ing within this (Cotter & McKee, 1997). Within
Pater, & Singer, 2001). However, only a very small primary care, Area Prescribing Committees make
number of studies have combined a focus on the use decisions on the use of health technologies and
of health economics analysis with a case-study there are increasingly links between PCTs and acute
approach (McDonald, 2002). This is the first study trusts in moves towards health economy-wide
that combines quantitative analysis of the informa- formularies and joint commissioning with local
tion requested by local formulary committees authority agencies (Wade, Smith, Peck, & Freeman,
(including the extent to which this includes cost- 2006). In some regions there are also Priorities
effectiveness data and analysis) with an in-depth Networks which make recommendations on the
exploration involving a series of qualitative case introduction of new treatments and which have
studies. representation from relevant local health orga-
A further strand to literature in this area focuses nisations (Newdick, 2005). In looking to research
on the attempt to apply flexible and applicable technology coverage decision making at a local
frameworks for priority setting at local levels. The level, such bodies represent a logical starting
most widely implemented of these is Programme point.
Budgeting and Marginal Analysis (PBMA): a Committees making these adoption and intro-
process which aims to maximise benefit, broadly duction decisions face a series of uncertainties, for
defined, by enabling explicit choices to be made example, in determining what the benefits and risks
from within a constrained budget (Mitton & of a new technology are as well as the likely
Donaldson, 2003). PBMA has been used at local financial implications of its adoption within their
decision-making levels across the UK, Australia, patient population (Gelijns, Brown, Magnell,
New Zealand and Canada with mixed results Ronchi, & Moskowitz, 2005). This would appear
(Mitton & Donaldson, 2001; Mitton, Patten, & to imply an increasingly central role in formulary
Donaldson, 2004). The PBMA literature indicates decision making for analytical disciplines such as
instances of progress in the incorporation of health economics (Bakst, 1995). However, as we
economic principles such as opportunity cost, have seen, research suggests that the impact of
particularly in situations where senior clinical and economic evaluation on local level decision making
management support is forthcoming. There are also is moderate at best. Reasons for this include a
a number of recurring difficulties experienced by perceived lack of availability of studies, especially
implementers of PBMA, including lack of informa- given the short timescales within which decisions
tion on the benefits and costs of medical technol- have to be made (Sloan, Whetten-Goldstein, &
ogies and the persistent challenge of implementing Wilson, 1997; Luce & Brown, 1995; Lyles, Luce, &
disinvestment in existing technologies (Bohmer, Rentz, 1997), and difficulties with study comprehen-
Pain, Watt, Abernethy, & Sceats, 2001; Posnett & sion and interpretation (Drummond et al., 1997;
Street, 1996). Although the study reported here did Haslé-Pham et al., 2005; Sloan et al., 1997). It is
not focus on the application of PBMA, its findings therefore important that committees receive rele-
may help to shed light on the barriers previously vant and timely information and can call on the
experienced by implementers. expertise needed to process this information in
arriving at a decision. In the UK context, Jenkings
The local decision-making environment and Barber (2004) indicate that within such
committees some pharmacist time is usually allo-
The complexity of local commissioning arrange- cated to evidence collection and synthesis but note
ments in England makes identification of the that this capacity is often highly constrained.
 ARTICLE IN PRESS
I.P. Williams, S. Bryan / Social Science & Medicine 65 (2007) 2116–2129 2119

Table 1
Summary of previous research into local decision making

Location & authors of Respondent populations Reported barriers to the use of economic evaluation
studies

UK
Drummond et al. Prescribing advisers, hospital directors of  System rigidities: difficulties in moving resources
(1997) pharmacy and directors of public health between budgets
 Lack of independent sponsorship
Walley et al. (1997) Medical and pharmaceutical advisors  Too many assumptions made in economic studies
Duthie et al. (1999) General practice, hospital and health authority  Short-term pressures to contain cost
decision makers
 Difficulty in the interpretation of studies
Hoffman et al. (2002) Health Authority decision makers  Inability to realise savings identified in studies
McDonald (2002) Local health commissioners  Results presented in too aggregate a manner
 Ethical objections to health economics
 Stakeholder resistance to rationing

Other European
von der Schulenburg A range of local (and national) decision makers  Lack of independent sponsorship
(2000)  Inability to realise savings identified in studies
 Lack of time to collect and analyse information
Spath et al. (2003) Pharmacists in the French health care system  Insufficient health economics training to interpret
Fattore and Torbica Clinical and other health professionals
studies
(2006)  Short-term perspectives of decision makers
 Excessive focus on expenditure control
 System rigidities: difficulties in moving resources
between budgets
 Tight budgets preventing the release of resources to
adopt new technologies
 Too many assumptions made in economic studies
 Difficulties in accessing information

US
Luce and Brown Decision makers from hospitals, HMOs and  Difficulties in accessing information
(1995) third-party payers  Lack of timeliness of studies
 Lack of independent sponsorship
Lyles et al. (1997) Pharmacy decision makers in managed care  Divergence between policy and HTA frameworks
organisations
 Lack of information on the realisable savings for
Sloan et al. (1997) Hospital pharmacists hospitals
Motheral et al. (2000) Pharmacists or physicians working in health  Inadequate expertise in health economics amongst
care organizations decision makers
Grizzle et al. (2000) USA managed care decision makers
 Lack of resources to generate or access
 Perceived lack of relevance

Other
Ross (1995) Decision makers at Commonwealth and State  Lack of timeliness of studies
levels  Difficulty in the interpretation of studies
Iglesias et al. (2005) Decision makers from health care systems in  Perceived poor quality of economic analysis and
Latin American countries research evidence
 Poor collaboration between decision makers and
researchers
 Lack of a clearly defined set of criteria to guide
decision making

Research also suggests that written evidence tends concerns (such as equity and ‘rule of rescue’) which
to be just one of a range of factors which influence will be:
deliberations. As well as accessing clinical studies
and systematic review reports, decision makers are subjectively factored into the process y and not
likely to take into account ethical and value-based necessarily quantified in a manner that would
 ARTICLE IN PRESS
2120 I.P. Williams, S. Bryan / Social Science & Medicine 65 (2007) 2116–2129

allow subjective information to be combined with advisors’ or ‘pharmaceutical advisors.’ Non-respon-

other objective, quantified criteria (e.g. cost- ders, at 3 months, were sent a reminder. Organisa-
effectiveness analyses) into a comprehensive tions that did not respond to either the initial letter
decision model. or the reminder were telephoned by a member of the
(Teutsch & Berger, 2005, p. 488) research team to find out reasons for non-response.
Jenkings and Barber (2004, p. 1764) found that
Qualitative case studies
the social and interactive process of reaching
decisions was also influenced by ‘clinician excite-
Case studies were then conducted of four
ment,’ ‘applicant personality’ and ‘patient demand’
decision-making committees in order to explore
and that ‘the less clear cut the reported efficacy, the
the decision-making process in more depth. Selec-
more other factors weighed in the discussion.’ The
tion of these was based, in part, on the desire to
context in which such committees operate—data
achieve some geographical coverage and to include
shortages, uncertainty and a locally-specific political
committees operating in or across both primary and
environment—means that negotiation and compro-
secondary health care sectors. Box 1 gives a brief
mise are often required in order for consensus
description of each selected committee. For each
decisions to be reached. These studies confirm that
case study site, a member of the research team
use of clinical and policy evidence can be impeded
contacted a representative of the organisation or
by both weaknesses in the interface with decision
committee in question to discuss the possibility of
makers and lack of consensus between actors in the
conducting the case study. Following this, the
deliberative process (Lehoux et al., 2005).
research team produced a draft protocol for the
proposed case study and attended a committee
Methods meeting where the proposed research was explained
to the committee. Verbal consent was obtained from
There were two strands to the research. The first committee members at this stage. Written consent
involved collation and analysis of information was then sought from the committee chair and
proformas used by decision-making committees from individual committee members partaking in
when considering proposed new technologies— interviews.
referred to hereafter as New Technology Request We used three main data collection methods in
Forms (NTRFs). The second strand involved quali- the case studies:
tative case studies of decision-making committees.
 Documentary analysis
Analysis of new technology request forms  Observation of committee meetings
 Semi-structured interviews with committee
The objectives of this exercise were: members

 to identify what, if any, NTRFs are used by local The research team had access to documents
NHS decision-making committees operating in relating to the committee and its activities—for
England, example mission statements, annual reports and
 to ascertain what information is routinely papers received in support of new formulary
requested by these committees when making applications. This helped to provide a context to
decisions on the inclusion of new technologies, observation and interview data. The team adopted
and an overt, non-participant approach to the observa-
 to gauge the extent to which cost-effectiveness tion of committee meetings involving open note
analyses are included within this information set. taking on general discussions and specific recording
of references to cost and cost-effectiveness.
All primary and secondary care NHS organisa- Although it was not always possible to ensure that
tions were mailed a letter requesting a blank copy of more than one note taker was present at meetings
the NTRF currently used in gathering information this was the preferred approach. Semi-structured
about a proposed addition to their formulary. interviews were then conducted with a sample of
Within secondary care trusts, ‘chief pharmacists’ committee members selected to reflect the different
were targeted, and within PCTs, ‘prescribing interest groups represented on the committee,
 ARTICLE IN PRESS
I.P. Williams, S. Bryan / Social Science & Medicine 65 (2007) 2116–2129 2121

such as: pharmacists, hospital clinicians, PCT

Box 1 representatives (including pharmaceutical advisors),
Case study committees general practitioners, lay representatives, senior
Case study A: County-wide Priorities nurses, financial specialists and academic advisors
Network (including specialists in law and pharmacy). Inter-
Members included senior representa- viewees were selected using information drawn from
tives of each NHS Trust and Primary Care preliminary contact with committees and from the
Trust (PCT) in the area, as well as other documentation on committee composition. Poten-
stakeholder organisations. The committee tial participants were given the choice of face-
met monthly and reported to each mem- to-face or telephone interview and all interviews
ber Trust and PCT who then decided were tape recorded after providing assurances of
whether or not to implement its recom- confidentiality. Interviews lasted approximately
mendations. 1 hour and were transcribed.
Case study B: Hospital medicine man- Analysis of all data was performed by a minimum
agement committee of two researchers operating independently, identi-
Committee membership consisted of fying themes using NVIVO. Analysts then com-
hospital pharmacists, a small medicines pared and discussed any differences in the themes
management team and a representative of each had identified. Once a list of coded themes had
each clinical directorate. Membership been established, the research team, in collabora-
from outside of the Trust included aca- tion, further interrogated the data from the inter-
demic and primary care representatives. views, observations and documents in order to
The committee met monthly to decide achieve greater refinement of the analysis. In
what technologies were introduced onto presenting findings here we have avoided verbatim
the Trust formularies but did not control quotation of committee discussion. Interviewee
the budget for new technologies—this comments are however used to illustrate themes
authority resided with individual directo- and findings: these are accompanied by an identifier
rates. code (i.e. Committee A, B, C or D and interviewee
Case study C: Primary care area medi- 1, 2, 3 etc.). Where a point was made by a large
cines management number of interviewees, this is indicated in the text
Members included pharmacists, PCT and where a point was made by a smaller number,
prescribing leads, secondary care consul- this is also clearly noted.
tants and specialists in pharmaceutical
public health. The committee met bi- Findings from the analysis of NTRFs
monthly and reported to the PCT Profes-
sional Executive Committees (PECs) and Response and coverage
the hospital Clinical Governance Commit-
tee on a quarterly basis. The committee’s For the secondary care survey, we approached a
role was advisory. total of 188 Trusts of which 116 responded: 12
Case study D: Interface medicines man- indicated that their organisation’s committee did
agement committee not make use of such a proforma, 101 provided
Membership included PCT prescribing the relevant proforma, and 3 responses gave
advisors and clinicians, hospital medical information that was not relevant. However, from
directors, local Medical Committee and the responses it became clear that we additionally
local Pharmaceutical Committee represen- had information that related to 5 of the non-
tatives and finance representatives. The responding organisations because there was a
committee met bi-monthly and reported to shared committee with one of the responding
the boards of the trusts in the area through organisations. This means that in total we have
their representatives on the committee relevant information (including the statement that
regarding medicines’ management issues, no such form is in use) for 118 secondary care
particularly those affecting the interface organisations, giving us a coverage rate of 63%.
between primary and secondary care. For the primary care survey, we approached a
total of 308 primary care organisations of which 165
 ARTICLE IN PRESS
2122 I.P. Williams, S. Bryan / Social Science & Medicine 65 (2007) 2116–2129

responded; 114 indicated that their organisation did Table 2

not make use of such a proforma, 40 provided the Analysis of NTRFs
relevant proforma and 11 responses gave informa- Question on proforma Number (%) of
tion that was not relevant. In line with the organisationsa where the
secondary care data, we again had information that question is used
related to non-responding organisations because of
Secondary Primary
the existence of committees shared with responding
organisations—there were 30 non-responding orga- General questions
nisations accounted for in this way. This means that What is the evidence of clinical 84 (79) 41 (59)
in total we have relevant information for 184 benefit?
Expected number of patients to 82 (76) 20 (29)
primary care organisations, giving us a coverage
receive treatment
rate of 60%. Is the technology an add-on or 79 (74) 22 (31)
replacement?
Committees and new technology request forms Implications for other sectors (e.g. 64 (60) 32 (46)
primary care)
Conflicts of interest 59 (56) 14 (20)
The majority of secondary care respondent
What are the ‘advantages’ of the 52 (49) 23 (33)
committees use a traditional name of ‘Drugs and new technology?
Therapeutics Committee’ or something very similar. Signature of budget holder required 51 (48) 12 (17)
Whilst many committees were concerned solely with Should there be restrictions on who 49 (46) 24 (34)
a single secondary care provider, it is clear from should prescribe?
Request for license information 44 (41) 23 (33)
some of the committee names that committees are
Length of treatment course 34 (32) 7 (10)
being established either to work across a number of Place in therapy 23 (22) 19 (27)
hospitals or to work jointly across primary and Prior identification of funding if 16 (15) 6 (8)
secondary care (e.g. ‘Area Prescribing Committee,’ technology supported
‘Locality Prescribing Committee,’ etc.). Does it represent a ‘therapeutic 12 (11) 2 (3)
advance’?
In the primary care setting, there is more
Clinical experience of the 11 (10) 3 (4)
variation in the names used by prescribing commit- technology locally
tees, indicating both that such committees are less Discussion with/support of 11 (10) 5 (7)
well established and that they perform more varied colleagues
roles. The most common names, however, were Previous appraisal by NICE/ 7 (7) 8 (11)
national guidance
‘Area prescribing committee’ or ‘PCT prescribing
Numbers needed to treat 2 (2) 1 (1)
committee.’ The terms ‘area,’ ‘interface’ and ‘joint’
are commonly seen in the committee names, Questions specifically concerning cost information
Some requirement for cost 91 (86) 39 (56)
supporting the view that joint working across
information
primary and secondary care is increasing. Cost impact of technology 62 (58) 33 (47)
All NTRFs asked for details of the product/ Costs when compared to 44 (41) 18 (26)
therapy in question and indications for use. Further alternatives/current treatment
information categories, and the frequency with Price of technology 33 (31) 11 (16)
Potential savings from using 22 (21) 8 (11)
which these were requested, are shown in Table 2.
technology
The pattern of questions appears to be similar in
both secondary and primary care settings. A focus on Total 106 70
the evidence of benefit comes through clearly with the a
The organisation is our denominator and so we have counted
vast majority of organisations requiring such evidence more than once cases where the same form is known to be used
in order to approve a new technology. It is also clear by a number of organisations/committees. Thus, we have
that the vast majority of committees on whom we have included organisations that did not respond themselves but
data raise cost issues. The main focus appears to be on where responses from other organisations provided information
on the particular form used.
the cost to be covered immediately rather than the
longer-term savings that might be generated. Other
issues frequently considered are the size of the clinical
problem and the scale of the potential investment, considerations also play an important role in many
highlighted by the question concerning the expected committees. For example, many NTRFs request
number of potential patients. Implementation information on who can use or prescribe medications.
 ARTICLE IN PRESS
I.P. Williams, S. Bryan / Social Science & Medicine 65 (2007) 2116–2129 2123

Again on the implementation side, the signature of the information manager and, at the other extreme, the
budget holder is required on many NTRFs, indicating Priorities Network receiving considerable detail in both
that the support of senior colleagues is crucial in the application itself and the supporting information.
bringing about successful adoption. All four committees had some in-house data generating
A relatively small number of committees routi- capacity but levels of this capacity varied.
nely ask for information on cost-effectiveness Observation and documentary analysis confirmed
(n ¼ 19 for secondary care (19%); n ¼ 16 for that committees were concerned with the implemen-
primary care (23%)). In some cases, it is not tation as well as the evaluation of technologies.
evidence on cost-effectiveness that is requested but Information and discussion explicitly focussed on
a judgement call on ‘value for money’ by the patient numbers and budgetary implications as well
clinician putting forward the case. However, for as differences between geographical locations within
some committees the questions are more focussed the health economy. Again there was variation with
and the request is for evidence of cost-effectiveness the Priorities Network apparently making more use
in the form of published papers and other support- of health technology assessment and other research
ing materials. The assumption would appear to be evidence. Other committees focussed less on pub-
that if evidence on cost-effectiveness is available it lished evidence and more on practical implementa-
would have been read and digested by the proposer tion issues of whether and how to restrict usage and
who is then asked to respond to questions relating control prescribing. The extent of variation in the
to cost-effectiveness. The broad pattern of not levels of scrutiny applied to each application was
routinely requesting formal information on cost- reflected in the number of decisions taken: whereas
effectiveness and only in a small number of cases the Priorities Network averaged two decisions per
asking for evidence from economic analyses is a meeting (with some decisions postponed over
pattern that seems to be common to both the several meetings due to the lack of evidence),
secondary and primary care settings. another committee made decisions on 12 applica-
tions in the course of the two meetings observed.
Findings from case studies Many committee members saw their role as being
to represent their particular organisation and/or
Decision making health care sector, as well as to consider the
objective merits of an application.
Researchers attended a minimum of two meetings
for each case study site. In total 11 meetings were CA My job is to put on the agenda things which are
observed and 31 people from across the four 9 of interest to (name of PCT), review the
committees were interviewed. Each of the committees background papers, or take my share in
had some responsibility for deciding on new tech- producing the background papers and
nologies and each was responsible for the manage- essentially contributing to the debate on behalf
ment of a formulary. The relationship to funding of (name of PCT) and arguing for that point of
streams of member organisations for the three view when it comes to the committee making a
committees was through their respective representa- recommendation.
tives (See Box 1). The Acute Trust Medicines CC I take things pertinent to nurse prescribing
Management Committee requested that applicants 2 because I’m the nurse prescribing lead for
seek authority for the funding of the new technology (name of area). So if we have issues, you know,
within the Trust as part of the application. around formularies, new products, then I’d see it
A similar process was followed by each of the case as my role to take the issues there on behalf of
study committees. Papers relating to proposed for- all the nurses.
mulary additions were distributed and a presentation CD I’ve got two roles, really. One is to come at it as
was made to the committee by the applicant. Questions 1 a GP, what I would consider to be appropriate
and discussion ensued before a decision was reached in and reasonable for me to prescribe and what
closed session. Each committee had some process for would be in the normal competencies of a GP to
reviewing decisions. There was, however, significant be able to do, and to advise the committee from
variation in the depth of information submitted with that point of view. I also come at it from the
one committee receiving a one-page application PCT point of view: perhaps our financial or
supported by a short supplement from the Trust clinical governance aspects.
 ARTICLE IN PRESS
2124 I.P. Williams, S. Bryan / Social Science & Medicine 65 (2007) 2116–2129

CB I drive the primary care agenda and make the product, it’s safety and efficacy profile, the
5 secondary care physicians aware that the latest evidence about it, projected costs if they
repercussions of the drugs they are are available, what the impact is going to be on
recommending and how they prescribe them do our local economy, those sorts of things.
quite often fall on to the heads of general CD I would consider that the safety and the
practitioners and primary care trusts. 1 therapeutic indication, and the benefit of it,
those would be the critical issues. The cost is of
secondary importance, but is there. The critical
This partiality was reflected in the discussion
issues are who is it going to help, by how much,
which frequently involved achieving consensus and
and what are the risks in doing it?
satisfying the specific concerns of stakeholders.
There were a number of possible outcomes of
committee deliberation. These included: As indicated in previous studies, other factors did
seem to influence the committees such as the
 Agreement to add the technology to the formulary perspectives of committee members, especially
 Decision not to provide coverage—usually on the clinicians. Interview responses seemed to suggest
grounds of lack of evidence of effectiveness that the approach to decision making adopted by
 A decision to postpone a determination until the committee did not permit the development of
more evidence became available precedent or case law to inform prospective
 Agreement to cover a technology subject to recommendations. Therefore, each application was
development of a treatment algorithm for pre- considered on its merits, effectively from a ‘blank
scribers slate’ position. In this context it was felt by some
 Agreement to cover a technology subject to respondents that the committees were susceptible to
prescribing restrictions (such as a clinical speci- powerful personalities on, or attending, the com-
ality within a Trust) mittee.

There were no instances where it was agreed to

Use of economic evaluation
disinvest in a technology or remove any intervention
from the formulary.
Individual members of two of the four commit-
Committees were unanimous in citing evidence of
tees had some health economics expertise. For a
a technology’s clinical benefit to patients as the
third committee, a member with health economics
primary factor influencing the decisions. With some
expertise had recently left. However, while the
variation in emphasis, cost implications of intro-
specific co-opting of health economics expertise
duction were identified by interviewees as being the
seemed to be viewed as beneficial, it was not
next most sought after information—although no
considered essential by the interviewees. Two of
technologies were refused on these grounds during
the four (the Priorities Network and the primary
the case studies.
care based committee) included a request for health
CB For me it’s about the clinical benefits of a drug: economics information as part of its NTRF.
4 are they really demonstrated as being much However, the main sources of economic evaluation
better than what we’ve got already? What is the in all four cases were NICE guidance and company
quality of the evidence? sponsored evaluations, with interviewees indicating
CB Well basically the quality of the published only limited access to other sources. Reference to
2 evidence. The published trials: the number of published cost-effectiveness analyses was rare in
patients involved, the number of centres committee discussion, and the actual accessing of
involved, preferably fewer rather than greater, these forms of analysis did not occur in the course
whether the trial was sponsored by the of the case studies. The Priorities Network was the
manufacturer of the drug. Cost is obviously only body who reported previously having commis-
something that needs to be considered these sioned a review incorporating cost-effectiveness
days. analyses.
CC A summary of the product characteristics is a Our data reveal some confusion amongst commit-
3 useful starting point. You need to know tee members over the terms ‘economic evaluation’ and
something about obviously the cost of the ‘cost-effectiveness.’ Despite this, it appeared that
 ARTICLE IN PRESS
I.P. Williams, S. Bryan / Social Science & Medicine 65 (2007) 2116–2129 2125

economic evaluation was attributed more importance concerns about the robustness of analyses, and;
in interviews with Priorities Network members than ethical objections to the perceived underlying values
the other bodies. In all four cases the cost impact—the of health economics.
implications for allocation of resources locally-seemed
CD I think that what would be critical is having the
to be of greater concern. Formal CEA appeared to be
1 information available. What we don’t have as a
less frequently accessed and less highly valued than
committee is the resources to be able to go and
other forms of cost information.
find the information and put it together, assess it
CA We very rarely look at cost utility calculations, to make sure that it’s robust and then deliver it
9 mostly because they’re not available, and to the committee. We don’t have the team to be
they’re quite often very difficult to do. able to do that sort of work.
Occasionally we may get QALY-type stuff, but CD A lot of the models that I’ve seen come out with
it’s usually straight numbers of lives saved per 5 a bottom line cost which is difficult to extract in
pounds spent, or numbers of admissions avoided cash terms. Saving beds is not, at this moment
per pound spent. in time with the pressures on the NHS, going to
CB The big problem is that most of the applications realise any savings at all, because that bed will
2 we get for drugs are at the time they are first be used for the next pressure that’s coming in.
launched and such data really doesn’t exist at CC Quite often this sort of information is available
that point other than guesstimates from the drug 1 from the actual drug companies themselves,
companies. isn’t it? I see that as a weakness with it. Because
CC I don’t think we actually need raw analysis for if a drug company is driving a product there
3 every application, because that’s not necessary may be the risk of bias.
and it’s just extra work for the applicant and for CB To be realistic I almost think the term needs to
ourselves, when the decision could actually be 4 be changed y A lot of clinicians think that
quite easily made around other measures for pharmacy’s all about money, that’s really all
that product. we’re concerned about, and if you say
CD A lot of NICE guidance has an element of ‘economics’ well that’s money really. So it’s
2 health economic data but I think many difficult to actually try and get away from
commissioning managers look at the additional looking at benefits to the patient without tying
financial cost implications of NICE guidance those to a financial element.
rather than the health economics data. So even
when it is available in that local commissioning
Respondents did talk about ways of overcoming
kind of environment it probably isn’t used as
these barriers. The need for a clear, standardised
much as it should be.
and generally accepted format for the presentation
of economic analysis including greater clarification
Using an open-ended question, interviewees were about the assumptions that went into models was
asked to identify the barriers to the use of economic emphasised. A national resource centre and archive
evaluation. The most commonly cited ones related of health economics tools and models that could be
to the accessibility of analyses: cost-effectiveness used was proposed together with information about
analyses were rarely available, especially within the the benefits of using these tools. Training for
strict timeframes in place. The next most commonly committee members, and the co-opting onto the
cited barrier concerned the potential for biased committee of health economics expertise in cases
analyses. This reflected the view that available where this was required were also suggested.
evaluations had most often been conducted by However, these prescriptions—which echo those
manufacturers of the technology under considera- reported in previous studies—do not address all of
tion. Those respondents who had been able to the stated or revealed reasons for the lack of use of
access independent studies reported problems relat- economic evaluation. For example, in each case
ing to the inability to realise, in practice, savings study example there was an absence of considera-
identified in analyses as well as difficulties inter- tion of the opportunity cost of decisions taken.
preting the cost-effectiveness analyses. Other less Committees rarely conducted formal comparison
frequently cited barriers identified included: exces- of new technologies with current practice and
sive variation in health economics methodologies; were rarely if ever in a position to recommend
 ARTICLE IN PRESS
2126 I.P. Williams, S. Bryan / Social Science & Medicine 65 (2007) 2116–2129

disinvestment in existing technologies. A small local levels. However, the review of NTRFs used by
number of interviewees—particularly those involved local decision-making bodies and the case studies of
in the Priorities Network—expressed concern on local decision-making committees show that local
this. decision making focuses primarily on evidence of
clinical benefit and the budgetary implications of
CA If you’re making an investment in something
introduction. Our main research finding at the local
5 that doesn’t come ring fenced with money it
level in the NHS in England is that it is an exception
inevitably, in a fixed budget, means a de facto
for CEA to inform technology coverage decisions.
disinvestment in something else. We’re less clear
Such information was being used in the Priorities
about, as I see it, what we’re disinvesting in.
Network but even here a number of significant
CA No one has yet found a way of taking money out
barriers existed. We have discussed elsewhere the
6 of anything. The whole time I’ve been working
importance of CEA being both accessible and
I’ve never ever found a situation where we’re
acceptable to decision makers if its use is to be
able to say ‘we no longer need to do that.’
increased (Williams & Bryan, 2007). Our data
CC We’re just adding to the formulary. That’s the
suggest that limited use of economic analyses can
7 problem with formularies: people will just carry
be traced back, amongst other things, to lack of
on adding things and adding things until it
capacity in three respects: generating CEAs, acces-
ceases to be a formulary and ends up just being
sing and reviewing CEAs, and interpretation of such
a directory of all the drugs known to man.
analyses. These issues of ‘accessibility’ consistently
Waste of time.
prevented the majority of local decision-making
bodies we researched from making full use of health
This avoidance of the ‘hard choices’ that cost- economics information. With the partial exception
effectiveness analysis is geared towards informing of the Priorities Network, there was very little
was perhaps a further disincentive to its use. capacity to generate and/or access studies, and little
Similarly, although the case study sites routinely formal expertise within the committees to make
requested information on costs associated with new sense of those that were obtained. This resonates
technologies, it is not clear how this was intended to with the previous literature and the reported
be used given that—with the exception of the difficulties faced by those adopting PBMA (Bohmer
Priorities Network who held a ‘NICE implementa- et al., 2001).
tion budget’—they did not directly allocate re- Simon (1957) characterised decision making as
sources and merely advised those charged with ‘problem solving’ and pointed out that it is generally
financial decision making. Despite all having remits carried out more effectively when the ‘problem’ is
that to some extent included the power to recom- well structured. Weiss (1979) echoes this, emphasis-
mend exclusion of, or disinvestment in, technolo- ing the need for a ‘well defined decision situation’.
gies, in practice the committees were more likely to Local committees typically operated in a less
focus on how, as opposed to whether, new treat- defined decision situation than national decision-
ments should be introduced. Decisions frequently making bodies such as NICE (Bryan et al., 2007;
took the form of prescriptions for the development Williams, Bryan & McIver, 2007). A range of
of guidelines, protocols and/or shared care policies considerations—such as the likely impact of deci-
across primary and secondary care. In this context it sions on the local health economy—were ‘brought
is difficult to assess and prescribe the value of to bear’ on their deliberations. Compared to a
decision-making frameworks such as those gener- national body, these committees clearly have con-
ated by economic evaluations. siderably fewer resources to put into the scoping of
each decision and as a result decision options tended
Discussion to be less well defined. This context shaped the
extent to which decision makers considered formal
It seems that for CEA to become an important CEA to be ‘acceptable’. The term ‘acceptability’ as
consideration in technology coverage at local levels employed here refers to the extent to which decision
a range of factors—some specific to health econom- makers were inclined to adopt the recommendations
ics and others pertaining more generally to bodies of studies that they had accessed and interpreted. In
making such decisions—need to be addressed. our research, a necessary requirement for the ‘use’
Clearly, there is substantial variation in practice at of a CE analysis was widely stated to be that it
 ARTICLE IN PRESS
I.P. Williams, S. Bryan / Social Science & Medicine 65 (2007) 2116–2129 2127

addressed the policy question facing the decision within broader structures and processes render them
maker. Frustration was expressed at the failure of unreceptive to cost-effectiveness information.
CE studies to reflect accurately the local policy
context. This criticism was principally directed at Conclusions
the CE analyst. Recommendations focussed on the
need for economic evaluations to take specific Our research supports the view that in order to be
account of local circumstances and of the particular useful, analysts need to take greater account of the
constraints placed on budget holders in these specific circumstances of local decision making. A
settings. In relation to those conducting economic number of features of the decision-making environ-
analyses, it was felt that there was a need for greater ment appear to militate against emphasis on cost-
sensitivity to the specific questions decision makers effectiveness principles. These include separation of
require answers to. However, this aspiration was appraisal and resource allocation functions and
hampered by the lack of clarity surrounding the confusion over the relationship between the two.
functions of the committees and the specification of They also include the extent to which decision
the decision aims and options. making is structured to encourage bargaining
A more specific issue concerns the remit of the between sectional interest groups—for example over
decision-making committee and the extent to which the financial burden of implementation. The study
this takes into consideration budgets and financial has implications for health care systems where
flows. It is important here to ask: similar conditions apply. It further problematises
narrow prescriptions for improving decision making
 Does the decision-making body (and the relevant such as increasing either the volume of economic
committee) have responsibility for a specific evaluations or providing more health economics
budget? training to decision makers. It highlights instead the
 Do parties or individuals making applications to need to address the relationship between the
the committee have to identify a source of attitudes and practices of decision makers and
funding for the proposed new technology in the institutional constraints in which they operate.
advance? There is a corresponding requirement for greater
 Does the committee have clear links to the clarity in both the objectives of technology coverage
financial and resource allocation processes of committees and their responsibilities within a
the organisations represented on it? broader political and financial context. This would
help to establish the terms on which the routine use
Clearly the answer to these questions will shed of cost-effectiveness analysis might become a
some light on the extent to which it makes sense for possibility.
committees to use cost-effectiveness information
routinely in their deliberations. Our research sug- Acknowledgements
gests that local committees in the NHS that are
making decisions on the coverage of drugs and We would like to thank all of the research
other technologies may operate without any re- participants for their time and contributions. We
quirement formally to consider the financial im- would also like to thank Ron Pate for his invaluable
plications of their decisions, and without a direct advice on the design of the research and reviewers
link to a budget that will be drawn on to fund their for their suggested amendments to the manuscript.
decision. This, combined with the information
deficits, appeared to result in committees making References
little attempt to disinvest in or ‘de-prioritise’ existing
Bakst, A. (1995). Pharmacoeconomics and the formulary
technologies. Consideration of cost issues was most decision-making process. Hospital Formulary, 30(1), 42–50.
often confined to a concern amongst committee Blumstein, J. F. (1997). The Oregon experiment: The role of
members that their department or organisation did cost–benefit analysis in the allocation of medicaid funds.
not bear an excessive financial burden resulting Social Science & Medicine, 45(4), 545–554.
Bohmer, P., Pain, C., Watt, A., Abernethy, P., & Sceats, J. (2001).
from introduction of a new treatment. Overall, the
Maximising health gain within available resources in the New
current picture, therefore, appears to be one in Zealand public health system. Health Policy, 55, 37–50.
which information deficits of technology coverage Bryan, S., Williams, I., & McIver, S. (2007). Seeing the NICE side
committees, combined with their specific location of cost-effectiveness analysis: A qualitative investigation of
 ARTICLE IN PRESS
2128 I.P. Williams, S. Bryan / Social Science & Medicine 65 (2007) 2116–2129

the use of CEA in NICE technology appraisals. Health Martin, D., Pater, J., & Singer, P. (2001). Priority-setting
Economics, 16, 179–193. decisions for new cancer drugs: A qualitative case study.
Cotter, S. M., & McKee, M. (1997). Models of hospital drug The Lancet, 358, 1676–1681.
policy in the UK. Journal of Health Services Research and Mays, N., Pope, C., & Popay, J. (2005). Systematically reviewing
Policy, 2(3), 144–153. qualitative and quantitative evidence to inform management
Drummond, M., Cooke, J., & Walley, T. (1997). Economic and policy-making in the health field. Journal of Health
evaluation under managed competition: Evidence from the Services Research and Policy, 10(Suppl. 1), 6–20.
UK. Social Science & Medicine, 45(4), 583–595. McDonald, R. (2002). Using health economics in health services:
Duthie, T., Trueman, P., Chancellor, J., & Diez, L. (1999). Rationing irrationally?. Open University Press.
Research into the use of health economics in decision-making Mitton, C., & Donaldson, C. (2001). Twenty-five years of
in the United Kingdom—Phase II is health economics ‘for programme budgeting and marginal analysis in the health
good or evil’? Health Policy, 46, 143–157. sector. 1974–1999. Journal of Health Services Research and
Fattore, G., & Torbica, A. (2006). Economic evaluation in health Policy, 6(4), 239–248.
care: The point of view of informed physicians. Value in Mitton, C., & Donaldson, C. (2003). Priority setting in a
Health, 9(3), 157–167. Canadian surgical department: A case study using pro-
Gelijns, A. C., Brown, L. D., Magnell, C., Ronchi, E., & gramme budgeting and marginal analysis. Canadian Journal
Moskowitz, A. J. (2005). Evidence, politics and technological of Surgery, 46(1), 23–29.
change. Health Affairs, 24(1), 29–40. Mitton, C., Patten, S., & Donaldson, C. (2004). Listening to the
Grizzle, A. J., Olson, B. M., Motheral, B. R., Armstrong, E. P., decision makers. Sustainability of PBMA in Alberta. Applied
Abarca, J., & Cox, E. R. (2000). Therapeutic value: Who Health Economics and Health Policy, 3(3), 143–151.
decides? Pharmaceutical Executive, 20(11), 84–90. Motheral, B. R., Grizzle, A. J., Armstrong, E. P., Cox, E. R., &
Haslé-Pham, E., Arnould, B., Späth, H., Follet, A., Duru, G., & Fairman, K. (2000). Role of pharmacoeconomics in drug benefit
Marquis, P. (2005). Role of clinical, patient-reported outcome decision-making: Results of a survey. Formulary, 35, 412–421.
and medico-economic studies in the public hospital drug Neumann, P. J. (2005). Using cost-effectiveness analysis to
formulary decision-making process: Results of a European improve health care. Opportunities and barriers. Oxford:
Oxford University Press.
survey. Health Policy, 71, 205–212.
Newdick, C. (2005). Evaluating new health technology in the
Hoffman, C., & von der Schulenburg, J. M. G. (2000). The use of
English national health service. In T. S. Jost (Ed.), Health care
economic evaluation studies in health care decision-making-
coverage determinations. An international comparative study.
summary report. In J. M. G. von der Schulenburg (Ed.), The
New York: Open University Press.
influence of economic evaluation studies on health care decision-
Posnett, J., & Street, A. (1996). Programme budgeting and
making. A European survey (pp. 3–16). Washington, DC:
marginal analysis: A case study of maternity services. Journal
IOS Press.
of Public Health Medicine, 18(2), 175–182.
Hoffmann, C., Stoykova, B. A., Nixon, J., Glanville, J. M.,
Rijkom, J. E. F. van, & Rutten, F. H. (2000). The Dutch results
Misso, K., & Drummond, M. F. (2002). Do health-care
from the EUROMET Project. In J. M. G. von der
decision makers find economic evaluations useful? The
Schulenburg (Ed.), The influence of economic evaluation
findings of focus group research in UK health authorities.
studies on health care decision-making: A European survey
Value in Health, 5(2), 71–78.
(pp. 65–73). Washington, DC: IOS Press.
Iglesias, C. P., Drummond, M. F., & Rovira, J. (2005). Health- Ross, J. (1995). The use of economic evaluation in health care:
care decision-making processes in Latin America: Australian decision makers’ perceptions. Health Policy, 31,
Problems and prospects for the use of economic evaluation. 103–110.
International Journal of Technology Assessment in Health Simon, H. A. (1957). Models of man: social and rational. New
Care, 21(1), 1–14. York: Wiley.
Jenkings, N. K., & Barber, N. (2004). What constitutes evidence Sintonen, H. (2000). The Finnish country report. In J. M. G. von
in hospital new drug decision-making? Social Science & der Schulenburg (Ed.), The influence of economic evaluation
Medicine, 58, 1757–1766. studies on health care decision-making: A European survey (pp.
Lehoux, P., Denis, J., Tailliez, S., & Hivon, M. (2005). 35–41). Washington, DC: IOS Press.
Dissemination of health technology assessments: Identifying Sloan, F. A., Whetten-Goldstein, K., & Wilson, A. (1997).
the visions guiding an evolving policy innovation in Hospital pharmacy decisions, cost containment and the use of
Canada. Journal of Health Politics, Policy and Law, 30(4), cost-effectiveness analysis. Social Science & Medicine, 45(4),
603–641. 523–533.
Luce, R. B., & Brown, R. E. (1995). The use of technology Spath, H. M., Charavel, M., Morelle, M., & Carrere, M. (2003).
assessment by hospitals, health maintenance organizations, and A qualitative approach to the use of economic data in the
third-party payers in the United States. International Journal of selection of medicine for hospital formularies: A French
Technology Assessment in Health Care, 11(1), 79–92. survey. Pharmacy World and Science, 25, 269–275.
Lyles, A., Luce, B. R., & Rentz, A. M. (1997). Managed care Teutsch, S. M., & Berger, M. L. (2005). Evidence synthesis and
pharmacy, socioeconomic assessments and drug adoption evidence-based decision-making: Related but distinct pro-
decisions. Social Science & Medicine, 45(4), 511–521. cesses. Medical Decision Making, September–October
Madden, S., Martin, D. K., Downey, S., & Singer, P. A. 2005.
(2005). Hospital priority setting with an appeals process: A von der Schulenburg, J. M. G. (2000). The influence of economic
qualitative case study and evaluation. Health Policy, 73, evaluation studies on health care decision-making. Oxford:
10–20. IOS Press.
 ARTICLE IN PRESS
I.P. Williams, S. Bryan / Social Science & Medicine 65 (2007) 2116–2129 2129

Wade, E., Smith, J., Peck, E., & Freeman (2006). Commissioning Williams, I., & Bryan, S. (2007). Understanding the limited
in the reformed NHS: Policy into practice. NHS Alliance/ impact of economic evaluation in health care resource
HSMC: University of Birmingham. allocation: A conceptual framework. Health Policy, 80,
Walley, T., Barton, S., Cooke, J., & Drummond, M. (1997). 135–143.
Economic evaluations of drug therapy: Attitudes of primary care Williams, I., Bryan, S., & McIver, S. (2007). Health technology
prescribing advisors in Great Britain. Health Policy, 41, 61–62. coverage decisions: Evidence from the N. I.C.E. ‘Experiment’
Weiss, C. H. (1979). The many meanings of research utilization. in the use of cost-effectiveness analysis. Journal of Health
Public Administration Review, 426–431. Services Research and Policy,, 12(2), 73–79.