See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/277298687

AIMS AND OBJECTIVES OF E-GOVERNMENT AND E-GOVERNANCE....6

Article · January 2002

CITATIONS READS
0 2,028

1 author:

Tony Bovaird
University of Birmingham
151 PUBLICATIONS 3,278 CITATIONS

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Resource contributions by citizens to co-production of public services and public value View project

Pricing policy for public services and local government View project

All content following this page was uploaded by Tony Bovaird on 26 December 2015.

The user has requested enhancement of the downloaded file.

OECD E-GOVERNMENT PROJECT

SEMINAR-VISION, RESPONSIVENESS AND MEASUREMENT

OECD, PARIS, MARCH 11-12, 2002.

Session 2: E-government Measurement and Evaluation

Draft Paper

Performance Measurement and Evaluation of e-Government

and e-Governance Programmes and Initiatives

Tony Bovaird

Bristol Business School

University of the West of England, UK
February 2002
 2

Performance Measurement and Evaluation of e-Government

and e-Governance Programmes and Initiatives

CONTENTS

INTRODUCTION........................................................................................................3
AIMS OF THIS PAPER ..............................................................................................4
ROLE OF THIS PAPER IN THE PROJECT ..........................................................5
METHODOLOGY ......................................................................................................6
AIMS AND OBJECTIVES OF E-GOVERNMENT AND E-GOVERNANCE ....6
KEY ISSUES IN PERFORMANCE MEASUREMENT AND EVALUATION
OF E-GOVERNMENT AND E-GOVERNANCE..................................................18
PERFORMANCE INDICATORS FOR E-GOVERNMENT AND E-
GOVERNANCE.........................................................................................................23
EVALUATION OF E-GOVERNMENT PROGRAMMES AND INITIATIVES
......................................................................................................................................29
CONCLUSIONS ........................................................................................................33
RECOMMENDATIONS...........................................................................................34
REFERENCES ...........................................................................................................36
APPENDIX 1. EXAMPLES OF PIs IN DIFFERENT COUNTRIES ........ Error!
Bookmark not defined.
APPENDIX 2. THE OVERALL LIBRARY OF POTENTIAL PIS FOR E-
GOVERNMENT AND E-GOVERNANCE ............................................................38
 3

Performance Measurement and Evaluation of e-Government

and e-Governance Programmes and Initiatives

INTRODUCTION

This paper was commissioned by the Public Management Service [PUMA] of the
OECD as part of its e-Government Task Force initiative. This initiative takes as its
starting point that e-government has the potential to be a major enabler in the adoption
of good governance practices, with a major focus on the longer-term vision (2005-
2010). The overall project will produce papers, policy briefs, and specific reports,
which will be considered at a series of seminars, and a Flagship report will be
finalised by the end of 2002.

The topic for this paper, one of the first to be commissioned in the process, is the need
for performance measurement and evaluation in e- government and e- governance. It is
intended to help in the design of an evaluative framework which will have a number
of complementary purposes:

?? to clarify what works and what does not work

?? to provide evidence for strategic choices and investments

?? to highlight critical success factors in implementation

?? to highlight possible side-effects and unintended consequences

The paper begins by setting out its aims, purpose and the methodology which has
been used. It then examines the aims and objectives of e-government and e-
government as highlighted by a number of different international bodies and
governments of OECD member countries. The paper then explores some key issues
in performance measurement of e-government and e-governance, and the options for
performance indicators for e-government and e-governance. It goes on to consider the
scope for evaluation of e- government programmes and initiatives, and possible
frameworks by which such evaluation might be undertaken. Finally, it sets out some
draft conclusions and recommendations, for consideration at the first OECD seminar
in March 2002.
 4

AIMS OF THIS PAPER

In line with these intentions, the aims of this paper are:

?? To develop a conceptual approach for the measurement and evaluation of

e-government and e-governance, while taking account of the differing
context in OECD Member Countries.

?? To identify key themes and issues in e-government and e-governance and

to suggest how they might be tackled.

?? To analyse the issues, with a focus on identifying potential and existing

solutions and approaches.

?? To identify a library of potential performance indicators for e- government

and e- governance, which are likely to be interesting to different key
stakeholders in the public domain.

?? To identify key information sources for performance information, current

gaps and approaches which might rectify these gaps.

?? To identify appropriate approaches for the evaluation of e-government and

e-governance programmes.

?? To make recommendations for priority areas in the measurement of the

performance of e- government and e- governance and the evaluation of e-
government and e-governance programmes.
 5

ROLE OF THIS PAPER IN THE PROJECT

The purpose of this paper is to contribute to the overall project in the following ways:

?? To form a basis of discussion at the OECD project seminar in March 2002

?? To form a basis for dialogue with OECD Member Countries and outside
experts

?? To act, as appropriate, as a basis to commission further research

?? To act as an input into the analysis of issues by the OECD E-Government

Task Force and thus contribute to the final report of the E-Government
project

The paper has therefore been written with these multiple audiences in mind. The
methodology used respects the need to incorporate the views of multiple stakeholders
in the performance assessment and evaluation of e- government and e- government
programmes.

Moreover, the final recommendations are broken down by stakeholder group, in order
to facilitate exchange of views between stakeholders on what success looks like in
this area of activity and how these programmes might be made more successful,
within the perspectives of the different stakeholders involved.
 6

METHODOLOGY

This paper has used three different sets of source materials:

?? OECD documents on e-government and e-governance and on other related

programmes and projects

?? Government and public sector agency websites on the internet

?? Academic and trade press literature (sometimes in print, otherwise on the internet)

Where material from these sources is used, they are cited in the text and full
references are given in the References section at the end of the paper. In most cases,
these are references to websites.

In compiling the library of possible performance indicators for e-government and e-

governance, the framework used was brought together from a number of sources,
including:

?? The good governance principles set out in the OECD document Project on the
Impact of E-Government .

?? The 2001 OECD publication, Citizens as Partners; Information, Consultation and

Public Participation in Policy-Making, discusses citizen engagement in policy
making, provides a list of gene ral principles for successful information provision,
consultation and participation, and describes practices undertaken by some OECD
Member Countries.

?? The 2001 OECD publication, The Hidden Threat to E-Government: Avoiding

Large Government IT Failures, which is based on country reports and experiences
of participants, presented at a meeting in October 2000.

?? The forthcoming OECD publication, From In-Line to On -Line, which describes

key service delivery objectives followed by most OECD Members countries.

?? Reports emerging from the overall OECD E-Government project, including

research papers dealing with the priority research topics of leadership, financing ,
relationships and skills (under ‘Strategic Implementation of E-Government’)
where there is a particular need to develop indicators.

?? Office the e-Envoy (UK Cabinet Office) report on Guidelines for Preparing
‘Implementing Electronic Government’ Statements., listed in the References.
 7

AIMS AND OBJECTIVES OF E-GOVERNMENT AND E-GOVERNANCE

There is no single source which can be used as a definitive statement of the meaning
and scope of the terms ‘e- government’ and ‘e-governance’, and the aims and
objectives held member governments in OECD countries in relation to e- government
and e- governance.

Definition of ‘e-government’

The OECD states that “the term e-government focuses on the use of new information
and communication technologies (ICTs) by governments as applied to the full range
of government functions. In particular, the networking potential offered by the
Internet and related technologies has the potential to transform the structures and
operation of government” (PUMA, 2001).

The World Bank suggests that : “e-government refers to the use by government
agencies of information technologies (such as Wide Area Networks, the Internet, and
mobile computing) that have the ability to transform relations with citizens,
businesses, and other arms of government. These technologies can serve a variety of
different ends: better delivery of government services to citizens, improved
interactions with business and industry, citizen empowerment through access to
information, or more efficient government management. The resulting benefits can be
less corruption, increased transparency, greater convenience, revenue growth, and/or
cost reductions” (www1.worldbank.org/publicsector/egov/definition.htm).

These definitions from OECD and the World Bank have strong parallels with the
definitions adopted by some individual OECD member governments. The Cabinet
Office in the UK, for example, suggests that “e- government … focuses on better
services for citizens and businesses and more effective use of the Government’s
information resources. Implementing it will create an environment for the
transformation of government activities by the application of e-business methods
throughout the public sector” (Cabinet Office, 2001).

The UK also makes it clear that telephone transactions may be included in ‘e-
government’, if certain conditions are met. Specifically, it suggests that ‘electronic’
service delivery means ‘delivery through internet protocols and other ICT methods
and includes delivery by telephone if the transaction carried out is electronically
enabled, i.e. the officer receiving the call can access electronic information and/or
update records on- line there and then” (DTLR, 2002; text explaining BV 157).

The New Zealand government suggests that “e-government is a way for governments
to use the new technologies to provide people with more convenient access to
government information and services, to improve the quality of the services and to
provide greater opportunities to participate in our democratic institutions and
processes” (http://www.e-government.govt.nz/evision/index.html).
 8

The Italian government uses the term ‘e-government’ to refer to the use of modern
ICTs in the processes of modernising the administration of the state and suggests that
it comprises the following categories of activity
(www.pianoegov.it/UserFiles/367.zip):

1. direct provision of information for improving the internal operating efficiency

of administrative units;
2. activities which lead directly to the informatising the delivery of services to
citizens and companies, which often implies the integration of services
provided by several administrative units;
3. activities which lead directly to providing end users with electronic access to
public services and to all relevant information about them.

Definition of ‘e-governance’

E-governance, however, is a term which is used much less often and for which there
are fewer definitions. This is rather odd, given that the topic of governance has been
very topical for almost a decade and many OECD governments have incorporated
governance issues in their reform programmes (e.g. the ‘activating state’ in Germany
and the ‘modernising government’ programme in the UK).

Richard Heeks proposes that the term ‘e-governance’ should be seen to encompass all
ICTs, but the key innovation is that of computer networks – from intranets to the
internet – which have created a wealth of new digital connections (Heeks, 2001; p. 2):
?? Connections within government – permitting 'joined-up thinking'.

?? Connections between government and NGOs/citizens – strengthening

accountability.

?? Connections between government and business/citizens – transforming service

delivery.

?? Connections within and between NGOs – supporting learning and concerted

action.

?? Connections within and between communities – building social and economic

development.

As a result, Heeks suggests, the focus of e- governance shifts from just parts of e-
administration, in the case of e-government, to also encompass e-citizens, e-services
and e-society.

The joint UNESCO-COMNET-IT study of e-governance (UNESCO, 2002) defines

governance as “the process by which society steers itself”. It goes on: “in this process,
the interactions between the State, Private Enterprise and Civil Society are being
increasingly conditioned and modified through the influence of ICTs. Examples of
these shifts in dynamics are exemplified by:
 9

?? the use of the Internet by Civil Society, NGOs and professional associations to
mobilise opinion and influence decision- making processes that affect them

?? the increasing electronic delivery of Government and commercial services and

information

?? the electronic publication of draft legislation and statements of direction for

public feedback

?? on the infrastructure side, the increased adoption of e-enabled community

centres, the liberalisation of telecomms markets and trends towards web-
enabled mobile telephony and digital television are facilitating this evolution”.

Jim Melitski (http://www.aspanet.org/solutions/egovworld.html) describes the “e-

government journey” as a continuum which begins with information provision when
organizations and public agencies publish static information to the Internet, but then
moves on as public organizations become more advanced and “are able to provide
more dynamic, transactional services. Ultimately the continuum leads to
organizational transformation, the transparency of public agencies, increased citizen
participation in government, and facilitation of democratic processes”. E-government
is at one end of this continuum, while e-governance is at the other, but it is not easy,
nor really worthwhile, to distinguish where exactly is the dividing line between these
concepts.

However, in order to keep the distinction as clear as possible, in this paper the term
‘e-government’ is essentially restricted to the electronic enablement of services (both
to the external stakeholders and to internal customers), while e-governance refers to
non-service specific activities of government and public agencies.

Aims and objectives of e-government and e-governance

The World Bank analyses the potential effects of e- government under the headings
(www1.worldbank.org/publicsector/egov/index.htm):

• Better service delivery to citizens

• Improved services for business
• Transparency and anti-corruption
• Empowerment through information
• Efficient government purchasing

As can be seen, the first third and fourth of these goals belong in the realm of ‘e-
governance’, as interpreted in this paper.

The eEurope project has the following goals (European Commission, 2002):
 10

“The overarching goal of eEurope is to connect Europe as fast as possible to

the net. In order to do reach this goal, the action plan foresees initiatives in
three areas:

i. a cheaper, faster and more secure internet

ii. investments in people and capabilities
iii. encouragement of the use of the internet

These goals and objectives clearly relate to the putting in place of infrastructure and
the critical success factors which are likely to contribute to the eventual impact of the
internet, rather than to the final impacts and outcomesof e-government and e-
governance.

At country level, again there is significant heterogeneity in the goals identified. The
UK government is committed to the three overarching objectives of the UK online
programme (http://www.e-envoy.gov.uk/ukonline/progress/anrep2001/01.htm) :

?? to make the UK the best and safest environment in the world for e-commerce
by 2002;
?? to ensure that everyone who wants it has access to the Internet by 2005; and
?? to make all Government services available electronically by 2005.

Again, these are instrumental objectives, cast in terms of putting in place a functional
infrastructure.

The Canadian government suggests, very much in line with the definitions used in this
paper, that e-government (Government On-Line) can provide (http://www.gol-
ged.gc.ca/rpt/gol- ged-rpt02_e.asp) :

- “Better service – through more convenient, faster access to information and

services
- Reduced costs for individuals and businesses – less search time, less time in
filling out forms, faster decisions
- Reduced costs for government – reduced data entry costs, lower error rates”

The Canadian Government set itself the following targets for the end of 2000
(http://www.comnet.mt/Unesco/CountryProfiles/Project/canada.htm), again primarily
in the realm of e-government, although the final target is more in line with public
governance concerns:

?? Up-to-date, accurate, bilingual information on key programs and

services available on- line.

?? Commonly-used forms available to download and print.

?? The ability to contact departments through the Canada Site.

 11

?? The Canada Site will continue to be revamped and organized around

citizen needs and topics of interest. A technology and policy
framework will be in place that protects the security and privacy of
Canadians in their electronic dealings with government.

Moreover, and perhaps more surprisingly, the targets set for the next few years were
also specified in terms of service-oriented following deliverables:

?? “Key federal programs and services – the ones that matter most to
Canadians – will be available on- line ... [including] … secure and
interactive transactions … [and] …electronic forms.

?? “Technical and content support will be provided through various help

services … [with] … published service standards.

?? “An easy to use, advanced search capability will be available on the

Government of Canada portal and all federal department and agency
web sites.

?? “One-stop access points (or portals) [will be] available through the
Canada Site, with information and services organized according to
types of activity, areas of interest and common citizen needs. Plans are
already underway to develop portals for seniors, consumers,
Aboriginals, the environment, and innovation resources for small- and
medium-sized enterprises.

?? Innovative partnerships. The Government On-Line initiative will place

increased emphasis on on- line service delivery partnerships with
provinces, territories, municipalities, businesses, volunteer
organizations and international partners.

While essentially remaining e-government oriented, the fourth of these points makes
it clear that Canada will develop its e-services with a strong stakeholder-orientation,
something which is missing from most other government strategies.

In Germany, the Bundesamt für Sicherheit in der Informationstechnik (BSI) states that
“Electronic government refers to the use of the internet and other electronic media to
engage citizens and enterprises in the activities of government, and to enhance
collaborative working within pub lic administration” (BSI, 2002). It continues “ The
core goal of e-government is establishment of a ‘digital administration’, which offers
on- line access to information, communications, services and participation
opportunities – in so far as they are possible and legally available – in a way which is
tailored to the needs of citizens and business, i.e. the requirements of the
administration’s customers” (BSI, 2002). Clearly, the final sections of these goals
begin to stray into e-governance territory.
 12

As highlighted earlier, the New Zealand vision of E- government included the

aspiration “to provide greater opportunities to participate in our democratic
institutions and processes”, which is essentially an e- governance task. The New
Zealand government is committed to the mission that by 2004 the internet will be the
dominant means of enabling ready access to government information, services and
processes (http://www.e-government.govt.nz/programme/dec01-exec/exec-
dec01.pdf). It suggests that the benefits of e-government will include (http://www.e-
government.govt.nz/evision/index.html) :

?? It will be easier for people to have their say in government.

?? People will get better services from government organisations (and

governments will have lower costs).

?? People will receive more integrated services because different government

organisations will be able to communicate more effectively with each other.

?? People will be better informed because they can get up-to-date and
comprehensive information about government laws, regulations, policies and
services.

However, going beyond this services-oriented approach, the NZ government suggests

that “The e-government vision supports two important goals. They are:

?? Restoring trust in government and providing strong social services. The e-

government vision will play an important role in achieving this goal. It will:
o increase collaboration between government organisations;

o strengthen the relationship between people and the state through

greater opportunities for participation; and

o provide the state sector with an opportunity to improve the

effectiveness and efficiency of their services to the public while, at the
same time, reducing the cost of delivery.
Those three factors, it suggests, will help restore trust in government and
provide strong social services.

?? Helping grow an inclusive, innovative economy for the benefit of all. The e-
government vision is all about inclusion – the ability of all people to take part
in our economy.”

How will New Zealanders know that e- government in 2004 is delivering the right
results for them? The Government has identified three broad characteristics
(http://www.e- government.govt.nz/programme/dec01-main/chapter4.html)
that mark out successful e- government:
 13

??Convenience and Satisfaction: Services provided anytime, anyhow,

anywhere. People will have a choice of channels to government information
and services that are convenient, easy to use and deliver what is wanted.

??Integration and Efficiency: Services that are integrated, customer-centric and

efficient. Information and services will be integrated, packaged and presented
to minimise cost for people, businesses and departments.

??Participation: Participation in government. People will be better informed

and better able to participate by having easier access to governme nt
information and processes.

Finally, we incorporate into later sections of the paper two of the potential benefits of
e-government and e-governance which are often mentioned by practictioners,
although they do not feature very widely either on government websites or in the
academic literature:

??increasing the transparency of public processes (e.g. by allowing citizen access

to information on the stage reached in documentation processing or decision-
taking);

??highlighting the incidences (or possible inc idences) of fraud or corruption

(within the ethics infrastructure)

Aims of other stakeholders

So far, this paper has focused on the aims and goals for e-government and e-
governance which have been highlighted by governments. Yet it is clear that other
stakeholders may have other priorities. At the very least we would expect some
diffrentiation in the goals and the priorities expressed by other stakeholders such as:

??Parliaments
??NGOs
??civil society associations
??local authorities

The analysis which is to be found in government and public sector documents about
e-government and e-governance tend to talk mainly about stakeholder requirements in
very general terms – they refer to the ‘public’ or to ‘people’. For example, in the
German context, the BSI (2002) suggests that “customers of the administration”
expect on- line offers which allow them to use public services quickly and in an
uncomplicated way, to pursue appropriate processes simply and to understand the
bases of decisions which have been made – i.e. they expect service and transparency.

Consequently, governments have not been stakeholder specific in their declared aims
and priorities in e- government and e- governance. Indeed, they have often not even
 14

distinguished between citizens and business, and there have been very few attempts to
target programmes at civil society through the non-profit sector. The role of the media
has also been understated and probably underestimated.

One partial exception to this tendency has been the World Bank, which makes an
analogy to e-commerce, where the distinction is often made between improvements
which allow businesses to transact with each other more efficiently (B2B) and those
which bring customers closer to businesses (B2C). Thus, the World Bank suggests
(www1.worldbank.org/publicsector/egov/definition.htm) that e-government aims to
make the interaction between government and citizens (G2C), government and
business enterprises (G2B), and inter-agency relationships (G2G) more friendly,
convenient, transparent, and inexpensive. However, once again there is little attempt
to differentiate between the goals and the requirements of these different stakeholders.

Constructing stakeholder-specific hierarchies of objectives for e-government and

e-governance

One of the areas of public administration which has typically been rather weak is the
construction of organisational objectives, which have often been bland, imprecise,
ambiguous, complex and inter-acting, unmeasurable and non-prioritised.

This is certainly the case in relation to the opportunities offered by the digital
revolution. Few of the documents on e-government and e-governance which have
been published by member governments to date have provided clarity on ojectives
and the priorities between them. Moreover, as argued in the previous section,
governments have not been stakeholder specific in their declared aims and priorities
in e-government and e-governance, often assuming that all stakeholder will perceive
and prioritise the same benefits (and disbenefits).

The basic idea of ‘management by objectives’ (MbO), as proposed by Drucker, was

extended by Ansoff (1969) to propose that the top level objectives in an organisation
could be cascaded down through the layers of the organisation. In this approach, the
objectives at the top of the hierarchy of objectives indicate the impacts which the
organisation wishes to have on users and other stakeholders, including society as a
whole. The achievement of these ‘impact’ objectives should be linked clearly to the
achievement of the next level of objectives, (‘service level’ objectives) which are
instrumental and essentially show the organisational outputs which are necessary if
the ‘impact’ objectives are to be achieved. Finally, the lowest level objectives
(‘logistical’ objectives) should be pictured, the achievement of which will enable the
service level objectives to be achieved.

The greatest innovation of the hierarchy of objectives was not that it allowed the
interactions between objectives to be modelled clearly, or that it highlighted clearly
the potential conflicts between some objectives, or that it encouraged performance
measurement at all levels of the hierarchy of objectives (and showed how measuring
performance at the lowest level of objectives might act as a proxy for measuring
performance at the highest level, if this was problematic). Rather it was that it
presented the map of inter-related objectives as a framework of hypotheses about
 15

'cause and effect' chains in the organisation (Bovaird, 2001). Since these hypotheses
could be contested (and often were in practice), this approach stimulated managers
and professionals to find logical arguments and evidence for their view of how the
hierarchy of objectives held together. Essentially, this approach encourage d and
embodied an ‘evidence-based management’ approach, a long time before that became
fashionable.

Furthermore, by presenting a model of the hypothesised links between aspirations, the

hierarchy of objectives allows stakeholders to contest the views of ‘reality’ held by
other stakeholders. The logic of the hierarchy of objectives is that each stakeholder
group should construct and fight for its own vision of the appropriate corporate
aspirations. Furthermore, it should attempt to gain acceptance for the ways in which
the organisation might hope to realise these aspirations, i.e. the rest of the cause-and-
effect chain which it hypothesises for the organisation.

This results in a set of ‘hierarchies of objectives’ within any organisation, rather than
just one. This appears much messier than the elegant simplicity of Ansoff’s original
idea, never mind the simple lists envisaged by Drucker.

What is the relationship between these stakeholders’ hierarchies of objectives? There

are three major ways in which they may be expected to differ:

?? differences in values held by stakeholders may lead to different objectives being

accepted as aspirations - this is especially likely to mean that stakeholder groups
may place different objectives at or near the top of their hierarchies of objectives;

?? differences in values or in interests may lead stakeholder groups to put different

priorities on top objectives and the pathways which lead down from them:

?? differences in experience or in logical reasoning may lead stakeholders to different

models of the cause-and-effect chains linking different subobjectives in the
hierarchy of objectives.

This means that a multiple stakeholder approach does not simply mean the overlay of
different stakeholder maps, each containing a sub-set of the overall hierarchy of
objectives. In practice, stakeholder maps may differ in the objectives they contain and
the logical cause-and-effect chains which are modelled within them.

Moreover, there are an infinity of maps which might be drawn by any one stakeholder
to analyse different problems or different issues - a stakeholder may wish to examine
the objective map for one its functions rather than for one of its products or one of its
clients - and each map would present the opportunity to envisage a different
formulation of what the organisation is trying to achieve.

This leads us to the conclusion that the search for ‘organisational objectives’ has been
fool’s gold twice over:

?? the organisation does not have objectives, only its stakeholders have objectives - so
that any attempt to state ‘organisational objectives’ is at base an attempt to impose
the objectives believed by the dominant stakeholder group or coalition to be the
 16

most advantageous public expression of its own interests

?? each stakeholder group or coalition is likely to need a different map of its

objectives, depending on the problem to be solved or the issue to be addressed

With this insight, we can then see each stakeholder’s set of objectives as a set of
windows into its ‘underlying world’ of objectives. In order to illustrate this, we will
use a simple hierarchy of objectives relating to one aspect of e-governance – the
concern of citizens about the security of internet transactions.

In Figure 1, we see in the background window (A) a set of objectives for reassuring
the public about the level of internet security. It contains objectives to deter fraud
(partially through prevention measures, partly through detection, and partly through
‘high visibility policing’. It also contains a different ‘pathway’ of objectives, focusing
around informing the public about the actual, as opposed to the feared, level of
internet fraud. The overall set of objectives set out in window A might, for example,
be held by a government agency trying to model decide the best policies by which the
public might be reassured to have greater confidence in internet security. One set of
this ‘window’ has been highlighted in a separate window (B) - this set of objectives
relates only to sub-objectives which attempt to inform the public about how over-
inflated is the common view about risks attached to internet transactions. This
narrower ‘window’ of sub-objectives might be held by public agency which is trying
to convince its users to make their payments over the internet rather than at cashier’s
offices (e.g. the tax collection service).

Of course, other stakeholders will have different sets of objectives. In Figure 1, a new
set of objectives has been highlighted in ‘window’ C, relating to a different
stakeholder. This might, for example, be the set of objectives held by the legal
services department in the public agency, which employs an internet monitoring
agency to police internet transactions in order to detect attempted attempted frauds. In
this example, detection is the key objective which the legal services department
believes should be pursued in order to reassure the public about internet security. This
stakeholder group does not see ‘high visibility policing of all internet transactions’ as
a high level sub-objective in its own right, but only as a means of detecting fraudulent
activity. Clearly, there is a dispute here about the empirical relationships which exist
in the specific environment in which the map is created. And this argument really
matters – the two versions of this map, at the heart of the dispute between the
stakeholders involved, lead to very different strategies for reassuring the public. A
‘high visibility policing’ policy which is given sufficient resources to actually detect
significant le vels of fraud is likely to require very large resourcing, whereas a ‘high
visibility policing’ policy which is essentially meant to scare off potential fraudsters
might be undertaken quite cheaply. Until these arguments can be resolved, major
resource wastage is likely to occur.

Clearly this example has been made simple for illustrative purposes. However, we
suggest it is sufficiently realistic to demonstrate the point that it would be unwise to
take at face value any set of ‘organisational objectives which purport to represent all
stakeholders and to aid decision making in all contexts.
Figure 1. Objective maps from three stakeholders
To increase the public’s feeling of internet security
A
To deter internet fraud To increase public’s knowledge
of actual levels of internet fraud

To use media
To prevent fraud To provide high
visibility policing of To use credit card
internet transactions payment bills to inform
To detect fraud
people
To use portal
advertising slots

B
C
KEY ISSUES IN PERFORMANCE MEASUREMENT AND EVALUATION
OF E-GOVERNMENT AND E-GOVERNANCE

In this section, we outline a number of key issues which need to be tackled in

developing appropriate frameworks for measurement and evaluation of e-government
and e- governance:

• What criteria can governments develop to assess trade-offs in policy making,

for example between efficiency and improved participation and
accountability?
• How can the benefits to recipients of services, such as portals, be quantified?
• How to measure quality of services, or consultative processes, and their
enhancement using e- government?
• Can progress in e-government be assessed? What are the key indicators? Can
we use these indicators at both micro and macro levels?
• How can the whole of governmental e-strategies be evaluated?

What criteria can governments develop to assess trade-offs in e-government and e-

governance?

Clearly, there are a number of difficult choices facing governments in deciding their
priorities in e- government and e- governance. For example, increased efficiency might
be gained if more communications could be made available only through telephone
and internet connections. However, this could disadvantage those people without
telephone or internet access. Again, e-governance mechanisms such as e-referenda
could improve the interactive participation of citizens and other stakeholders in
government decision- making – but it could also slow down decision- making
processes and could give rise to unrepresentative pressures being placed on
governments and parliaments if security systems are not sufficiently watertight to
eliminate bogus or multiple voting.

In order to deal with this potential problem, governments will need to consider ways
of setting priorities between the goals of e-government and e-governance. However,
this will be problematic, both because the explicit setting of priorities is always
difficult in a political setting and also because the goals of e-government and e-
governance are so closely inter-related.

To make these trade-offs systematically will therefore require that the full portfolio of
effects of e-government and e-governance programmes are be borne in mind in the
decision- making process. It will be dangerous to focus on only a limited number of
goals or targets at any one time, as the knock-on effects of some achievements will be
so significant.

It would also be valuable if more detailed modelling were done into the cause-and-
effect chains linking high level aims to low level sub-objectives in both e-government
and e- governance. The absence of an agreed framework for considering these cause-
 19

and-effect chains – and for carrying out rigorous tests into their credibility –
magnifies the risk involved in having to make a choice between different ‘pathways’
in the hierarchy of objectives.

How can the benefits to particular recipients of intermediate technical services be

quantified?

Many of the key steps in e-government and e-governance are intermediate outputs
(for example, reducing fraud by enabling system-wide checks of behaviour by welfare
claimants, or increasing public confidence in financial transactions on the internet by
establishing secure systems for electronic signatures). It is often in the nature of such
intermediate outputs that they are not well known to nor understood by those people
who benefit from them – indeed, many of the services which produce these
intermedia te outputs may not even be noticed by many users.

In such cases, there are a limited number of ways in which these ‘intermediate
outputs’ can be evaluated:

a. they can be evaluated by professional peer groups who are aware of their of
their purpose – an independent qualitative judgement process;
b. they can be evaluated through benchmarking of their characteristics by
professional peer groups – an independent objective judgement process;
c. they can be evaluated on the basis of the overall impacts of the package of
outputs of which they form a part, without seeking to separate out the
contribution of individual components of the package – a ‘joint products’
evaluation, based on outcome measures;
d. they can be evaluated by testing against a ‘theory of change’ which sets out
the ways in which they are hypothesised to work, so that each link in the chain
of argument can be tested.

None of these approaches is entirely convincing by itself and none will be entirely
easy to undertake. In some cases, even the simply of these approaches can be costly
and can take time.

Consequently, where it is important to undertake evaluations which separate out the

contributions of ‘intermediate outputs’ from ‘final outputs, several of these
approaches may need to be combined and the eva luation may take a considerable
period of time to come up with significant results.

How to measure the quality of services, or the quality of consultative processes, and
their enhancement using e-government?

Since one of the major drivers behind e-government is improvement to the quality of
services, it would clearly be advantageous to be able to measure the extent to which e-
government has increased the quality of public services provided by organisations in
the public sector and other providers to which services have been outsourced.
 20

However, there is a significant gap in quality management systems in the public

sector in most OECD member countries. While quality assurance systems have
become very common, and they have helped to stop the occurrence of sudden
‘glitches’ in quality levels, the measurement of actual quality levels is much less
advanced. Most performance measurement systems still focus primarily on inputs and
outputs (and, to a lesser degree, outcomes) which can easily be quantified, and most
of these measures throw little light on the quality of services.

Although this is being partly addressed by some initiatives, such as the ‘results’
sections of the EFQM Excellence Model, and the customer satisfaction surveys which
have been built into the performance management framework for local government in
the UK, these approaches are still relatively young and untried. Consequently, the
determination of the success of e- government in improving the quality of public
services will face major measurement challenges.

In order to take this challenge further, it is necessary for an organisation or a

stakeholder to decide on the definition of quality which is going to be used. As shown
below in Figure 2, there are interesting and potentially valuable methods of measuring
quality, whatever the definition chosen. However, without taking this first (and
admittedly difficult) step of defining the concept of quality, it is unlikely that any
clarity will be achieved in assessing its level and its direction of change.

Figure 2. Measuring ‘quality’

Definition of ‘quality’ Potential measurement method

‘Conformance to specification’ 1. Level of defects (‘failure rate’).

(engineering and ‘contract culture’ 2. Full conformance to all provisions
approaches of specification or not (‘yes-no’
certification)
‘Fitness for purpose’ 1. Scoring against each ‘purpose’ or
(systems analysis approach) ‘objective
2. Aggregate score against weighted
set of ‘purposes’ or ‘objectives’
‘Meeting customer expectations’ 1. Checking against checklist of all
(consumer psycho logy approach) stakeholder expectations
2. Weighted score against all
stakeholder expectations
‘Producing passionate emotional 1. Questionnaire to stakeholders to
involvement in the stakeholder’ see if they identify with the
(social psychology approach) service
2. Exploration with stakeholder to
see if they would be willing to
contribute (time, money,
reputation) to the service, as
evidence of their emotional
involvement
 21

Can progress in e-government and e-governance be assessed? What are the key
indicators? Can we use these indicators at both micro and macro levels?

The assessment of progress in e-government and e- governance requires some

measures of performance which can command credibility in the eyes of the major
stakeholders who need to know whether the programmes are working.

This has been put forcibly by Di Maio: "While e- government has sparked countless
best-practice exchange and comparison initiatives, which are useful in accelerating
efforts in regions that are lagging behind, they risk missing the real point. Initiatives
must be based on quantifying value and cost for constituents, as well as governments,
and address service-delivery targets alongside the other aspects of e- government
transformation. Progress must be measured against national needs, not on the basis of
what other countries think and do." (Di Maio, cited in Gartner, 2001 –
www.Gartner.com).

Clearly, this assessment process will more powerful and more convincing if there is
some link between the measures used to assess the success of the overall programmes
and the measures used to assess the success of individual initiatives within the
programme. In fact, there are two competing tendencies here:

??It is easier to measure the intermediate outputs of relatively small initiatives,

since they are easier to aggregate than the large range of intermediate outputs
which emerge from a national programme of e- government or e- governance.

??However, if we want to measure outcomes, then it is usually easier to do this

at programme level, because at the level of projects or individual initiatives, it
is very difficult to separate out the effect of individual projects on outcomes
which have been brought about by a package of projects and initiatives.

Consequently, when we see to measure the success of e-government and e-

governance programmes, it is likely to be valuable to pursue both the measurement of
intermediate outputs and of outcomes, and to consider the relationship between them.

How can the whole of governmental e- strategies be evaluated?

Following on from the previous issue, there is a need to evaluate not only the
outcomes of e-strategies at governmental level, but also whether the strategies chosen
have been well calculated to achieve their goals. This is because it is not simply
important to know the end or ‘summative’ results of a strategy. It is usually essential
to know if the strategy could have been made more successful by changes which
might have been brought about by the policymakers and managers concerned, so that
‘formative’ conclusions can be reached which can allow policy learning to take place.

This requires a ‘theory of change’, which explains in a logical and systematic way the
cause-and-effect chain by means of which the initiatives within the e-programme have
brought about the outcomes desired by the government.
 22

Such a ‘theory of change’ needs a clear statement of the elements of the ‘rational
decision- making model’ – the problems which were addressed, the objectives
formulated in relation to that problem area, the options generated, the evaluation
criteria used, the process of selecting an option, the implementation process used for
the chosen strategy, and the monitoring and review processes which are used to check
if the chosen strategy is working.

Once such a ‘theory of change’ has been elaborated, all its elements can be tested
against the evidence, including its assumptions, its problem specification, the
objectives which it proposes for the programme, the alternative solutions which it
generates, the evaluation criteria and the implementation and monitoring regimes.
Without such a theory of change, it is very difficult to undertake a practical evaluation
over any period of time, because of the likelihood that so much will change during the
period of the evaluation. The ‘theory of change’ model allows the evaluator to gauge
whether the government policymakers were able to respond to these changing
external factors in ways which allowed them to achieve the key results in which they
were interested.

Of course, in practice governments often do not have clearly mapped out strategies –
rather, they ‘muddle through’. Even where strategies exist, they often are not
prioritised and they do not have a project plan. In these circumstances, the ‘theory of
action’ may have to be hypothesised by the evaluator, who may then have to seek to
validate it by checking out with a range of stakeholders as to whether it accords with
their sense as to what was the original ‘theory of change’. The greatest problem with
this approach is that each stakeholder may have its own ‘theory of change’ and may
not care greatly for – or even notice – the theories of change held by other
stakeholders, so that this approach to retrospective validation of the ‘theory of
change’ may not work well.

Consequently, the more shadowy is the e-government or e-governance strategy, and

the more frequently it has been changed over time, the more difficult it is to evaluate
whether ‘the strategy’ worked in practice. What is then more interesting is whether
the changes in the strategy were well made, given the signals from the environment –
both from the external customers and from the internal stakeholders. This moves the
focus away from the outcomes of a long-term strategy (which may not exist or may be
very difficult to discern) towards the intermediate outputs of a set of changing short-
term strategies (which are readily discernible but may hide the underlying strategy - if
there was one).
 23

PERFORMANCE INDICATORS FOR E-GOVERNMENT AND E-

GOVERNANCE

The key issues which will be considered in this section are:

• Principles in designing and choosing PIs

• Potential uses of performance indicators – benchmarking progress, assessing
quality, measuring benefits and costs, assessing effectiveness of e-
government and e-governance programmes and initiatives
• Scope of PIs required
- effects of e-government and e-governance on responsiveness of
governments and other key stakeholders
- impacts on public administration made possible by e-government and e-
governance
- effects on operational and financial performance, effectiveness and policy
cohesion across the whole of the public sector and public services, arising
from implementation of e-government and e-governance
- effects on e-government of critical success factors in service delivery
• Draft library of PIs for a range of stakeholders
• Information collection issues around PIs, including issues of comparability
across OECD Member Countries
• Potential for turning PIs in some areas into standards
• Some silver rules of performance management

Principles in designing and choosing PIs

The key principles in designing PIs are:

• Each PI should be relevant to at least one of the objectives of the programme

or initiative
• It should be clear whether a change in a PI indicates that the objective(s) is
being achieved to a greater or lesser degree than before
• The score on the PI should be, at least partly, subject to influence, if not full
control by the politicians or managers concerned.
• The score on the PI should not easily be subject to manipulation
• Scores should be normalised

The key principles in choosing PIs to fit into an assessment system are:

• A portfolio of PIs should be chosen which cover all the main dimensions
which we wish to evaluate, including
o level of activity (‘outputs’)
o volume of use
o social or other groups to which users belong (‘targeting’)
o quality of experience offered to users
o unit cost.
 24

• Only those PIs should be chosen which vary significantly througho ut the
population to be studied, so that the PI allows us to distinguish between
different strategies.

Potential uses of performance indicators

Performance indicators can have a number of uses within performance management

systems:

?? appraising potential projects, programmes or policies in advance

?? monitoring progress against predefined targets
?? increasing the transparency of public processes (e.g. by allowing citizen
access to information on the stage reached in documentation processing or
decision-taking)
?? highlighting the incidences (or possible incidences) of fraud or corruption
(within the ethics infrastructure)
?? benchmarking progress against other agencies, other programmes (or even
other countries)
?? assessing quality against predefined standards (often in mid-project or mid-
term review, since quality PIs are not easy to collect on an ongoing basis)
?? measuring benefits and costs as part of an overall assessment of individual
initiatives or projects (usually in mid-project and at the end of an initiative or
project, since overall benefits and costs are often expensive to measure and to
collate)
?? assessing overall cost-effectiveness of e-government and e-governance
programmes and policies (again typically at mid-term or end of such
programmes or at the stage of a major policy review)

It can be seen that these uses of PIs correspond essentially to the ‘appraisal,
monitoring and ex-post review’ stages of evaluative activity.

Scope of PIs required

The PIs to be formulated need to cover the full range of issues in assessing e-
government and e-governance. This means that they need to be able to cover at lesast
the following dimensions:

- effects of e-government on responsiveness of governments and other key

stakeholders to the needs of their users, covering such effects as
??more user-oriented service planning
??more user-oriented service delivery
??more user-oriented service complaints mechanisms

- effects of e-governance on responsiveness of governments and other key

stakeholders to the needs of their citizens, covering such effects as
??more citizen-oriented policy- making processes
 25

??more citizen-oriented processes of accountability

- impacts on public administration made possible by e-government, which

will cover inter-organisational issues such as:
??policy cohesion across the whole of the public sector and
public services (i.e. achievement of ‘joined-up policy-
making’)
??synergy between programmes across the whole of the
public sector and public services (i.e. achievement of
‘joined- up service delivery’)

- effects of e-government of critical success factors in service delivery

(OECD draft report From in-line to on-line: delivering better services),
including:
??Leadership
??Strategy building
??Responsiveness to stakeholders
??Skills
??Budget
??Integration
??ICT management
??Partnership working
??Change management and innovation

Draft library of PIs for a range of stakeholders

Clearly, the PIs developed should also cover all the key aims and objectives specified
in the previous section. As these key aims and objectives are stakeholder-specific, it
will usually be important that the PIs should also reflect stakeholder interests and that
PI reporting systems allow main stakeholders to see clearly how performance is being
achieved in e- government and e- governance, from their perspectives.

In Appendix 1 we set out a draft library of PIs which might be used in the assessment
of e- government and e- governance programmes.

Each PI in Appendix 1 is associated with at least one stakeholder – and often it is

clear that PIs will be interesting to ma ny stakeholders.

Under e-government, the areas of activity in which PIs are suggested are:

??General impacts on the public

o Public access achieved
o Digital divide and public access
o Public usage achieved
o Digital divide and public usage
o Developing access channels
 26

??Impacts on corporate management system

o Corporate strategy
o Corportate leadership
o Corporate culture change
o Corporate procurement of e- government infrastructure

??Impacts on corporate communication

o developing joined-up communications with legislatures, other public
bodies and PSOs
o customer relations management
o developing front-office integration
o developing back office integration (partnership management, supply
chain management, knowledge management, e-learning)

??Impacts on service planning and service commissioning

??Impacts on service delivery

o Meeting service needs of customers

o Enhancement of service quality
o Time and cost implications

Under e-governance, the areas of activity in which PIs are suggested are:

??General

o Usage achieved
o Digital divide and usage

??Management of interface with external stakeholders

o Interactions with the public sector enabled by electronic means

o Transparency of public decision- making processes
o Ethical conduct in internet interactions
o Establishing citizen confidence in the security of the internet

??Developing networks with external stakeholders

In a small number of cases, there is a target value of the PI available as an example

from one country or another (mainly the UK).
 27

Information collection issues around PIs, including issues of comparability across

OECD Member Countries

In Appendix 1, we also set out some suggestions as the data sources which might be
used in collecting the information necessary to calibrate these PIs. In most cases, it is
suggested that this data might come from:

??Survey of population
??Survey of households
??Survey of staff in PSOs and government
??Management Information Systems (MIS) in the public service organisations
concerned.

However, a range of other sources is also suggested for specific PIs.

Potential for turning PIs in some areas into standards

When PIs are particularly important, it may be desirable to turn them into standards,
against which performance can be benchmarked. Such standards may be used as
major targets in national programmes.

Not all PIs are suitable for standard-setting. In particular, if a PI can be easily
manipulated by managerial action – e.g. through changing the dates or times at which
the PI is measured, or by altering the population for which it is measured – then it is
unlikely to be appropriate as the basis for a standard.

Some silver rules of performance management

While there are no ‘golden laws’ of performance management which apply in all
circumstances, there are some ‘silver rules’ which appear to have quite widespread
validity.

• Be clear about purposes – these may include imposing control, giving

strategic direction, giving hands-off empowerment, and encouraging learning
about ‘what works’. The control purpose is likely to raise the hackles of staff
and, if it is the only purpose, it is likely to be significantly undermined by the
‘perverse control syndrome’ (Bovaird, 2000).

• Relate PIs to objectives, unless the objectives are poorly defined – e.g. if the
objectives are vague, ambiguous, complex and interacting, partially hidden or
unstated, unmeasurable or unrealistic.

• Set targets (i.e. target values within specified time periods) for each of the
PIs in order to indicate the short and medium term pathways towards
objectives - but don’t allow targets to overwhelm objectives – the targets are
inevitably short-termist, myopic and narrow compared to the objectives,
 28

which in the end are much more important.

• Set balanced portfolios of PIs – this will often involve separate PIs for
economy, efficiency, effectiveness, outcomes, equity and quality

• Self-assessment is normally preferable to external independent assessment

because it is more knowledgeable, cheaper and involves those who will be
key to implementation of lessons learnt – but it requires rigorous and credible
audit, in order to stop staff “putting a gloss” on their self- reported results and
to ensure that alternative viewpoints are surfaced.

• Agree the performance management system, don’t impose it, because imposed
systems will be easy to undermine by disaffected and uncommitted staff –
unless it proves too difficult to agree a system, in which case an imposed
system may kick-start dialogue and debate. (However, assume any imposed
system which has not become agreed after two years has already been
undermined and that its data are either too contaminated to be useful, in
which case the whole exercise has been damaging and counterproductive).

• Set priorities for each service and assess the performance of the priorities first
– you are unlikely to have the time to undertake a comprehensive assessment
of all aspects of what you do. Similarly, decide what aspects of the
performance management system are priorities, in terms of their likely pay-
off relative to their likely costs, and implement these parts of the performance
management system first.

• Make comparisons over time and between departments/agencies – while it is

true that all comparisons are at least partly misleading, it is also the case that
we only learn through making comparisons, however imperfect they may be.

• Organise for performance management – it will not happen automatically and

performance management systems will not be taken seriously unless senior
managers show that they find them interesting and use them in decision-
making.

• In spite of the complications introduced by all the above ‘silver rules’, the
overall system of performance management must remain “short, sharp,
snappy”, because otherwise it will be too cumbersome to be memorable, and
unmemorable systems do not affect everyday decision-making, and are thus
relatively ineffective.
 29

EVALUATION OF E-GOVERNMENT PROGRAMMES AND INITIATIVES

In this section, the key issues which will be considered are:

• Role of evaluation
• Key service delivery issues which need to be covered by evaluation
frameworks
• Key governance issues which need to be covered by evaluation frameworks
• Evaluation methodologies
• Using evaluation results

Role of evaluations

It is important to be clear why evaluations are being carried out, as the purpose will
help to decide the design of the evaluation – and will also condition the response to it.

The purposes of evaluation may include:

- political and managerial accountability of organisations, partnerships and

networks, through producing information on how successful (or
unsuccessful) their efforts have been;
- policy learning in policy networks, through producing information on
which aspects of policy have been most successful;
- supporting leadership through informing strategic direction and allowing
this strategic direction to be justified to all those stakeholders who may
have reasons for resisting it;
- activation of stakeholders, through alerting them to areas in which they are
less successful in receiving the benefits of projects (or areas in which they
are especially likely to be bearing the costs).

As in the case of performance management in the previous section, if the first of these
purposes – accountability – is interpreted as meaning that there is a hunt for a
scapegoat, who will be blamed for lack of success, then the whole evaluation effort
may well be undermined by the behaviour of staff, who will ensure that the data
provided to the evaluation obscure much of what really went on. Only if some at least
of the other purposes are widely believed to be important is there a real hope that the
former purpose can also be achieved.

Finally, a reflection on how such roles may be relatively short-term: Whinston et al

(2001; p. 3) suggest “the Internet is increasingly becoming part of the basic business
model for many companies, laying the groundwork for even more impressive growth
during more favourable economic conditions. The Internet is rapidly becoming a part
of the traditional economy – like telephones, elevators and personal computers over
the years – leading to the day when there will be no separate measure of the internet
economy”. The same logic may well mean that, before very long, the internet is so
 30

much part of the woodwork in the public sector and in public services that it will no
longer merit separate consideration as a key driver of change and improvement.

Key electronic service delivery issues which need to be covered by evaluation

frameworks

In the evaluation of the e-government programme, the following service delivery

issues are likely to be central aspects which will need to be evaluated:

- Improving access
- Reducing administrative burdens
- Reducing costs to administration
- Providing integrated services
- Improving quality of services
- Tailoring services to customer needs
- Incorporating citizen feedback
- Ensuring privacy and security
- Providing strategic direction
- Ensuring oversight and control
- Adapting to changes

The PIs suggested in Appendix 1 provide a set of measures which would allow all of
these variables to be measured, to varying degrees of precision. The latter three issues
may well be the most difficult, as they are areas in which it is especially problematic
to devise PIs.

Key e-governance issues which need to be covered by evaluation frameworks:

- Need for multi-stakeholder analysis

- Transparency of decision- making
- Accountability of stakeholders to other (multiple) stakeholders
- Engagement of all stakeholders in planning, decision- making and
evaluation
- Activation of all stakeholders, including those difficult-to-reach or
difficult-to-motivate
- Access to information within clearly agreed privacy constraints
- Joined-up working in partnerships
- Holistic perspectives on quality of life of stakeholders
- Ethical behaviour and integrity
- Flexibility and adaptability of organisations, institutions and mechanisms
- Coherence of programmes
- Legitimacy of institutions

The PIs suggested in Appendix 1 again provide a set of measures which would allow
all of these variables to be measured, to varying degrees of precision. Once again, it is
 31

the latter three issues which may well be the most difficult to evaluate, as they are
areas in which it is especially problematic to devise PIs.

Evaluation methodologies

There are a number of evaluation models that are, or could be, used to measure e-
government and e-governance implementation and impact.

First, there are models for evaluating the aggregate of programme outcomes,
which typically fall into 4 categories:

a. goals achievement matrix , where there is an attempt to score the initiatives and
strategies against a weighted set of goals;

b. balanced scorecard approaches, where there is an attempt to score the

initiatives against four or five sets of performance indicators, which may or
may not be linked to goals and where weighting may or may not apply

c. cost-benefit models, where there is an attempt to quantify most of the benefits

and most of the costs in money terms;

d. cost-effectiveness models, where there is an attempt to assess which strategy,

out of a set of strategies with approximately equal cost, is likely to score best
against a weighted set of goals, or which strategy, out of a set of strategies
judged to have approximately equal effectiveness, is likely to be least cost.

In recent years, there has been less enthusiasm for evaluation methodologies which
imply aggregating across categories (such as cost-benefit analysis), on the grounds
that this is an area where the decisions should primarily be political, not technical, in
character.

Secondly, there are models for measuring the individual outcomes within each
generic type of outcome which occurs in a programme. These typically fall into 4
categories:

a. changes in the satisfaction levels of users or other relevant stakeholders –

these are quantifiable, subjective data;

b. changes in the levels of certain states (e.g. employment status, health status,
educational qualifications, etc.) characteristic of users or other relevant
stakeholders – these are quantifiable, objective data;

c. changes in the attributes or features (e.g. married or unmarried, holder or

non-holder of driving license, with or without hospital appointment for
condition reported to local doctor) of users or other relevant stakeholders –
these are non-quantifiable, objective data;
 32

d. changes in the attitudes or mental conditions (e.g. happy or sad, optimistic

or pessimistic, envious or content) of users or other relevant stakeholders –
these are non-quantifiable, subjective data.

Thirdly, there are methods for assessing whether the changes observed in the
evaluation process are likely to have been associated with the programme being
evaluated:

a. randomised trial with double-blind experimental and control groups;

b. randomised trial with single-blind experimental and control groups;

c. randomised trial with fully aware experimental and control groups;

d. area-based trials, where all residents in a particular area are in the same group,
whether experimental or control;

e. before-and-after trials, with control groups;

f. before-and-after trials without control groups.

Generally, the strength of conclusions which can be drawn from an evaluation

decreases as we move from a. to e. in the above list (although area-based trials can
have both some special advantages and some special disadvantages in specific cases
which mean that they move a little up or down this ‘ranking’).

Using evaluation results

Evaluations are not of any value unless their results become used. The history of
evaluations in both public sector and in the private sector shows that a high proportion
of evaluations do not have a significant effect in the organisations which they were
designed to help. As Hegel remarked “What history teaches us about governments is
that governments do not learn from history”.

In order to make it more likely that the performance management and evaluation
approaches outlined in this paper will be put to some use, it is important that they are
operated jointly by the stakeholders who will be responsible for learning the lessons
which they throw up, and for implementing the consequent changes.

This means that stakeholder-based evaluation is much more likely to be successful

than evaluation done to stakeholders. Of course, this brings with it some attendant
dangers, particularly of lack of independence and imaginative innovation.
Consequently, stakeholder-based evaluation must include such a range of stakeholders
that the final evaluation is likely to have a rounded perspective and a creative search
for alternative ways forward.
 33

CONCLUSIONS

??Performance indicators for e-government and e-governance need to relate

closely to a longer term strategic vision and strategic direction for government
and for public service organisations. They should clearly tie in closely with a
hierarchy of objectives, which sets out the aspirational agenda, rather than
being stand-alone.

??Organisations do not have objectives – it is stakeholders who have objectives.

It is important that the differences between stakeholder perspectives and
priorities are understood when setting direction and monitoring performance
of e- government and e- governance programmes.

??Performance indicators must cover a range of different objectives, including

level of access (particularly across social groups), level of usage, effects on
public service quality and cost, effects on organisational excellence, effects on
network and partnership working and effects on key governance issues, such
as stakeholder consultation and participation, transparency of decision-
making, and standards of ethical conduct.

??It is possible to devise a large library of PIs which might be useful to

particular stakeholders for specific purposes at a given time. However, in
practice, it is unwise to try to collect and to use too many PIs at once.

??Evaluation methods can make use of essentially the same library of PIs as
performance management approaches.

??Whichever evaluation methods are used, close involvement of stakeholders is

essential if the results are to have any chance of influencing future behaviour.

??The most popular forms of evaluation in recent years have not attempted to
aggregate benefits across categories – this has been seen as an area for
primarily political decision- making. This is likely to remain true in e-
government and e-governance.
 34

RECOMMENDATIONS

In this section, we make some brief recommendations for priority areas in the
measurement of the performance of e-government and e-governance and the
evaluation of e-government and e-governance programmes. They are broken down by
major stakeholder.

• For OECD

? ? The library of PIs presented in this paper should be extended and

refined in consultation with stakeholders in OECD Member
Countries and a range of demonstration projects should be
undertaken to test the comparative success of e-government and e-
governance in a sample of Member Countries which currently are
taking a leading position in this field.

• For governments of OECD Member Countries

? ? A wider range of performance measures should be used to test the

success of current e-government and e-governance programmes.

? ? Given the cost of these programmes, and the risks which they entail
to both governments and to the public, the most ambitious e-
government and e-governance programmes in each country should
be evaluated as a matter of urgency, using and refining clusters of
the PIs in the library of PIs recommended in this paper, so that the
results can be widely disseminated

• For local government

? ? Given the costs and risks associated with these programmes, it is

important that a range of varied and imaginative e-government and
e-governance programmes be supported in each Member Country,
with a careful evaluation programme, using and developing the
library of PIs outlined in this paper.

• For NGOs and civil associations

? ? Given the potential but also the risk which e-government and e-
governance programmes pose, particularly to disadvantaged and
vulnerable groups, it is of paramount importance that these
programmes should be designed imaginatively, to incorporate a
wide degree of diversity, and should be evaluated thoroughly and
quickly. Both the design and evaluation of these programmes
 35

should be a multi-stakeholder responsibility, and NGOs and civil

society associations should play a major role in these processes – if
necessary, raising their own resources to ensure that this evaluation
is done in a way which is appropriate to reflect the interests of their
members

• For ICT suppliers

? ? Evaluation of ‘what works and what does not’ is going to be a key

theme of the digital revolution. Electronic service delivery and e-
governance are intrinsically well-placed to record informatio n on
the characteristics of users. However, current systems are still not
well-designed to expedite performance management and evaluation
– this is an area which is likely to give suppliers a competitive
advantage in the future.

? ? It will important to design future systems with the needs of

multiple stakeholders in mind, and not just the stakeholder who is
putting the most money up front. This also means the need to build
in stakeholder based performance measures and evaluation criteria
when the systems are being designed.

• For other businesses

? ? Government to Business (G2B) service delivery and governance

issues will grow in importance as governments catch up with the
pace of change which has characterised e-commerce. It will be
important for business to understand government requirements, so
that the design of new digital-based information systems can reflect
these likely future needs. Further, it will be valuable for business if
it is made clear to government how the new e-government and e-
governance systems can be designed in ways which reduce the
burden of administrative costs which is entailed by public
regulation. This is most likely to be effective if undertaken in
collaboration with other business partners and, in some cases, with
organisations in the third sector.
 36

REFERENCES

Bovaird (2001), “Strategic objective maps as 'windows' into worlds of cause-and-

effect”, Paper presented at Reframing Organisational Performance, EAISM
Conference, Brussels, October 2001.

BSI (2002), Chefsache E-Gove rnment: Leitfaden für Behördenleiter. E-

Government Handbuch. Bonn: Bundesamt für Sicherheit in der Informationstechnik
(www.e-government-handbuch.de).

Cabinet Office (2001), E-government: a strategic framework for public services in

the Information Age. London: Cabinet Office (http://www.e-
envoy.gov.uk/ukonline/progress/estrategy/summary.htm)

CyberAtlas (2001), E-Government May Not Mean Efficiency.

(http://cyberatlas.internet.com/markets/professional/article/0,,5971_929471,00.html)

DTLR (2001), e-Government: Delivering Local Government Online –

Guidelines for Preparing ‘Implementing Electronic Government’
Statements. London: Department of Transport, Local Government and the
Regions. (http://www.detr.gov.uk)

DTLR (2002), Best Value Performance Indicators 2002/2003. London:

Stationery Office. (http://www.local-
regions.dtlr.gov.uk/bestvalue/indicators/pi2002-03/06.htm)

European Commission (2002), e-Europe 2002: e-Europe Benchmarking Report.

Brussels: European Commission.

Richard Heeks (2001), Understanding e-Governance for Development. i-

Government Working Paper Series, Working Paper No. 11. Manchester: Institute for
Development Policy and Management, University of Manchester.
(http://idpm.man.ac.uk/idpm/igov11.htm)

OECD (2001), Understanding the Digital Divide . Paris: OECD.

OECD (2001???), Project on the Impact of E-Government. PUMA(2001)10 REV

2. Paris: OECD.

OECD (2001), Citizens as Partners; Information, Consultation and Public

Participation in Policy-Making. Paris: OECD.

OECD (2001), The Hidden Threat to E-Government: Avoiding Large

Government IT Failures. Paris. OECD.
 37

OECD (2001???), From In-Line to On-Line. Paris: OECD.

PUMA (2001), E-Government: Analysis Framework and Methodology.

PUMA2001/16/ANN/Rev1. Paris: OECD.

UNESCO (2002), Joint Unesco And Comnet-It Study Of E-Governance.

Development Of Country Profiles.
(www.comnet.mt/Unesco/CountryProfiles/Project/joint_unesco_and_comnet.htm)

Andrew Whinston et al (2001), Measuring the Internet Economy. Cisco Systems

and University of Texas. (http://www.internetindicators.com/jan_2001.pdf).
 38

APPENDIX 1. THE OVERALL LIBRARY OF POTENTIAL PIS FOR E-GOVERNMENT AND E-GOVERNANCE

Area of public Key Performance Sub-indicator Example of a Notes on scope Collection Stakeholders likely
sector and public Indicator target of PI method to be interested
service activity

E-GOVERNMENT

General impacts on
public
- public access achieved % of households with Telecoms provider Government
telephone access at statistics Public
HOME
% of population with Survey of population Government
internet access at and/or businesses Public
HOME OR WORK Business
% of households Survey of households Government
where at least one and/or businesses Public
member has Business
internet access at
HOME OR
WORK
% of population with 100% of Survey of households Government
internet access at population to and/or internet Public
CONVENIENT have free access facilities at public
PUBLIC BUILDING to internet in buildings
local library by
2005 (UK
government
% of population with Survey of population Government
internet access at Public
HOME
 39

% of households Survey of households

with internet
access at HOME

- digital divide and % of households in Survey of households Government

public access lower socio-economic NGOs
groups with telephone Voluntary groups
access at HOME
% of ‘vulnerable’ Survey of households Government
households with NGOs
telephone access at Voluntary groups
HOME
% of population in Survey of population Government
lower socio-economic NGOs
groups with internet Voluntary groups
access at HOME OR
WORK
% of households Survey of households Government
in lower socio- NGOs
economic groups Voluntary groups
where at least one
member has
internet access at
HOME OR
WORK
% of ‘vulnerable’ Survey of population Government
population with NGOs
internet access at Voluntary groups
HOME OR WORK
% of ‘vulnerable’ Survey of households Government
households where NGOs
at least one Voluntary groups
member has
internet access at
HOME OR
 40

WORK
% of population in Survey of population Government
lower socio-economic NGOs
households with Voluntary groups
internet access at
HOME
% of households Survey of households Government
in lower socio- NGOs
economic groups Voluntary groups
with internet
access at HOME
% of ‘vulnerable’ Survey of population Government
population with NGOs
internet access at Voluntary groups
HOME
% of ‘vulnerable’ Survey of households Government
households where NGOs
at least one Voluntary groups
member has
internet access at
HOME

- public usage achieved % of population who Broken down by Survey of population Government
have contacted public purpose – seeking PSOs
services by telephone information/advice, NGOs
from HOME or seeking on-line Voluntary groups
services
% of population who Broken down by Survey of population Government
have used internet at purpose – seeking PSOs
HOME OR WORK to information/advice, NGOs
visit public service or seeking on-line Voluntary groups
websites services
% of households Survey of households Government
where at least one PSOs
 41

member has used NGOs

internet at HOME Voluntary groups
OR WORK to
visit public service
websites
% of po pulation who Broken down by Survey of population Government
have used internet at a purpose – seeking PSOs
convenient public information/advice, NGOs
building to visit public or seeking on-line Voluntary groups
service websites services
Number of users Survey of internet Government
of internet facilities at public PSOs
facilities at buildings NGOs
CONVENIENT Voluntary groups
PUBLIC
BUILDING (as a
proportion of local
catchment area)
% of population who Broken down by Survey of population Government
have used internet at purpose – seeking PSOs
HOME to visit public information/advice, NGOs
service websites or seeking on-line Voluntary groups
services
% of households Survey of households Government
where at least one PSOs
member has used NGOs
internet at HOME Voluntary groups
to visit public
service websites

- digital divide and

public usage
% of population in Three different Survey of population Government
lower socio-economic levels of skills NGOs
groups with different could be defined = Voluntary groups
 42

skill levels in internet basic, intermediate, Training organisations

usage (basic, high.
intermediate, high)
% of ‘vulnerable’ Three different Survey of population Government
population with levels of skills NGOs
different skill levels in could be defined = Voluntary groups
internet usage basic, intermediate, Training organisations
high.

% of young people Age 16 is chosen Survey of population Government

(aged 16) from because this is the or survey of school NGOs
‘vulnerable groups’ minimum age at education Voluntary groups
who have different which young Training organisations
skill levels in internet people can leave
usage school.

% of households in Broken down by Survey of households Government

lower socio-economic purpose – seeking PSOs
groups who have information/advice, NGOs
telephoned public or seeking on-line Voluntary groups
services from HOME services
% of population in Broken down by Survey of population Government
lower socio-economic purpose – seeking PSOs
groups who have information/advice, NGOs
accessed public or seeking on-line Voluntary groups
services websites at services
HOME OR WORK
% of households Survey of households Government
in lower socio- PSOs
economic groups NGOs
where at least one Voluntary groups
member has
accessed public
services websites
at HOME OR
 43

WORK
% of population in Broken down by Survey of population Government
lower socio-economic purpose – seeking PSOs
groups who have information/advice, NGOs
accessed public or seeking on-line Voluntary groups
services websites at services
CONVENIENT
PUBLIC BUILDING
% of households Survey of households Government
in lower socio- PSOs
economic groups NGOs
where at least one Voluntary groups
member has
accessed public
services websites
at a
CONVENIENT
PUBLIC
BUILDING

% of ‘vulnerable’ Broken down by Survey of population Government

population who have purpose – seeking PSOs
accessed public information/advice, NGOs
services websites at or seeking on-line Voluntary groups
HOME OR WORK services
% of ‘vulnerable’ Broken down by Survey of households Government
households where purpose – seeking PSOs
at least one information/advice, NGOs
member has or seeking on-line Voluntary groups
accessed public services
services websites
at HOME OR
WORK
% of ‘vulnerable’ Broken down by Survey of population Government
population who have purpose – seeking PSOs
 44

accessed public information/advice, NGOs

services websites at a or seeking on-line Voluntary groups
CONVENIENT services
PUBLIC BUILDING
% of ‘vulnerable’ Survey of households Government
households where and/or survey of PSOs
at least one internet facilities in NGOs
member has public buildings Voluntary groups
accessed public
services websites
at a
CONVENIENT
PUBLIC
BUILDING

% of population in Broken down by Survey of population Government

lower socio-economic purpose – seeking PSOs
households who have information/advice, NGOs
accessed public or seeking on-line Voluntary groups
services websites at services
HOME
% of households Survey of households Government
in lower socio- PSOs
economic groups NGOs
where at least one Voluntary groups
member has
accessed public
services websites
at HOME
% of ‘vulnerable’ Broken down by Survey of population Government
population who have purpose – seeking PSOs
accessed public information/advice, NGOs
services websites at or seeking on-line Voluntary groups
HOME services
% of ‘vulnerable’ Survey of households Government
 45

households where PSOs

at least one NGOs
member has Voluntary groups
accessed public
services websites
at HOME

Developing access Extent to which access Difficult to find a Government

channels channels (for services) single PI for this PSOs
have been increased variable, so it is ICT suppliers
best measured by
the following sub-
indicators
% of services (by It will be expected Government
value) with full that this will rise in PSOs
range of access early phase of e- ICT suppliers
channels government. Public
However, it may NGOs
later fall if Voluntary groups
electronic channels
take over from
some traditional
channels. This
could have serious
consequences if
digital divide
problems have not
been overcome,
hence the need for
some associated
sub-indicators
% of services (by Government
value) where face- PSOs
to-face contact is ICT suppliers
 46

not available (and Public

% of these NGOs
services used by Voluntary groups
lower socio-
economic groups
and by
‘vulnerable’
groups)
% of services (by Government
value) where only PSOs
electronic access is ICT suppliers
available (and % Public
of these services NGOs
used by lower Voluntary groups
socio-economic
groups and by
‘vulnerable’
groups)

% of population with ‘Relevant services’ Government

‘smart card’ access to means all services PSOs
ALL relevant services for which smart ICT suppliers
cards are available Public
NGOs
Voluntary groups
% of population Government
with ‘smart card’ PSOs
access to SOME ICT suppliers
relevant services Public
NGOs
Voluntary groups
% of ‘vulnerable’ Government
population and PSOs
lower socio- ICT suppliers
income groups Public
 47

with ‘smart card’ NGOs

access Voluntary groups

Impacts on corporate
management system
- corporate strategy Existence of a This is a ‘yes or no’ MIS of PSOs Government
corporate strategy for answer PSOs
e-government ICT suppliers
Existence of a This is a ‘yes or no’ MIS of PSOs Government
corporate plan and answer PSOs
work programme for ICT suppliers
transition from
traditional to
electronic service
delivery
Existence of targets This is a ‘yes or no’ MIS of PSOs Government
and milestones in the answer PSOs
plan ICT suppliers
Media

- corporate leadership Existence of credible Not easy to Government

commitment of measure as a PSOs
politicians and variable – better ICT suppliers
leadership to e- assessed by the
government strategy following two
indicators
% of staff who Staff survey Government
believe that design PSO staff
and PSOs
implementation of
electronic delivery
of services is a
high priority
% of partner Survey of senior staff Government
 48

agencies who in key partner Partner agencies

believe that the agencies PSOs
public sector
organisation gives
a high priority to
the design and
implementation of
electronic service
delivery
% of Action Plan MIS in PSOs Government
targets missed in PSOs
two successive ICT suppliers
half-years Partners
Media

- corporate culture % of politicians on- MIS of public sector Government

change line organisations
% of politicians Government
with e-government
training
% of politicians MIS of public sector Government
who have used organisations
internet in last
month
% of top management MIS of PSOs Government
on-line
% of top Government
management with PSOs
e-government
training
% of top MIS of PSOs Government
management who PSOs
have used internet
in last month
 49

% of staff on-line
% of staff with e- Government
government Staff
training PSOs
% of staff who
have used internet
in last month

% of transactions with Periodic survey of Government

key partners which communication flows Partners
are transacted in partnerships PSOs
electronically

% of staff with MIS of PSOs Government

agreements to work Staff
from home when PSOs
appropriate
% of staff who Staff survey Government
have worked from Staff
home in last PSOs
month

Corporate procurement Existence of Best assessed by Government

of e-government partnerships for e- following sub- PSOs
infrastructure government provision indicators ICT suppliers
with other relevant
public bodies and ICT
providers
% of ICT services MIS of PSOs Government
(by value) PSOs
provided in ICT suppliers
partnership with Partners
other public
bodies
 50

% of ICT services MIS of PSOs Government

(by value) PSOs
provided by ICT suppliers
external private
ICT firms

Impacts on corporate
communication

- developing joined-up Extent to which each Not easy to Government

communications with organization has measure as a PSOs
legislatures, other public capacity to answer variable, so best ICT suppliers
bodies and PSOs queries on behalf of all measured through Partners
its partners the associated sub-
indicators
% of ALL MIS in PSOs and/or Government
contacts by public survey of population PSOs
which have to be Partners
referred to ICT suppliers
partner agency or
PSO
% of telephone MIS in PSOs and/or Government
contacts by public survey of population PSOs
which have to be Partners
referred to ICT suppliers
partner agency or
PSO
% of internet MIS in PSOs and/or Government
contacts by public survey of population PSOs
which have to be Partners
referred to ICT suppliers
partner agency or
PSO
% of internet Internet survey of Government
 51

contacts to which citizens making PSOs

the response is contacts Partners
within agreed time ICT suppliers
standards
% of internet Internet survey of Government
contacts which citizens making PSOs
lead to satisfactory contacts Partners
conclusion without ICT suppliers
referral to another
officer or agency

- customer relations % of all SERVICE Government

management USERS on an Public
integrated data base
% of all such Government
customers whose PSOs
data can be shared Public
with all or some Partners
partners
% of all such 100% for Government
customers who compliance with PSOs
have given data protection Public
permission for laws (within EU) Partners
such data to be
shared with other
relevant PSOs
% of all CITIZENS on Citizen data bases are Government
an integrated data often compiled so that PSOs
base promotional Public
campaigns and market
research can be
targeted at non-users
% of all such Government
citizens whose PSOs
data can be shared Public
 52

with all or some Partners

partners
% of all such 100% for Government
citizens who have compliance with PSOs
given permission data protection Public
for such data to be laws (within EU) Partners
shared with other
relevant PSOs

- developing front office Increase in staff Government

integration working solely on PSOs
telephone reception Staff
Increase in staff Including the time MIS in PSOs Government
time working on of staff who also PSOs
telephone work on other Staff
reception tasks.
Officers involved
must be
electronically
enabled
Increase in staff FTEs MIS in PSOs Government
working solely on PSOs
websites Staff
Increase in staff Including the time Government
time working on of staff who also PSOs
websites work on other tasks Staff

Reduction in public Government

service offices with PSOs
public counters which Staff
are staffed Public
Taxpayers
Reduction in staff FTEs Government
on public counters PSOs
at public offices Staff
 53

Public
Taxpayers

-developing back -office

integration
- partnership % of partners on an MIS of PSOs Government
management integrated data base PSOs
Partners

- supply chain % of suppliers on an MIS of PSOs Government

management integrated data base PSOs
Suppliers

- knowledge % of officers who have Survey of staff Government

management full e-access to all data PSOs
bases needed for their Staff
work (‘electronic
office’)

% of internal process Survey of staff Government

innovations which PSOs
involve extended use Staff
of internet

- e-learning % of staff who have MIS of PSOs Government

undertaken e-learning PSOs
programmes Staff
% of all staff who MIS of PSOs Government
have undertaken PSOs
e-learning Staff
programmes use
of
internet/intranet
% of professional MIS of PSOs Government
staff who have PSOs
 54

undertaken e- Professional staff

learning
programmes in
their professional
area
% of managerial MIS of PSOs Government
staff who have PSOs
undertaken e- Managerial staff
learning
programmes in
their managerial
area

Impacts on service % of population Survey of population Government

planning and service contributing to service PSOs
commissioning planning processes by Public
telephone or internet
% of population Survey of population Government
who believe that PSOs
they have Public
contributed to
service planning
processes by
telephone or email
% of population MIS of PSOs Government
recorded by PSOs PSOs
as having Public
contributed to
service planning
by telephone or
email contacts
% of services (by MIS of PSOs Government
value) where there is PSOs
regular, systematic Public
 55

contribution by the
public to the planning
process which is
conducted by
telephone or internet
% of services (by MIS of PSOs Government
value) where more PSOs
inputs from the Public
public in the
service planning
process come from
electronic access
than from face-to-
face interaction

% of services (by MIS of PSOs Government

value) where e- PSOs
procurement has been Other suppliers
undertaken ICT suppliers

% of services (by MIS of PSOs Government

value) where e- PSOs
procurement has Other suppliers
been undertaken ICT suppliers

Impacts on service
delivery
- meeting service needs % of users whose ‘Service needs’ May not be easily Government
of customers service needs are met include information measurable directly – PSOs
by telephone-based on services may be better Public
service available and their measured by the
likely benefits and following two sub-
costs, direct access indicators
to service (e.g.
 56

advice,
counseling),
information on
follow-up services
which may be
advisable, access to
feedback process to
record user’s views
of the service, and
access to a
complaints (and/or
redress) procedure
where desired
% of inquiries by Inquiries recorded From records of Government
telephone which by PSO as having telephone inquiries to PSOs
are dealt with been fully resolved PSOs Public
purely by over the telephone
telephone-based (broken down into
service those resolved
during first call,
and those requiring
further calls).

Officers contacted
must be
electronically
enabled
% of such Inquiries recorded Telephone survey of Government
inquiries by by PSO as having sample of telephone PSOs
telephone which been fully resolved inquiries recorded by Public
are dealt with over the telephone PSOs
purely by (broken down into
telephone-based those resolved
service and where during first call,
user is satisfied and those requiring
 57

with the response further calls)

% of population whose Survey of population Government

service needs are met PSOs
by internet-based Public
service
% of inquiries by Inquiries recorded From records of Government
internet which are by PSO as having internet inquiries to PSOs
dealt with purely been fully resolved PSOs Public
by internet-based by internet
service correspondence
(broken down into
those resolved by
first contact, and
those requiring
further contacts)
% of inquiries by Inquiries recorded Internet survey of Government
internet which are by PSO as having sample of internet PSOs
dealt with purely been fully resolved inquiries recorded by Public
by internet-based by internet PSOs
service and where correspondence
user is satisfied (broken down into
with the response those resolved by
first contact, and
those requiring
further contacts)
- enhancement of service % reduction in errors MIS in POS Government
quality in transactions PSOs
processed Public
electronically
compared to
traditional processing
methods

% of service users Government

 58

involved in monitoring PSOs

of service quality Public
% of service users Government
who have given PSOs
electronic Public
feedback on
service quality

% of service users who Not easily measurable Government

are ‘co-producers of – better measured PSOs
services’ through sub-indicators Public
below
% of service users Survey of service Government
who regard users PSOs
themselves as ‘co- Public
producers of
services’
% of service users Survey of staff Government
who are regarded PSOs
by staff as ‘co- Public
producers of
services’

- time and cost Staff time increases FTEs Government

implications needed to deal with PSOs
telephone contacts Staff
related to specific
services
Staff time increases FTEs Government
needed to deal with PSOs
internet contacts Staff
related to specific
services
Staff time saving FTEs saved Government
released by use of through faster PSOs
 59

telephone and internet processing of Staff

service delivery inquiries,
transactions, etc.

Monetary value of Euros or US $ Government

overall staff time PSOs
saved by move to e- Public
service delivery Taxpayers
Staff

E-GOVERNANCE

General
- usage achieved % of population who Broken down by Survey of population Government
have contacted public purpose of contact PSOs
sector organisations by – consultation, Public
telephone from home participation,
for non-service specific exercising
reasons accountability
% of population who Broken down by Survey of population Government
have used internet at purpose of contact PSOs
home or work to visit – consultation, Public
public sector websites participation,
for non-service specific exercising
reasons accountability
% of households Survey of households Government
where at least one PSOs
member has used Public
internet at home
or work to visit
 60

public sector
websites for non-
service specific
reasons
% of population who Broken down by Survey of population Government
have used internet at purpose of contact PSOs
home to visit public – consultation, Public
sector websites for participation,
non-service specific exercising
reasons accountability
% of households Survey of households Government
where at least one PSOs
member has used Public
internet at home
to visit public
sector websites for
non-service
specific reasons

- digital divide and % of households in Broken down by Survey of households Government

usage lower socio-economic purpose of contact PSOs
groups who have – consultation, Public
telephoned public participation, NGOs
sector organisations exercising Voluntary groups
from home for non- accountability
service specific reasons
% of ‘vulnerable’ Broken down by Survey of households Government
households who have purpose of contact PSOs
telephoned public – consultation, Public
sector organisatons participation, NGOs
from home for non- exercising Voluntary groups
service specific reasons accountability
% of population in Broken down by Survey of population Government
lower socio-economic purpose of contact PSOs
groups who have – consultation, Public
 61

accessed public sector participation, NGOs

websites at home or exercising Voluntary groups
work for non-service accountability
specific reasons
% of households Survey of households Government
in lower socio- PSOs
economic groups Public
where at least one NGOs
member has Voluntary groups
accessed public
sector websites at
home or work for
non-service
specific reasons
% of ‘vulnerable’ Broken down by Survey of population Government
population who have purpose of contact PSOs
accessed public sector – consultation, Public
websites at home or participation, NGOs
work for non-service exercising Voluntary groups
specific reasons accountability
% of ‘vulnerable’ Survey of households Government
households where PSOs
at least one Public
member has NGOs
accessed public Voluntary groups
sector websites at
home or work for
non-service
specific reasons
% of population in Broken down by Survey of population Government
lower socio-economic purpose of contact PSOs
households who have – consultation, Public
accessed public sector participation, NGOs
websites at home for exercising Voluntary groups
non-service specific accountability
 62

reasons
% of households Survey of households Govern ment
in lower socio- PSOs
economic groups Public
where at least one NGOs
member has Voluntary groups
accessed public
sector websites at
home for non-
service specific
reasons
% of ‘vulnerable’ Broken down by Survey of population Government
population who have purpose of contact PSOs
accessed public sector – consultation, Public
websites at home for participation, NGOs
non-service specific exercising Voluntary groups
reasons accountability
% of ‘vulnerable’ Survey of households Government
households where PSOs
at least one Public
member has NGOs
accessed public Voluntary groups
sector websites at
home for non-
service specific
reasons

% of people Internet survey of Government

responding to internet consultees PSOs
consultation who come Public
from ‘target groups’ NGOs
Voluntary groups
% of people Internet survey of Government
participating in participants in PSOs
decision-making decision-making Public
 63

processes who come processes NGOs

from ‘target groups’ Voluntary groups

Management of
interface with external
stakeholders

- interactions with the The number of types 100% by 2005 ‘Types of Government
public sector enabled by of interactions that are (UK interaction’ means PSOs
electronic means enabled for electronic Government) any contact
delivery (as a between the citzen
proportion of the types and the council,
of interactions that are including providing
legally permissible for information,
electronic delivery) – collecting revenue,
BV 157 in the Best providing benefits
Value PIs of the Audit and grants,
Commission (2002) consultation,
regulation,
applications for
services, booking
venues, resources
and courses, paying
for goods and
services, providing
access to
community,
professional or
business networks,
procurement, etc.

‘Enabled’ presumes
that all services are
capable of being
 64

enabled for
electronic service
delivery unless
there is a legal or
operational reason
why this cannot be
done.
% of corporate The % of policies Government
policies where there is should be weighted PSOs
regular, systematic by the FTEs of Public
contribution by the staff employed on NGOs
public to the planning developing those Voluntary groups
process which is policies (e.g. staff
conducted by employed in
telephone or internet ‘equalities’ work,
in dealing with
ethical codes of
conduct, in
supporting
politicians)
% of policies The % of policies Government
where more inputs should be weighted PSOs
from the public in by the FTEs of Public
the planning staff employed on NGOs
process come from developing those Voluntary groups
electronic access policies (e.g. staff
than from face-to- employed in
face interaction ‘equalities’ work,
in dealing with
ethical codes of
conduct, in
supporting
politicians)

- transparency of public % of activities which MIS of public sector Government

 65

decision-making citizens or service organisations PSOs

processes users can track Public
electronically to NGOs
monitor progress Voluntary groups
Business

- ethical conduct in % of citizens who Government

internet interactions believe they have PSOs
experienced unethical Public
conduct from NGOs
politicians or officers Voluntary groups
in the course of Business
internet interactions

- establishing citizen % of citizens Survey of population Government

confidence in the expressing confidence PSOs
security of the internet in the internet as a Public
secure method of NGOs
communicating with Voluntary groups
government Business
Number of complaints MIS or PSOs and MIS Government
by citizens relating to of government data PSOs
misuse of personal protection registration Public
data arising from agencies NGOs
internet transactions Voluntary groups
Business
% of citizens using MIS of PSOs Government
internet for payments PSOs
for government Public
services or taxes NGOs
Voluntary groups
Business
% of citizens Internet survey of Government
expressing citizens using e- PSOs
confidence in the government channels Public
 66

security of NGOs
internet payments Voluntary groups
to PSOs Business
% of citizens using Need for a protocol Government
internet for exchange on which type of PSOs
of confidential information is Public
information ‘confidential’ NGOs
Voluntary groups
Business
% of citizens Internet survey of Government
expressing citizens using e- PSOs
confidence in the government channels Public
security of the NGOs
internet for Voluntary groups
exchange of Business
confidential
information with
PSOs

Developing networks Extent to which all Not easily be Government

with external strategic partners, key measured – better PSOs
stakeholders stakeholder groups assessed by means of Partners
and priority target sub-indicators Public
groups are integrated NGOs
into internet portals of Voluntary groups
public sector Business
organisatoins
% of strategic By inspection of Government
partners sharing websites PSOs
portal Partners
% of key By inspection of Government
stakeholder websites PSOs
groups integrated Other stakeholders
into portal
% of priority By inspection of Government
 67

target groups with websites PSOs

own section in the Priority target groups
portal

Glossary for Appendix 1

MIS – Management Information Systems

PSOs - Public Service Organisations (which include all agencies which undertake public services directly in the public sector or under
contract to the public sector, including those in the voluntary and private sectors).
‘Target priority groups’ - groups which are target groups for government policy, which might for example be because of their low income or
their vulnerable status in society.

‘Vulnerable groups’ – groups which are target groups for government policy, for reasons other than low income – they are likely to include,
for example, groups of people with disabilities (particularly sight disabilities), people with language difficulties (particularly ethnic minority
groups), people who are likely to rely particularly on non-traditional forms of service delivery (e.g. people who find it difficult to move around
outside their house), etc.

Notes on Appendix 1
 68

1. Many of the impacts measured under ‘e- government’ also reflect changes in e- governance (e.g. number of staff involved in producing
websites) – they would therefore be over-estimated using this methodology, while the impacts measured under ‘e-governance’ in this Table
will probably be under-estimated.

2. Public sector organizations include all parliaments, executive branches of government, executive agencies of government, non-departmental
public bodies and all agencies of directly-elected local government.

View publication stats