EVALUATION: NOVEMBER 2018 PUBLICATION: FEBRUARY 2021

SUSTAINABLE WATER
AND SANITATION IN
ZAMBIA
Impact evaluation of the ‘Urban WASH’ project

Effectiveness Review Series 2018/19

The Chawama Environmental Health and Sanitation Committee works as a door-to-door health promotion team.
Credit: Bekki Frost/Oxfam.

JAYNIE VONK
Data gathering led by CSK Consortium
OXFAM GB

www.oxfam.org.uk/effectiveness
ACKNOWLEDGEMENTS
The author would like to recognize and thank everyone who participated in and contributed to
this Effectiveness Review. Thank you to those at Oxfam in Zambia, especially Dailes Judge,
Takaindisa Sanangurai, Teddy Kabunda and Chantry Mweemba, for their engagement in the
evaluation process, as well as Joseph Yondela, Yvonne Chibiya and others for their support.
Also, thank you to Village Water, particularly Happy Nyasulu and Mampi Silumesii, for
participating in the evaluation design workshop and supporting data gathering as needed.

All data gathering was done by CSK Consortium, including Kelvin Luputa, Chris Shanda and
Kenneth Chola, and a team of 14 interviewers – thank you, together you were a great team to
work with on this evaluation. Appreciation is also due to all the interviewees, representing
institutions and households, who took time out of their days to share their knowledge and
experiences as part of this evaluation.

Thanks also to Simone Lombardini and Alexia Pretari for their support throughout the
evaluation, to the Oxfam GB sustainable water team – Joanna Trevor, Jola Miziniak, Louise
Medland and Tom Wildman – for input and feedback on the Sustainable Water and Sanitation
Index, and to the HWISE network, especially Sera Young, Josh Miller and Chad Staddon, for
support during this initial collaboration using the HWISE scale.

Finally, thanks to Sarah Davies and Oxfam GB’s publishing team for its publication.

MANAGEMENT RESPONSE
Oxfam in Zambia supports the findings and learnings that have emerged from this evaluation.
As part of the process, key findings have been discussed together among Oxfam and Village
Water staff who were involved in project implementation as well as the evaluation. Due to a
change in strategic direction, WASH is no longer currently part of Oxfam in Zambia’s portfolio,
however we expect the Programme Learning Considerations that were co-developed will be
useful for Village Water and other organisations that will continue to implement urban WASH
projects now and in the future.

The learning with a broader scope, that goes beyond WASH interventions, is around ongoing
power analysis. This learning has been reflected on by the Oxfam in Zambia team, and where
possible will be incorporated into new programme design and implementation approaches going
forward as Oxfam in Zambia transitions to the new Oxfam southern Africa cluster.

Signed off by: Dailes Judge, Country Director, Oxfam in Zambia

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
2
CONTENTS

Executive Summary 4

1 Introduction 10

2 Project description 12

3 Evaluation design 14

4 Data 15

5 Measuring Sustainable Water and Sanitation 20

6 Results 23

7 Conclusions 44

References 67

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
3
EXECUTIVE SUMMARY
Oxfam GB’s Global Performance Framework is part of the organization’s effort to better
understand and communicate its effectiveness, as well as enhance learning across the
organization. Under this framework, a small number of completed or mature projects are
selected each year for an evaluation of their impact, known as an ‘Effectiveness Review’ (ER).

During the 2018/19 financial year, one of the projects selected for an ER was ‘Promoting the
right to safe water, sanitation and hygiene (WASH) of peri-urban communities in Lusaka,
Zambia’ – referred to as the ‘Urban WASH’ project. It was carried out in George and Chawama
compounds in Lusaka between July 2013 and June 2017 by Oxfam together with implementing
partner Village Water. The project aimed to improve provision and sustainable management of
WASH services by engaging citizens to hold duty bearers and service providers to account.
Oxfam collaborated with local institutions on an array of activities, engaging stakeholders to
create a conducive environment for service provision and improving capacities and practices.

EVALUATION APPROACH
The ER, for which the fieldwork was carried out in November 2018, aimed to evaluate the
success of this project to increase the sustainability of water and sanitation systems and
services. Using a quasi-experimental evaluation design, we assessed impact among the
households in communities where the project was active in comparison to households in a
similar community where we expected baseline characteristics would have been the same. We
combined this household-level quantitative assessment with community-level qualitative key
informant interviews (KIIs) with relevant institutional representatives. Using this approach, the
evaluation identified effects of the project at the household level and the community level. Any
impacts of broader activities conducted across the entire local area (e.g., radio programming,
roadshows, district-level activities) were not evaluated.

The evaluation was carried out in three compounds of Lusaka, Zambia, the two compounds
where the project was carried out – George and Chawama – and Kanyama Compound as the
comparison community. Kanyama was chosen as the best possible comparison group area for
two reasons. First, Oxfam ran a humanitarian response to a severe cholera outbreak in 2018 in
the three compounds of George, Chawama and Kanyama. This outbreak and the response
were key considerations in the evaluation design. Second, household characteristics in
Kanyama were expected to be similar as those in George and Chawama before the project
started.

Households were randomly selected for interviews (among the entire population within each
compound), and the gender of the interviewee representing each household was randomly
allocated. A total of 1,014 household interviews were completed – 383 in the intervention areas
(George and Chawama) and 631 in the comparison area (Kanyama). At the analysis stage,
propensity score matching (PSM) and multivariate regression were used to control for apparent
baseline differences (using recalled baseline data) between the groups. Key informants were
selected for interviews to represent the project’s main compound-based institutional partners,
with comparable institutional representatives selected in the comparison compound. In total, we
interviewed two representatives in George, four in Chawama and two in Kanyama.

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
4
RESULTS
The evaluation results are summarized in Table 1 and discussed below. Because of large
differences observed between the two compounds where the project was implemented –
George and Chawama – all results are shown by compound. The primary aim of the evaluation
was to investigate ‘the extent to which inequalities brought about by poor water access,
management, governance have been reduced and the significance of the intervention's
contribution’, as the final outcome indicator for all Sustainable Water ERs. We measured this
concept using a multidimensional Sustainable Water and Sanitation Index comprised of six
dimensions. We also assessed water and sanitation access as well as water insecurity, which
was measured using the Household Water Insecurity Experiences scale (HWISE, 2019).

Table 1: Summary of Effectiveness Review results

Outcome Evidence of impact in Evidence of impact in
George Compound Chawama Compound
Improved Water Access No Positive

Improved Sanitation Access No Positive

Reduced Water Insecurity (HWISE) Negative Negative

Sustainable Water and Sanitation: Mostly Negative Mixed

Overall

Sustainable Water and Sanitation: Negative No

Water Security

Sustainable Water and Sanitation: Negative Positive

Equity

Sustainable Water and Sanitation: Negative No

Institutions

Sustainable Water and Sanitation: No Mixed

Operations

Sustainable Water and Sanitation: Negative Negative

Well-being

Sustainable Water and Sanitation: Negative Negative

The Environment

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
5
Figures 1 and 2 show the project impact by compound for the index and each of the six
dimensions (top row) and 16 indicators (below each dimension). Overall, we find the project had
a negative impact in George Compound for the Sustainable Water and Sanitation Index (-8.62,
p<0.01) and Water Insecurity (HWISE: 6.34, p<0.01); no significant impact on water and
sanitation access is observed. On the other hand, in Chawama, we find the project had a
positive impact on access to piped water on premises (4.42, p<0.1) and improved sanitation
(8.95, p<0.01). Despite this, we do not see a significant impact on the Sustainable Water and
Sanitation Index overall (although several significant impacts are observed among the
dimensions and indicators) and we find a significant negative impact on Water Insecurity
(HWISE: 2.03, p<0.05).

Figure 1: Sustainable Water and Sanitation Index dimensions and indicators of project
impact in George Compound (PSM estimates; * p < 0.1, ** p < 0.05, *** p < 0.01)

Figure 2: Sustainable Water and Sanitation Index dimensions and indicators of project
impact in Chawama Compound (PSM estimates; * p < 0.1, ** p < 0.05, *** p < 0.01)

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
6
Based on subgroup analysis, we find differential impacts for the Sustainable Water and
Sanitation Index by gender and disability, but not by home ownership status. In George we see
a larger negative project impact among men than among women and in Chawama we see a
positive impact among households having members with a disability.

Throughout this report, tangible and intangible impacts are combined in the indicators and
dimensions of the Sustainable Water and Sanitation Index (see Table 2). Both the aspects are
important for measuring this concept, to capture differences ranging from access to
infrastructure to personal experiences and opinions. Physical infrastructure alone does not tell
the whole story because that system may not work for everyone all the time. Personal
experiences and opinions are key to understanding how people are really affected and what
factors are contributing the sustainability of water and sanitation systems.

Table 2: Example tangible and intangible aspects of the Sustainable Water and
Sanitation Index dimensions and indicators
Dimension Tangible Aspects Intangible Aspects
Water Security Water service interrupted or limited (low Worrying, feeling angry, ashamed,
water pressure, less water than excluded, stigmatized because of
expected), not enough water for problems with water
drinking, food preparation, washing
(hands, body, clothes, etc.)
Equity Women and men equally share Services are perceived to be equally
responsibilities for water collection and inclusive for diverse groups (for women
unpaid care work related to water (not and men, young and old, people with
measured in this evaluation) disabilities)
Institutions Access to information, taking action to Confidence in holding the government
hold the government and service and service providers to account,
providers to account for water and transparency of decision-making
sanitation systems and services process, trust in institutions
Operations Infrastructure access (type and Feeling safe accessing infrastructure,
location), existence of a water and confidence in the water and sanitation
sanitation committee, calculation of committee to carry out duties (O&M,
household expenses on water and repairs, replacements), opinions
sanitation service pricing and affordability
Well-Being Incidence of health problems Life satisfaction, feeling stressed,
(diarrhoea, fever, cholera, etc.), school subjective health
attendance of children

The Environment Existence of water safety and waste Implementation and awareness of
management plans (not measured in community water safety and waste
this evaluation), household water management plans
conservation and source protection
practices

This distinction between the tangible and intangible also helps with interpreting the impact of the
project. Most negative impacts are related to the intangible – worry and anger due to problems
with water, opinions about equity, confidence, satisfaction, trust, opinions on affordability and
fairness of pricing, feeling safe and subjective health. Meanwhile, we see little to no impact on
tangible aspects – water fee structure (including options for those who cannot pay), proportion
of monthly expenses going to water, sanitation and health, types of water and sanitation
services and systems used, physical terrain between the household and the water source,
hours of access per day, incidence of specific health issues, school attendance and measures
taken to conserve and protect water resources. This distinction is more apparent in George,
where we see more negative impacts, than in Chawama, where we see more positive impacts.

The KIIs highlight the role of specific project activities, such as water quality monitoring and
supporting community-based enterprises (CBEs). For example, there is evidence that water

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
7
quality monitoring is continuing and that the comparison area – Kanyama – is also benefiting by
occasionally borrowing the water quality monitoring kits provided in George and Chawama. As
for the CBEs, some are still active, but not as active as they used to be during the project. Some
CBEs are mentioned as still working with the Lusaka Water and Sewerage Company (LWSC).
At the same time, concerns were raised about solid waste management CBEs, some of which
were said to not be disposing of waste properly. Overall, water systems and services were
described as making good progress, with plans already underway to resolve issues such as
intermittent water supply. However, sanitation remains an immense challenge, with clarity on
how to set up safe, equitable and effective solid waste management systems still lacking.

In Chawama, the results are relatively straightforward – the project achieved several positive
impacts, although alongside some negative impacts. On the other hand, in George, the mostly
negative results may reflect increased awareness of water and sanitation issues among
residents and subsequent frustration with seeing little progress made to remedy those issues. It
is possible for mobilization to lead to negative results if residents are not listened to, or if those
listening have little power to make real changes in their communities. A power analysis would
help to illuminate such disconnects between the residents and duty bearers, extending beyond
the compound to district and national levels as well (Hopkins, Brady & Brownlie, 2014).

PROGRAMME LEARNING CONSIDERATIONS

Consider conducting a power analysis during programme
design and regularly revisit it during implementation
Based on the mixed impacts found through this evaluation – the same approach worked
relatively well in Chawama but not in George – future programmes could use a power analysis
as the basis for further tailoring project designs to specific populations and local structures. A
power analysis should be done for each community, to understand the of centres of power both
inside and outside that community (e.g., district and national level) and how these power
relations might alter project effectiveness. Regular reflections on this analysis should be done
with a broad cross-section of the community. Based on these reflections, the power analysis
can evolve (and strategies may need to change) over time as power structures (hopefully)
change throughout the course of the project.

Based on the findings in this evaluation, reflect on mobilization

strategies
While the project activities were the same in George and Chawama, mobilization appears to
have been more successful in Chawama because the project was able to work through
established local-level structures. In George, local-level structures had to be set up as part of
the project (e.g., Ward Development Committees – WDCs). The maturity of these local
structures likely influenced the impact of the project in each compound.

Also, based on the low levels of project exposure seen in this evaluation, the project could have
looked for additional ways to connect with more community members more often. Linking to the
first recommendation above, mobilization can also be part of the power analysis. For example,
look for ways to further develop mobilization strategies that account for how project
implementation may be affected by differences in existing community structures.

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
8
When shifting from rural to urban WASH programming, build in
training opportunities for staff to adapt their skills
This project was a first attempt at urban WASH programming in Lusaka. Together, Oxfam and
Village Water (implementing partner) had successfully carried out various rural WASH projects
in the past. Transitioning this work to an urban environment was both a new and different
challenge, as well as a learning experience. Communities within peri-urban Lusaka can have
very different experiences when it comes to water and sanitation systems and services.

Understanding this reflects a need for further training for staff when transitioning from rural to
urban WASH programming, to improve knowledge and skills for working in such a different
context. With shifts from rural to urban accelerating in the sector, driven at least in part by
increasing urbanization, strategies are often being adapted from rural to urban contexts. In this
process, ensure that training opportunities for staff are built in early on so they can adapt and
learn about aspects that differ between rural and urban WASH programming.

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
9
1 INTRODUCTION
Every year since 2011, Oxfam Great Britain (GB) has conducted rigorous impact evaluations
known as Effectiveness Reviews (ERs) as part of its Global Performance Framework. For these
reviews, we randomly select projects that have been active for at least two years and have a
minimum budget of £200,000. We look for evidence of a cause–effect relationship between the
project activities and any observed outcomes and impacts to understand whether our work
leads to positive changes in the lives of the women and men with whom and for whom we work.

For the financial year 2018/19, we selected from projects under five thematic areas –
Livelihoods, Women’s Empowerment, Resilience, Good Governance and Sustainable Water
and Sanitation. The ‘Promoting the right to safe water, sanitation and hygiene (WASH) of peri-
urban communities in Lusaka, Zambia’ project was selected under the thematic area of
Sustainable Water and Sanitation. Throughout this report, this project is referred to as the
‘Urban WASH’ project.

The Urban WASH project operated in two peri-urban areas of Lusaka – George and Chawama
Compounds – from July 2013 to June 2017. These areas are characterized by dense, mostly
informal settlement patterns, where water and sanitation access and management face ongoing
challenges. Flooding, common during the wet season due to insufficient drainage, coupled with
inadequate waste management, results in cyclical cholera outbreaks and other public health
issues. Project activities to address these (and related) concerns were carried out with our
implementing partner, Village Water. The overall budget was £1,098,813. An estimated 51,000
individuals benefited directly from the project, while 222,000 individuals benefited indirectly.

The ER data-collection phase, in November 2018, took place one year and five months after the
end of the project in June 2017. It is important to note that Lusaka had a severe cholera
outbreak from January to June 2018 (WHO Africa, 2018), to which Oxfam ran a humanitarian
response in three compounds – George, Chawama and Kanyama. This outbreak and the
response were key considerations in the evaluation design. Kanyama Compound was identified
as the best possible comparison area (comparison group), since it was also included in the
humanitarian response to the cholera outbreak and community characteristics were otherwise
similar to the project areas. Figure 1.1 shows the locations of these areas. Note that the
northernmost section of Kanyama is an industrial area and the southern section is residential,
which means the evaluation comparison area did not border George Compound as indicated on
the map.

The primary aim of the evaluation was to investigate ‘the extent to which inequalities brought
about by poor water access, management, governance have been reduced and the significance
of the intervention's contribution’, as the final outcome indicator for all Sustainable Water ERs.
The questions guiding the evaluation were:

1. How did the project impact water and sanitation access and Water Insecurity at the
household level?
2. How did the project impact Sustainable Water and Sanitation at the household level?
3. How do the perspectives of relevant institutional representatives relate to the
household-level experiences?
4. How do impacts differ:
a. by gender?
b. for households with disabled or chronically ill members?
c. by home ownership status?

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
10
Figure 1.1: Location of Lusaka in Zambia (left) and the intervention (George and
Chawama) and comparison (Kanyama) areas within Lusaka (right) (Eubank, 2014)

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
11
2 PROJECT DESCRIPTION
2.1 PROJECT ACTIVITIES
The project collaborated with local institutions on an array of activities, engaging stakeholders to
create a conducive environment for service provision and improving capacities and practices. A
summary of project activities follows, grouped by institutional stakeholder:

• Health Centres to (a) support Environmental Health Technicians (EHTs) in water

quality testing, through training and provision of test kits, (b) support volunteer
Community Health Workers (CHWs) with hygiene promotion training and supplies for
distribution (e.g., chlorine disinfection and soap), (c) generate Water and Sanitation
(WatSan) and disease surveillance reports, and (d) mobilize the communities.

• Schools to (a) support EHTs to teach pupils about personal hygiene and sanitary pad
disposal, (b) form sanitation clubs, (c) train teachers on key hygiene messages, (d) add
a hygiene guide to the curriculum, and (e) place advocacy billboards.

• Ward Development Committees (WDCs) to (a) coordinate interactive fora to bring

community members together with authorities, (b) form pressure groups, (c) train
community members in activism for reporting issues to service providers, and (d)
mobilize the communities.

• Zambia Federation of Disability Organizations (ZAFOD) to (a) identify people with

disabilities, (b) train community leaders on inclusive WASH, (c) conduct a WatSan
audit, (d) gather design input for inclusive WASH, and (e) provide inclusive WASH
demonstrations (models).

• Lusaka City Council – Public Health Department (LCC – PHD) to (a) train
community leaders on the Public Health Act (PHA), (b) train community-based
enterprises (CBEs) to do proper and safe solid waste management, and (c) support the
assessment of facilities for legal enforcement of the PHA.

• National Water Supply and Sanitation Council (NWASCO) to (a) share information
on tariff setting and legal requirements in the interactive fora and (b) facilitate regulation.

• Lusaka Water and Sewerage Company (LWSC) to facilitate information sharing and
establish new lines of communication between LWSC and communities.

• Zambia Police (ZP) to (a) provide security to CBEs and (b) support gender
mainstreaming activities and sensitization.

• District Health Office (DHO) to (a) support the district with supplies for distribution
(e.g., water quality test kits) and (b) facilitate awareness-raising events, including World
Water Day (WWD), World Toilet Day (WTD) and Global Handwashing Day (GHWD).

The project also supported city-wide activities, such as radio programming and roadshows,
placing advocacy billboards in market areas, and forming a WhatsApp group. These activities
were all focused on sharing and discussing information and issues related to WASH.

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
12
2.2 THEORY OF CHANGE
The project activities fit into three broad dimensions of change – (1) promoting an enabling
environment, (2) capacity building, and (3) improving practices. While these dimensions are
interconnected, the project did expect certain intermediate outcomes related to each dimension.
For promoting an enabling environment, the expected outcomes were improved communication
and awareness, improved stakeholder engagement, and inclusive WASH models. For capacity
building, the expected outcomes were: improved water quality testing, increased knowledge
about policies and hygiene at the individual and community levels, and improved WASH
practices (e.g., water collection, storage and treatment, handwashing and personal hygiene,
sanitation facilities, disposal of solid waste and wastewater, and stormwater drainage).

The project outcomes aimed for impact – positive change for people and their communities. The
expected impacts were improved health (e.g., reduction in diarrhoeal diseases, prevention of
disease outbreaks), improved school attendance (especially for girls), inclusive WASH facilities,
efficient WASH service provision, improved coordination, and raising the people’s voices and
their voices being heard in terms of WASH facilities and services. Many impacts are linked but
each also follows a distinct causal path from the project’s activities and expected outcomes.

Figure 2.2.1: The project’s Theory of Change (recreated with Oxfam and Village Water
during a workshop held October 30–31, 2018, Lusaka, Zambia)

The Theory of Change made assumptions about how change would happen. Any invalid
assumptions may reduce the impact of the project. The key assumptions raised during the
workshop were (1) WASH is a main factor in disease burden, (2) improving WASH practices will
reduce disease burden, (3) people would prefer to pay for safe water rather than drinking
unsafe water for free (i.e., people value safe water and they can afford it), (4) the government
will increase its investment in WatSan infrastructure, (5) resources will be available and
prioritized to WASH, (6) capacity is there, (7) engagement and communication will improve the
efficiency of WASH service delivery, (8) models will influence future WASH facilities, (9) people
will trust and believe new knowledge and information that is shared with them, and (10)
knowledge and awareness will lead to changes in practices.

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
13
3 EVALUATION DESIGN
The central problem in evaluating the impact of any programme is understanding what changes
are attributable to project activities versus what would have happened otherwise. In this
evaluation, the situation in project areas was examined through quantitative household surveys
and qualitative key informant interviews (KIIs) – but clearly, we could not directly observe what
the situation would have been without the project. This ‘counterfactual’ situation can only be
estimated.

Given a large number of households, we follow the common practice of estimating the
counterfactual by comparing households that were part of the project (intervention group) to
households that were not (comparison group). Assuming these two groups are the same,
except for the project, observing the situation in both groups provides a good estimate of the
counterfactual.

The ideal approach from an evaluation perspective would have been to randomly assign
households (or groups of households, communities, etc.) to the intervention and comparison
groups. This type of randomization minimizes the probability of systematic differences between
the groups and maximizes the confidence that any observed impacts were caused by the
project. However, this approach is not possible when evaluating projects that have already been
implemented and is often not ideal for large-scale implementation either.

Thus, we have adopted a ‘quasi-experimental’ evaluation design using propensity score

matching (PSM) to answer the evaluation questions for households in the intervention group in
contrast to similar households in the comparison group. The matching process was done with a
pre-defined set of baseline characteristics, including information about the interviewee,
household demographics, income sources, wealth, group participation, and water and sanitation
access. In this case, we also included a matching variable to account for the effect of the 2018
cholera outbreak. To ensure sufficient data for the matching process, we interviewed three
comparison interviewees for every two intervention interviewees.

The baseline data needed for PSM are not available, so survey interviewees were asked basic
questions about their situation from 2012 (the year before project implementation began in
2013), thereby creating recall data (Nicola & Giné, 2012; Godlonton et al., 2018). While recall
data may not be completely accurate, we do not expect it to bias the evaluation results because
systematic variation between how interviewees remember information in the intervention and
comparison groups is unlikely. Using recall data to recreate a baseline is not the ideal approach
(methodologically); we opt to use it as a second-best option (pragmatically) when sufficient
baseline data is not available.

To contextualize the household survey data, we also conducted community-level KIIs with
institutional representatives in each compound. We used this qualitative data to contextualize
quantitative results and further evaluate institutional differences between the intervention and
comparison groups related to the project activities. Overall, this evaluation design allows us to
see project impacts at the household and community level. Any impacts of broader activities
conducted across the entire local area (e.g., radio programming, roadshows, district-level
activities) are not explicitly evaluated.

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
14
4 DATA
4.1 INTERVENTION AND COMPARISON
For this evaluation, the intervention group includes George Compound (Lima Ward 26) and
Chawama Compound (Ward 2), which comprise the entire area where the project was
implemented. Through discussions with project staff, it was revealed that the project originally
decided to work with these communities based on the range of water and sanitation challenges
being faced, including their propensity to waterborne disease outbreaks, namely cholera.

The comparison group is Kanyama Compound (Ward 11), which has a similar settlement
pattern to George and Chawama and is understood to have faced similar water and sanitation
challenges back in 2012 (before the project), including cholera outbreaks. This comparison area
was selected in consultation with project staff and CSK Consortium, all of whom have extensive
experience working in peri-urban Lusaka. Project staff also explained that Kanyama would have
been likely to be included in the project if more resources had been available. Population
statistics for these areas are shown in Table 4.1.1. For a map refer to Figure 1.1.

Table 4.1.1: Population statistics for intervention and comparison compounds in 2018*
Group Compound Population* Number of Households* Average Household Size
Intervention George 59,600 9,600 6.2 people
Intervention Chawama 131,000 21,500 6.1 people
Comparison Kanyama 169,250 28,000 6.0 people
*Central Statistical Office (CSO) of Zambia, projected figures based on the most recent census in 2010.

Another factor considered in selecting the comparison area was a severe cholera outbreak that
occurred after the project was finished (but before this evaluation). It began in Lusaka in
October 2017, peaked in January 2018, and spread to seven provinces before concluding in
May 2018. In total, there were around 5,900 cases and 114 deaths (WHO Africa, 2018; CDC,
2018). Chawama, George and Kanyama compounds were all heavily affected. For example, of
210 new cases the week of 18 March 2018, Chawama sub-district had 22 (three in Chawama
Compound), Matero sub-district had 41 (27 in George Compound), and Kanyama sub-district
had 64 (28 in Kanyama Compound) (WHO Africa, 2018). These three sub-districts accounted
for over 60% of that week’s cases, with George most impacted considering population density.

The response to this outbreak involved many stakeholders working together with the Ministry of
Health (MoH) (e.g., Zambia National Public Health Institute, WHO, CDC, LWSC) through an
incident management system. Response activities, to which Oxfam contributed, included water
trucking, chlorine and soap distribution, and hygiene education. Additionally, shallow wells were
located and filled in and around 1 million people were vaccinated against cholera. According to
WHO Africa (2018), vaccination rates were higher in Chawama (97%) and Kanyama (142%)
sub-districts compared to Matero sub-district (82%), where George Compound is located.

Kanyama was the only non-project area that was also part of Oxfam’s response to this
outbreak, which occurred between the end of the project and this evaluation. While this was
expected to influence the results in absolute terms, we did not expect our estimates to be
biased due to the cholera outbreak as the subsequent response was seen as a constant factor
across the intervention and comparison groups. However, it is also possible that the outbreak
and response had disproportionate effects in the different compounds. To better understand
how our results might be influenced by these events, we include additional information in the
results section on the cholera outbreak gathered at both the household and institution level.

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
15
4.2 HOUSEHOLD SURVEYS
Following a three-day training course, a team of 12 interviewers (contracted and managed by
the evaluation consultant, CSK Consortium) conducted the household surveys from 13–28
November 2018. The questionnaire was reviewed in English and Nyanja and Bemba languages
during the training for translation quality and to develop a common understanding of all
questions among the team. All surveys were conducted digitally with SurveyCTO on mobile
devices (with daily uploading); paper questionnaires were available as a back-up. Throughout
data gathering, privacy and data protection were prioritized, following principles and protocols
based on Oxfam’s Responsible Data Policy and GDPR (Vonk, 2019).

In each compound, survey interviewees were randomly selected from all current residents using
a random walk protocol as follows:

1. From the health clinic, pairs of interviewers were assigned to zones in coordination with
the Environmental Health Community Group. From the approximate centre of the
assigned zone, they spun a pen (as a dial) to indicate in which direction to walk.

2. One interviewer counted households on the left and one counted those on the right.

3. At every 25th household, the interviewer asked to interview a household member of a

specified gender (randomly indicated by SurveyCTO) who was knowledgeable about
the household’s water and sanitation situation.

a. If a suitable interviewee was available and willing to participate in the survey,

the interviewer proceeded with the interview.

b. If a suitable interviewee was temporarily not available, the interviewer asked if

they could come back at a more convenient time.

c. If a suitable interviewee either did not exist in the household or was away for an
extended period, the interviewer proceeded to interview a interviewee of the
opposite gender, if possible.

d. In case of refusal, or if a suitable interviewee could not be found after two

attempts, the interviewer recorded the reason for no interview and proceeded to
the next household according to the protocol.

We aimed for a sample size of 1,000 (600 comparison, 400 intervention). Interviewee gender
was randomly indicated for each household, so we expect something close to a 50:50 ratio. In
total, the interviewers visited 1,031 households, of which 16 had no interviewee available (e.g.,
not at home, too busy) and one did not consent to be interviewed, thereby completing 1,014
surveys (631 comparison, 383 intervention). Most those unavailable were in Chawama (n=8)
and George (n=6). Of those surveys, 17 were dropped from analysis due to irreconcilable errors
in the household roster. The final sample of interviewees is shown in Table 4.2.1. Additional
summary statistics are provided in Appendix 2.

Table 4.2.1: Final sample of interviewees by compound and gender

Group Compound Total Sample (N) Men Women
Intervention George 182 85 (47%) 97 (53%)
Intervention Chawama 192 94 (49%) 98 (51%)
Comparison Kanyama 623 295 (47%) 328 (53%)
Total All 997 474 (48%) 523 (52%)

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
16
4.3 KEY INFORMANT INTERVIEWS
In each compound, representatives from the project’s compound-based institutional partners
were invited for interviews. From the list of project activities (Section 2.1), there were four
partners with representatives based locally in each compound. These were key stakeholders in
promoting an enabling environment and capacity building activities, as articulated in the
project’s Theory of Change. They were the local duty bearers the project sought to connect with
compound residents – the right holders – to discuss and address their WASH challenges. We
aimed to interview the following key informants in each compound:

• Environmental Health Technicians (EHTs) at the health clinic

• Health and nutrition (SHN) teachers at the local school

• Ward Development Committee (WDC) members

• Lusaka Water and Sewerage Company (LWSC) representative.

In George (intervention group), we completed interviews with representatives from the health
clinic and LWSC. However, due to scheduling conflicts, we were unable to complete the WDC
and school interviews in George. In Chawama (intervention group), we completed interviews
with representatives from all four institutions. In Kanyama (comparison group) we completed
interviews with health clinic and school representatives. However, due to security issues, we
were unable to complete the WDC and LWSC interviews in Kanyama. All interviews were
conducted independently by the evaluation consultant, CSK Consortium, following principles
and protocols based on Oxfam’s Responsible Data Policy and GDPR (Vonk, 2019). Table 4.3.1
summarizes which interviews were conducted by compound.

Table 4.3.1: Final set of KIIs by compound.

Group Compound Health Clinic Local School WDC LWSC
Intervention George X X
Intervention Chawama X X X X
Comparison Kanyama X X

The KIIs were conducted in an open-ended manner with a pre-defined set of talking points. For
all interviewees, we asked them to share information about specific topics (implicitly related to
the project) and to describe any challenges they have faced in carrying out their mandate. The
following is a list of the specific activities we asked about, by institution:

• Health Clinic – water quality monitoring activities, hygiene education activities.

• Local School – handwashing activities, sanitation club activities (if applicable),

inclusiveness of WASH facilities at the school, absenteeism of students (by gender and
reason, if possible).

• WDC – status of interactive fora (if applicable), activity of WASH CBEs, levels of
communication and stakeholder engagement, WASH representation and prioritization
within the committee.

• LWSC – current WatSan services provided in the compound, payment structures, plans
for expansion and improvement, revenue and non-revenue water information,
collaboration with or support of CBEs.

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
17
4.4 MATCHING PROCESS OVERVIEW
Because of large differences observed between the two compounds where the project was
implemented – Chawama and George – all quantitative analysis, including PSM, was done
separately by compound. Therefore, the sample of comparison households differs slightly for
George versus Chawama. Before PSM, we filtered the data based on residence duration to
exclude households that had moved to the community less than two years before the survey
(i.e., towards the end of the project or after it ended). This resulted in dropping 119 households
– 26 in George, 30 in Chawama and 63 in Kanyama – from the analysis. In the matching
process for George, data from two intervention and 119 comparison households were dropped
because no adequate matches existed. Similarly, in the matching process for Chawama, data
for seven intervention and 43 comparison households were dropped. Table 4.4.1 shows the
final sample sizes and gender distributions included in the analysis by intervention compound.

Table 4.4.1: Final sample of interviewees after PSM by compound and gender.
Comparison Total Sample Men Women
Compound Intervention
(Kanyama) (N) (Int. / Comp.) (Int. / Comp.)
George 150 441 591 74 / 206 76 / 235
Chawama 151 517 668 74 / 239 77 / 278

An overview of the most pertinent information from the PSM process and other descriptive
information is provided below. Further details on how we do PSM and full specifications for this
evaluation are provided in Appendix 3. In short, before matching, we find several significant
differences between intervention and comparison groups as shown in Table 4.4.2. By using
PSM, we can adjust for these differences when estimating impacts; when we checked the
balance variables after matching, we no longer find any significant differences.

One key matching variable we use is a wealth index, which is based on household ownership of
various assets (e.g., furniture, livestock, equipment) and the condition of the house in 2012.
When generating the index, we (1) verify internal consistency using Cronbach’s alpha, following
the guidance of Bland and Altman (1997), and (2) use principal component analysis (PCA) to
assign appropriate weights to each variable in the index, following the approach of Filmer and
Pritchett (2001). We ensure comparability of the wealth indexes from 2012 (based on recall
data) and 2018 (based on the household’s situation at the time of the survey), by pooling data
by time period before undertaking PCA. We use wealth index quintiles for PSM, to avoid over
constraining the matching process (i.e., households are matched to those with similar wealth).

The significant differences before matching can tell us a few important things. Many differences
between the intervention group and comparison group are similar for George and Chawama –
when compared with Kanyama, they both had slightly more household members in 2012, more
of them moved to their respective communities after 2012, more of them owned their home in
2012, fewer of the household heads and interviewees were in the household back in 2012, and
levels of education and literacy are lower among the household heads and interviewees.

In comparison to Kanyama, fewer households in George had improved water, piped water, or
improved sanitation in 2012 and they were also more affected by the 2018 cholera outbreak.
But in Chawama we see the opposite trend, that in 2012 those levels were higher than in
Kanyama (although the difference for improved sanitation is not statistically significant). We also
find opposite trends for group participation, household members being fit to work (an indicator of
disabilities), and wealth. Based on the 2012 wealth quintiles, we see that Chawama had more
wealthy residents (in the fifth quintile) in comparison to Kanyama. At the same time, George had
fewer wealthy residents (in the first quintile) than Kanyama (Figure 4.4.1 illustrates this point).

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
18
Table 4.4.2: Selected balance variables with significant differences before matching
Variable Difference between Difference between
George and Chawama and
Kanyama Kanyama
Interviewee age (years) 0.71 2.55*
Interviewee completed secondary school (%) -10.11* -4.74
Interviewee participated in a group in 2012 (%) -20.17*** 5.66
HH head age (years) 1.97 2.92*
HH head completed secondary school (%) -14.16** -9.63*
HH head is literate (%) -8.91** -7.66**
Number of HH members in 2012 0.54** 0.30
HH has at least 80% members over 5 years old fit to 6.17* -9.36**
work (%)
HH lived in the community in 2012 -3.87 -5.47**
HH owned their home in 2012 (%) 14.08** 20.42***
HH was in the lowest 20% of wealth distribution in 10.19** -3.81
2012 (%)
HH was in the highest 20% of wealth distribution in -1.65 12.96***
2012 (%)
Main drinking water source was improved+ in the dry -11.56*** 4.46**
season in 2012 (%)
Main drinking water source was piped on premises in -13.80*** 20.05***
the dry season in 2012 (%)
HH had an improved sanitation facility (e.g., toilet) in -32.72*** 2.03
2012 (%)
HH affected more by 2018 cholera outbreak relative 30.98*** -32.95***
to previous outbreaks (%)
Observations 716 722
* p < 0.1, ** p < 0.05, *** p < 0.01. Variables dated 2012 are estimates, based on recall data. Construction of the wealth
index is described in Section 4.4.

Figure 4.4.1: Wealth quintiles in 2012, by compound (descriptive only, before matching)

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
19
5 MEASURING SUSTAINABLE
WATER AND SANITATION
Oxfam GB’s Sustainable Water and Sanitation Strategy (Mizniak et al., 2017) states that to
achieve sustainability, we must ‘work with essential partners in a unified approach to connect
poor and vulnerable communities to water and sanitation services, ensuring they are working
equitably’ to ‘keep water and sanitation systems operational, accessible, and affordable’. For
Sustainable Water ERs, our aim is to investigate ‘the extent to which inequalities brought about
by poor water access, management, governance have been reduced and the significance of the
intervention's contribution’, as the final outcome indicator.

The Urban WASH project under review was specifically aimed at sustainably improving water
and sanitation facilities and services. Building on our approach for Effectiveness Reviews when
measuring other complex themes, such as Women’s Empowerment (Bishop & Bowman, 2014;
Lombardini et al., 2017) and Resilience (Hughes & Bushell, 2013; Fuller & Lain, 2015), we
gathered a set of ‘dimensions’ to create a household-level Sustainable Water and Sanitation
Index, as shown in Table 5.1. These dimensions are based on the following:

1. Oxfam GB’s Sustainable Water and Sanitation Strategy (Mizniak et al., 2017) and
Outcomes-based Monitoring Framework (Medland, 2018)

2. Sustainable Development Goal 6 (UN, 2018)

3. A review of existing frameworks and indexes for assessing Sustainable Water and
Sanitation (Banerjee & Morella, 2011; Bartram et al., 2014; Bratton & Gyimah-Boadi,
2016; Giné-Garriga & Pérez-Foguet, 2018; Kayser et al, 2013; Porteous, 2016; Shilling
et al., 2013; Thomas et al., 2018; WHO/UNICEF, 2017; Wilbur & Danquah, 2015).

Table 5.1: Six dimensions of Oxfam’s Sustainable Water and Sanitation Index
Dimension Definition
Water
Security* People have consistent and sufficient access to preferable and acceptable
water; water is not a source of worry, shame or anger.

Equity
WatSan facilities, services, and related management systems and decision-
making processes are inclusive of all genders and vulnerable users.

Institutions
WatSan duty bearers are accountable and transparent, levels of trust are
high, and there is an enabling environment for service provision.

Operations
WatSan systems and services have an asset management scheme in place
to enable easy and continuous access that is affordable and financially viable.

Well-Being
People are satisfied with life, they are healthy and unburdened by waterborne
diseases, and their children are attending school.

The
Environment Communities have plans in place for water safety, waste management, water
conservation, etc. and people are aware of and participate in these systems.

*Adapted from the Household Water Insecurity Experiences (HWISE) scale by applying a threshold and averaging 12
items, as indicated in Appendix 1 (HWISE, 2019; Young et al., 2019).

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
20
The goal is to capture all key aspects that are known or suspected to contribute to the
sustainability of water and sanitation systems and services. Having a single aggregate number
(the index score) gives a general sense of the water and sanitation situation of a population. At
the same time, we can look at different dimensions and indicators to understand possible
stronger and weaker points. In terms of impact, comparing scores for intervention and
comparison groups gives a more holistic measurement of our successes and failures (or both).

The index is designed to be adaptable to the given context. The six dimensions remain
constant, but the set of indicators used to measure each dimension can vary, although
indicators should not be removed simply because the project did not work on that aspect. The
adapted version of the index for this evaluation shown below was developed together with
Oxfam and Village Water staff and CSK Consortium. It is constructed using a total of 16
indicators that are assessed using 108 questions.

Figure 5.1: Sustainable Water and Sanitation Index dimensions (top row) and indicators

Each indicator is measured with one or more questions, where each question has a pre-defined
threshold of acceptability, adapted from the approach of Alkire and Foster (2011). A positive
result (at or above the threshold) receives a score of ‘1’, while a negative result (below the
threshold) receives a score of ‘0’. Each indicator is calculated by averaging the scores for the
individual questions and is reported as a percentage on a scale from 0 to 100. Similarly, each
dimension is calculated as the average of its indicators and the overall index is the average of
the six dimensions, again on a scale of 0 to 100. Detailed tables of questions and thresholds by
indicator are provided in Appendix 1 (see Table A1.1 through Table A1.6). Each table also
shows whether each indicator is directly related to the project’s Theory of Change.

We give equal weight to each dimension in the index; we have no theoretical justification to
claim one is more important than another. Within each dimension, we also give equal weight to
its indicators for the same reason. Therefore, individual questions are not given equal weight
within the index; some indicators are measured with one question while others are measured by
averaging 10 or more. Take the Equity dimension as an example. It has two indicators – Gender
Inclusive and Inclusive of Vulnerable Users. Gender Inclusive is the average value across the
following three variables: womanleaders, equalwat_gender, equalsan_gender. Inclusive of
Vulnerable Users is the average value across the following five variables: pwdleaders,
equalwat_disability, equalsan_disability, equalwat_children, equalsan_children. Then, Equity
would equal Gender Inclusive plus Inclusive of Vulnerable Users, divided by two.

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
21
With the lens of the index focused on households, we also acknowledge the importance of other
system-level, sub-national and national-level indicators that are key to monitoring overall
sustainability of water and sanitation in these compounds and beyond. However, to evaluate
project impact in this review, we mainly want to understand whether there have been any
meaningful differences in the lives of individual residents and their households.

Alongside the index, we also consider the following information in this evaluation:

• Household water and sanitation access (primary system or facility used)

• Household Water Insecurity (12-item HWISE scale)

• Institutional perspectives

• Information on the 2017–2018 cholera outbreak.

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
22
6 RESULTS
Here we present combined results from household surveys and key informant interviews (KIIs).
All quantitative information is based on two final datasets from 851 household surveys, after
filtering and propensity score matching (PSM) the data gathered as described in Section 4.4. All
qualitative information is based on eight KIIs, as described in Section 4.3. In this section,
significant impacts are highlighted in light green if positive and red if negative. Insignificant
results are not highlighted. Direct quotes from KIIs are shown in italics.

6.1 PROJECT EXPOSURE

In this section, we look at what proportion of interviewees said they had participated in water
and sanitation (WatSan) activities in the last five years (which would include the period of
project implementation) and in the last one year (after the project had ended). For those who
reported participating in activities, we asked follow-up questions to better understand project
exposure. We would expect interviewees in intervention areas – George and Chawama – to
have participated in project activities mainly with Village Water, although they may have been
made aware of Oxfam’s role. It is also possible that interviewees in all three compounds may
have participated in WatSan activities with Oxfam during the 2018 cholera outbreak response.

Overall, project exposure in the last five years appears to be low among a representative
sample of all households in each compound. In Chawama, less than 5% of interviewees said
they had participated, while in George that figure was only 1%. In the comparison area of
Kanyama, less than 0.5% of interviewees said they had participated. This means to some
extent Kanyama is a good comparison, with relatively higher levels of participation in WatSan
activities in the intervention areas (particularly in Chawama).

In Chawama, interviewees said the activities they participated in were organized by the
government, local groups, and NGOs, including Oxfam and Village Water. In George, they only
referred to NGOs, including Oxfam and Village Water, as activity organizers. In comparison, in
Kanyama, interviewees said activities were organized by the government, local groups, and
NGOS, including Oxfam (but not Village Water), likely related to cholera response.

Figure 6.1.1: Exposure to water and sanitation activities, by compound

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
23
6.2 WATER AND SANITATION ACCESS
To contextualize, we start with a descriptive
overview of the water and sanitation situation in
Water Source Categories
George, Chawama and Kanyama compounds.
(WHO/UNICEF, 2017, p. 9)
Figure 6.2.1 shows the proportion of households by
primary drinking water source during the dry season Piped on premises – water tap in the
(which was the current season during data house or on the plot/yard
collection). Water sources reported for the wet
season were mostly the same as those for the dry Improved – public tap, standpipe or
season so, for simplicity, we focus on the dry kiosk, tube well or borehole, protected
season throughout this report. dug well, protected spring, rainwater
collection
As noted in the previous section, the situations in
the two intervention compounds – George and Unimproved – unprotected dug well,
Chawama – were different before the project in unprotected spring, cart with a small
2012. Households in Chawama reported no tank/drum, tanker truck, surface water
unimproved sources and much higher rates of piped (river, dam, lake, pond, stream, canal,
water on premises (53%). Households in George irrigation channel), bottled water
were using unimproved sources at a higher rate
(15%) and fewer reported piped water on premises
(19%). In general, these trends persist in 2018.

Figure 6.2.1: Primary drinking water source (dry season) in 2012 (left) and 2018 (right),
by compound (descriptive only, before matching)

The KIIs help to illuminate the types of sources people have access to – boreholes and piped
water connections from the Lusaka Water Works (water treatment plant) – through Lusaka
Water and Sewerage Company (LWSC). Note that LWSC states that it is unable to meet the
current demand in George, while in Chawama it has enough supply to promote more individual
connections for piped water on premises.

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
24
 WATER SYSTEMS IN GEORGE AND CHAWAMA COMPOUNDS

‘We are providing piped water through a network around George. We have boreholes
that are the main source of the water that we supply…we use individual connections
and also communal taps. In terms of water, we are also trying to expand our water
sources because currently we are not able to meet the demand. We are looking at
additional water sources for George.’
– Lusaka Water and Sewerage Company Office Manager for George Compound

‘[Everyone] from Chawama, they get water from Lusaka Water. We have two
sources…one source coming from Lusaka Water Works feeding Chawama and we
also have another source, which is coming from the boreholes that we have in
Chawama. The [new connections] that we are putting in Chawama, the unit is
individual, for individual households. We are encouraging people to get individual
connections.’
– Lusaka Water and Sewerage Company Engineer for Chawama Compound

Figure 6.2.2 shows the proportion of households by

sanitation facility category. In 2012, improved Sanitation Facility Categories
access levels were lower in George (39%) and (WHO/UNICEF, 2017, p. 9)
higher in Chawama (74%) and Kanyama (71%). In
all three compounds, most households with Improved – pit toilet/latrine that is
unimproved sanitation reported using an closed/covered, flush toilet/latrine
open/uncovered pit toilet/latrine. Again, these trends
persisted in 2018. Of those reporting access to Unimproved – no facility (open
improved sanitation in 2018, many were sharing the defecation), pit toilet/latrine that is
facility with at least one other household. Those with open/uncovered
private improved sanitation were 54% in George,
33%, in Chawama and 32% in Kanyama.

Figure 6.2.2: Sanitation facility used most in 2012 (left) and 2018 (right), by compound
(descriptive only, before matching)

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
25
Sanitation stands out as one of the biggest challenges faced in these communities. While the
rates of improved water access are quite high, most households (77%) are using shared and/or
unimproved sanitation facilities. Overall, 35% report using unimproved facilities and of those
75% report sharing with other households. Of those that do have access to an improved facility,
65% report sharing with other households. As we can see through the KIIs, the institutional
stakeholders are aware of this situation. They spoke candidly about the challenges, including
open defecation, in their communities. In George and Chawama, they also talked about plans to
improve sanitation systems and services in the near future.

SANITATION SYSTEMS AND SERVICES: CHALLENGES

‘There is a lot that has to be done to improve water and sanitation in the community.
It’s not healthy for five households to share one toilet…when one member of a
household is using the latrine, the other members of the other households may be
stranded...there is a big shortfall of toilets, they really need toilets.’
– Guidance Teacher at Chawama Primary School

‘Kanyama sits on a rock. And they have to put up toilets. Toilets, pit latrines will
require space, and after some time the toilet gets full. They’ve got to put up at another
place. Where would they put it up? Because there’s no space. Pit emptying services
are there. It has helped to some extent. But also it has its own challenges.’
– Environmental Health Technician at Kanyama Health Clinic

‘There are active Community-Based Enterprises (CBEs), but not as active as it used
to be. There is no motivation now. We do not have an active formal CBE who is
collecting solid waste now. There are some who do this informally. But they are not
dumping this waste properly.’
– Ward Development Committee Representative in Chawama Compound

SANITATION SYSTEMS AND SERVICES: PLANS

‘With regards to George, we don’t have any sanitation services yet…we have a
project that is on board, that is Lusaka Sanitation, and George is one of the areas that
will be worked on in terms of sanitation. There will be 2,000 toilets that will be
constructed…there will also be faecal sludge management facilities, that will be
constructed.’
– Lusaka Water and Sewerage Company Office Manager for George Compound

‘In Chawama, we went through with some donors who want to do sewer
network…that’s why [you can see the sewer network is being laid]. That sewer, it will
come here. Otherwise, Chawama is also catered for… in the plans of doing sanitation,
sewer network.’
– Lusaka Water and Sewerage Company Engineer for Chawama Compound

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
26
Now, we estimate the impact of the project on water and sanitation access using propensity
score (PS) weighted difference-in-difference estimates (details provided in Appendix 3). Table
6.2.1 shows the results for improved water access, piped water on premises, and improved
sanitation for George, we do not find any statistically significant impacts. In Chawama, we do
see positive impacts for increased access to piped water (4.42 percentage points) and improved
sanitation (8.95 percentage points).

Table 6.2.1: Impact of the project on water and sanitation access

Improved water+ Piped water Improved
(%, dry season) (%, dry season) sanitation (%)
Difference (Impact): George 1.87 (1.24) 3.55 (1.78) 3.30 (1.44)
Difference (Impact): Chawama 0.52 (0.76) 4.42* (2.50) 8.95*** (3.95)
Observations (intervention): George 150 150 150
Observations (intervention): 157 157 157
Chawama
Observations (total) 851 851 851
+During the dry season; improved water includes piped water on premises. * p < 0.1, ** p < 0.05, *** p < 0.01; PSM
estimates are bootstrapped with 1,000 repetitions. Standard errors are in parentheses.

In reviewing differences between 2012 and 2018 for these three indicators, overall changes in
the situation were modest. For example, only 4% of households in George, none in Chawama
and 1% in the comparison area reported changing from unimproved in 2012 to improved water
access in 2018 – it is not surprising we do not see any impact on improved water. For piped
water, the figures are similar, although several households in the comparison area reported
change in the opposite direction, from having piped water on premises in 2012 but not in 2018.
That was also the case in the comparison area for sanitation, with 6% of households going from
improved back to unimproved sanitation (this did not happen in the intervention areas, except
for one household in George). This suggests that, while increases in piped water and improved
sanitation were small, the project appears to have contributed to increased sustainability of
these systems, largely preventing reductions in improved access.

6.3 WATER INSECURITY

Next, we look at household Water Insecurity using the
Household Water Insecurity Experiences (HWISE) Household Water Insecurity
scale. The 12-item HWISE scale is a standardized set Experiences (HWISE) scale
of 12 questions that has been tested for reliability and
validity (HWISE, 2019; Young et al., 2019). The full list The 12-item HWISE scale (0 to 36)
of HWISE scale questions can be found in Appendix 1 is based on the following possible
(see the first 12 questions for the Water Security responses to a set of standardized
dimension; ignore the threshold column). Households questions about the frequency of
receive a score for each question based on their water insecurity incidences over
response. To calculate the scale, each question the past month (HWISE, 2019):
receives a score from zero to three and then the
scores for all questions are added together. The final 0 = Never (0 times)
12-item scale ranges from zero to 36, where higher 1 = Rarely (1–2 times)
scores indicate greater Water Insecurity. 2 = Sometimes (3–10 times)
3 = Often or Always (>10 times)

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
27
Table 6.3.1 shows project impact on Water Insecurity, based on PSM estimates for HWISE.
Overall, Water Insecurity is much higher in George compared with Chawama. In both
intervention compounds, the results suggest that Water Insecurity has increased (negative
finding, given the aim to decrease Water Insecurity). This negative impact, which is larger in
George (6.34) than in Chawama (2.03), is statistically significant in both cases.

Table 6.3.1: Impact of the project on Water Insecurity (HWISE) by compound

George Compound Chawama Compound
Intervention group mean 16.77 9.84
Comparison group mean 10.55 7.79
Difference (Impact) 6.34*** 2.03**
Standard error (1.23) (0.95)
Observations (intervention group) 136 138
Observations (total) 529 610
* p < 0.1, ** p < 0.05, *** p < 0.01; PSM estimates are bootstrapped with 1,000 repetitions.

In Section 6.2, we did not see any significant differences in water access in George. The KIIs
from institutional representatives in George show agreement, that demand is not currently being
met and intermittent water supply is a known issue. At the same time, we did see a positive
impact on improved water access in Chawama. The KIIs do not indicate the same level of
challenges in meeting demand as those in George, although they do talk about ongoing issues,
especially with low water pressure. Given the project’s aim to raise people’s awareness of water
issues, it is possible that this actually led to an increase in experiences of Water Insecurity, in
the absence of any big improvements in water supply coupled with ongoing challenges, such as
intermittent supply and low water pressure.

LINKS TO WATER INSECURITY IN GEORGE COMPOUND

‘Everyone is receiving but not really 24 hours. A few areas maybe receive 24 hours,
but on average it’s about 8 to 12 hours. So, we are looking at increasing our water
sources [to] improve our supply hours.’
– Lusaka Water and Sewerage Company Office Manager for George Compound

‘The other issue is of inconsistent [water] supply. Most of these communal taps, the
report which we have been receiving and the complaint is that frequency of water
supply at these communal taps is a little bit of a challenge. They open the taps maybe
at five, then by seven they are closed. Two hours, in the morning. Then two hours in
the evening.’
– Environmental Health Technician at George Health Clinic

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
28
 LINKS TO WATER INSECURITY IN CHAWAMA COMPOUND

‘Water is there 24 hours. Now, the problem is the network. Pipes are blocked. Some
pipes are not there. But water is there. It’s calcium [from the boreholes]. Those fed by
the boreholes, we have a problem with calcium. Water is there in Chawama, but the
network is old.’
– Lusaka Water and Sewerage Company Engineer for Chawama Compound

‘Accessing clean water is a challenge in some areas because the tap has less
pressure.’
– Ward Development Committee Representative in Chawama Compound

In Table 6.3.2, we review the differences in each of the 12 items in the scale to better
understand which aspects are driving increased Water Insecurity in each compound (see Table
6.4.2). In George, the largest differences are seen for items 1 (worrying about not having
enough water to meet all needs), 2 (experiencing interrupted or limited water supply), 3 (not
having enough water to wash clothes), 4 (having to change schedules or plans due to water
problems) and 9 (feeling angry about the water situation). In Chawama, the largest differences
are seen for three of the same items – 1, 2 and 9 (worrying, experiencing interruptions and
feeling angry).

Table 6.3.2: Impact of the project on Water Insecurity (HWISE) for each item
George Compound Chawama Compound
HWISE Worry (item 1) 0.66*** (0.11) 0.46*** (0.10)
HWISE Interrupt (item 2) 0.68*** (0.11) 0.38*** (0.11)
HWISE Clothes (item 3) 0.58*** (0.13) 0.15 (0.11)
HWISE Plans (item 4) 0.82*** (0.11) 0.06 (0.10)
HWISE Food (item 5) 0.44*** (0.14) -0.14 (0.10)
HWISE Hands (item 6) 0.30** (0.12) -0.07 (0.09)
HWISE Body (item 7) 0.40*** (0.13) 0.00 (0.09)
HWISE Drink (item 8) 0.20 (0.14) -0.04 (0.09)
HWISE Angry (item 9) 0.55*** (0.13) 0.31*** (0.11)
HWISE Sleep (item 10) 0.08 (0.15) 0.05 (0.08)
HWISE No Water (item 11) 0.30** (0.14) 0.02 (0.10)
HWISE Shame (item 12) 0.36*** (0.12) 0.07 (0.10)
* p < 0.1, ** p < 0.05, *** p < 0.01; PSM estimates are bootstrapped with 1,000 repetitions. Standard errors are in
parentheses.

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
29
6.4 SUSTAINABLE WATER AND SANITATION
In terms of Sustainable Water and Sanitation, the results of the PSM estimates for the
multidimensional index introduced in Section 5 (Table 6.4.1) indicate that the project had a
significantly negative impact on the overall index in George, while there was no significant
impact in Chawama. In George, these results are driven by significantly negative impacts in five
dimensions. In Chawama, there are mixed results among the dimensions, with a significant
positive impact for the Equity dimension and a significant negative impact for the Well-Being
dimension, resulting in a null result overall.

Table 6.4.1: Impact of the project on the Sustainable Water and Sanitation Index
George Compound Chawama Compound
Intervention group mean 38.77 54.74
Comparison group mean 47.39 53.80
Difference (Impact) -8.62*** 0.94
Standard error (1.52) (1.49)
Observations (intervention group) 150 151
Observations (total) 585 662
* p < 0.1, ** p < 0.05, *** p < 0.01; PSM estimates are bootstrapped with 1,000 repetitions.

In contrast, from the KIIs, an EHT in George spoke about the sustainability of their programme.
They described ongoing community engagement, skills development and continuation of
activities that began as part of the project (e.g., water quality monitoring and hygiene
education). However, in Chawama the WDC explained that there has been a decrease in
activity since the end of the project, mainly due to limited incentives and resources.

ON SUSTAINABILITY

‘George is very vast, as us individuals we can’t manage. We’ll just be scratching on

top. So that’s why [there is] community involvement so that sustainability of the
programme is there…it’s a little bit difficult to say ‘this is the impact’…there were
trainings which were taking place, building skills in the community, there was proper
community engagement. And those skills have remained in the community, of which
they are still helping us right now...there’s that sustainability because of the impact
which had remained, the knowledge is there, definitely. The community is aware and
there’s that continuation phase of implementing these activities.’
– Environmental Health Technician at George Health Clinic

‘The WDC is not very active now, people come and go for their own gain but if it
doesn’t work they leave their projects dormant…Now we are continuing the project
activities with the community, but voluntary and without resources. It’s not enough.’
– Ward Development Committee Representative in Chawama Compound

The overall impact for each dimension is shown by compound in Tables 6.4.2 to 6.4.7. For a
visual overview of the index and impact measurements, see Figure 6.4.1 (George) and Figure
6.4.2 (Chawama). Each dimension is also discussed in more detail, focusing on significant
differences observed. We also weave in quotes from the KIIs based on thematic links to each
dimension (e.g., accessibility for people with disabilities in relation to Equity, water quality
monitoring in relation to The Environment).

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
30
Figure 6.4.1: Sustainable Water and Sanitation Index dimensions and indicators of
project impact in George Compound (PSM estimates; * p < 0.1, ** p < 0.05, *** p < 0.01)

Figure 6.4.2: Sustainable Water and Sanitation Index dimensions and indicators of
project impact in Chawama Compound (PSM estimates; * p < 0.1, ** p < 0.05, *** p < 0.01)

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
31
Water Security
In George, the overall impact for Water Security (-18.55) shows increased Water Insecurity,
confirming what we saw with the HWISE scale. In Chawama, Water Security is higher (i.e.,
Water Insecurity is lower) relative to George, but we find no significant impact of the project.

Table 6.4.2: Impact of the project on the Water Security dimension

George Compound Chawama Compound
Intervention group mean 27.00 57.75
Comparison group mean 45.55 54.84
Difference (Impact) -18.55*** 2.88
Standard error (4.48) (3.89)
Observations (intervention group) 150 150
Observations (total) 585 661
* p < 0.1, ** p < 0.05, *** p < 0.01; PSM estimates are bootstrapped with 1,000 repetitions.

Equity
For Equity, the negative impact in George (-15.68) is driven by negative results in both
indicators – Gender Inclusive (-18.68) and Inclusive of Vulnerable Users (-12.81). The data
suggest this is due to lower agreement that water and sanitation needs are being met equally
for all genders and vulnerable people, including children and people with disabilities and chronic
illness.

In Chawama, we see a positive impact for Equity (14.20), which includes a significant impact for
Gender Inclusive (10.76) and Inclusive of Vulnerable Users (17.64). The data suggest the
opposite of that which was seen in George, that there is higher agreement that water and
sanitation needs are being met equally for all genders and vulnerable people.

In the KIIs we heard about ongoing inequities – violence against women and the lack of
accessible facilities at schools – in Chawama and Kanyama, but there were no such references
in George, and overall equity did not emerge as a major theme in the KIIs.

Table 6.4.3: Impact of the project on the Equity dimension

George Compound Chawama Compound
Intervention group mean 38.73 61.74
Comparison group mean 54.41 47.54
Difference (Impact) -15.68*** 14.20***
Standard error (3.30) (3.37)
Observations (intervention group) 150 151
Observations (total) 585 662
* p < 0.1, ** p < 0.05, *** p < 0.01; PSM estimates are bootstrapped with 1,000 repetitions.

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
32
 ONGOING INEQUITIES

‘Sometimes people were opening the tap late at night and then there were issues of
violence against women at those later hours.’
– Ward Development Committee Representative in Chawama Compound

‘When it comes to inclusiveness, I should admit that we haven’t done much in that
area. Our buildings do not have facilities for the physically challenged. But for our
classrooms, we have done something… we haven’t yet extended this exercise to the
[sanitation facilities].’
– Guidance Teacher at Chawama Primary School

‘The facilities are not suitable for learners with [physical] disabilities…and we have
such pupils.’
– Guidance Teacher at Kanyama Primary School

Institutions
For Institutions we find a negative impact in George (-4.01), with three indicators driving this
trend – Accountability (-7.49), Transparency (-7.28) and Trust (-6.64). At the same time there
was a positive, but not statistically significant, impact for the Enabling Environment indicator.
The data suggest lowered confidence in being able to hold duty bearers to account for improved
water and sanitation services, reduced understanding of how decisions are made regarding the
water and sanitation management, less satisfaction in feeling that their needs are considered in
decision-making processes and lowered trust in local government, water and sanitation service
providers, and the Area Councillor.

In Chawama, there is no significant impact overall for Institutions. The data does suggest
slightly more satisfaction with decision-making processes for water and sanitation systems and
services management and increased trust in community-based organizations (CBOs),
community-based enterprises (CBEs) and the Area Councillor.

From the institutional perspective in the KIIs, according to representatives from clinics, schools
and the service provider, they are working together with the community and have networks and
systems in place to manage concerns and complaints. However, challenges remain in serving
such large communities with limited resources. They also cited discrepancies between
knowledge and practice and explained that complaints are not always channelled efficiently.

Table 6.4.4: Impact of the project on the Institutions dimension

George Compound Chawama Compound
Intervention group mean 25.85 37.64
Comparison group mean 29.86 38.82
Difference (Impact) -4.01* -1.19
Standard error (2.16) (2.13)
Observations (intervention group) 150 151
Observations (total) 585 662
* p < 0.1, ** p < 0.05, *** p < 0.01; PSM estimates are bootstrapped with 1,000 repetitions.

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
33
 CHANNELLING CONCERNS AND COMPLAINTS

‘We work closely with the community in general. Because when we just start with the
selling of our water, or the product we are selling, we have vendors [including CBEs]
who we identify through community leaders. So, we have the WDC, the water
committees, the churches that we are working with who identify representatives, and
those representatives make up the water committee. So, the water committee
facilitates whatever activities between Lusaka Water and the community. If we have
any kind of information we work with the water committee to go and disseminate this
information to the community. Equally, if the community have got concerns, they come
through the water committee and then they are put forward to Lusaka Water.’
– Lusaka Water and Sewerage Company Office Manager for George Compound

‘We have a water committee, which we have in Chawama. We are working with them.
They are helping us distribute bills, they are helping us sensitizing, those are the
people that know the community. These are the people we are using. We pay them,
as Lusaka Water [to do this work].’
– Lusaka Water and Sewerage Company Engineer for Chawama Compound

‘[For our] human resources, we are adequate. The way we’ve done it now, even the
health post that never used to have environmental health personnel, there are now
environmental health personnel. So, accessibility and responding to complaints as
well as finding out what’s going on in the community has been improved from that
aspect. The challenge which used to be there, although it’s still there, was the issue of
transport. Maybe we receive a complaint which is very far and now to move from here
to that side to go and assess the situation, that was a challenge. But as of now,
decentralizing and making sure that these health posts have got environmental health
personnel [has helped].’
– Environmental Health Technician at George Health Clinic

‘The sanitation club is very active, it was formed in February or March of 2018. Before
that there was no sanitation club. We’ve formulated small groups in the school, a
sanitation club, who may talk on behalf of the learners. They’ve approached us to say,
“we don’t have soap in the toilet or at the handwashing facility”.’
– Guidance Teacher at Kanyama Primary School

‘At times, they would even show you, because the way those water kiosks were built,
they were built to have a standpipe on top. But the water pressure was so low that
water would not come out on top, so most of those they had to remove the standpipe
and put a [hose] which can easily bend so [the water comes out]. So, they
complained, but the complaints are channelled to people like us. But even if they were
to be channelled to Lusaka Water, somehow, people have gotten used to the
situation. They’ve adapted. Because this situation has been like this, it has become
normal to the people to just go for that. And that’s why they’ve found comfort in
shallow wells, they’ve found comfort in the same water that we say is not safe.
Because for them, they feel even if they talk, even if they demand for safe water, it’s
just not there.’
– Environmental Health Technician at Kanyama Health Clinic

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
34
Operations
For Operations, we do not see any significant impacts in George for this dimension or any of its
indicators. The data suggest the null result is due to mixed experiences. For example,
households in the intervention group were more likely to report using an official water source,
have better terrain between their house and water source and spend a reasonable amount on
water and sanitation (less than 5% of their monthly expenses for each). At the same time, they
were less likely to have access to more than one improved water source, use an official
sanitation facility or feel safe while collecting water and accessing their sanitation facilities and
more likely to say water and sanitation services are unfairly priced and that it is difficult to come
up with the money to pay for sanitation. They also reported having more confidence in their
water and sanitation committee to manage water system repairs but less confidence for
operation and maintenance (O&M), system replacement and managing sanitation systems
through a shock or crisis.

In Chawama, we also see no impact overall for Operations, although we do observe a

significant positive impact for Financial Viability (6.27) and a significant negative impact for
Asset Management (-11.36). In terms of Financial Viability, it appears that households in the
intervention group were more likely to value paying for water and sanitation, spend a
reasonable amount on water and sanitation (less than 5% of their monthly expenses for each)
and say that both are affordable and fairly priced. For Asset Management, they had less
confidence in their water and sanitation committee to do O&M for water systems and system
replacement for both water and sanitation.

Table 6.4.5: Impact of the project on the Operations dimension

George Compound Chawama Compound
Intervention group mean 48.54 61.21
Comparison group mean 49.82 62.91
Difference (Impact) -1.28 -1.70
Standard error (1.43) (1.24)
Observations (intervention group) 150 151
Observations (total) 585 662
* p < 0.1, ** p < 0.05, *** p < 0.01; PSM estimates are bootstrapped with 1,000 repetitions.

The KIIs provide examples of water and sanitation payment systems in each compound.
According to the service providers, demand for water is high and households are willing to pay,
especially for piped water on their premises. For sanitation, the KIIs highlight a challenge – even
for those paying for waste collection, it is not always disposed of safely – the problem is simply
being moved from one place to another. Issues of vandalism were also raised, which adds
additional challenges in terms of asset management, when resources are directed to fix such
damages less is available for regularly scheduled O&M, repairs and replacements.

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
35
 PAYING FOR WATER AND SANITATION

We have a rising block tariff…we want to cover [most] people between zero to six
[cubic metres], and then there is another band from six to about 30 [cubic metres],
and then anyone who consumes above 30 [cubic metres] has a different amount, so
you pay as you consume. So, this is for the metered customers. Then we also have a
fixed tariff for those who are not metered…in two folds – the ones who are using the
communal taps are paying about 30 kwacha per month per household and then those
who have individual connections, on average, they are paying about 76 kwacha per
month…Then, [at] communal taps where we are using bucket sales, that is a 20-litre
bucket at 10 ngwee. So, depending on how much somebody’s able to spend on water
then they can get an equivalent of the buckets.’
– Lusaka Water and Sewerage Company Office Manager for George Compound

‘In Chawama, we have fixed and there are some who are metered. So, now our
programme is to meter everyone… People, they are hungry [for water connections].
The challenge is we need to do the network so that people have water… Otherwise
we need to go on the ground. In this area, there’s no water. In this area, there’s water.
And they are eager to pay.’
– Lusaka Water and Sewerage Company Engineer for Chawama Compound

‘Collection of that waste became a problem. The companies would say “we are willing
to collect but the community is not willing to pay”. The community would say “we are
willing to pay but the companies are not willing to collect”. They would give us
evidence, for example, to say, what they do is they…collect the money…and then
they don’t come to collect. So, what happens is they now opt for those who just pass
with wheelbarrows. But where they take, they just transfer the problem from your
place and push it to another place…there is a heap where the council will come and
collect but people still go and heap there. So, there are such areas…and that is where
the biggest issue is.’
– Environmental Health Technician at Kanyama Health Clinic

ISSUES OF VANDALISM

‘We have an issue of vandalism…there’s just some extent of vandalism especially

where it concerns our communal taps. Sometimes when there’s low pressure,
sometimes the community takes matters in their own hands. They would think maybe
by removing the meter the pressure would improve. So, that is giving us a lot of work
because now we have to find materials, fittings to go and reinstall that meter.’
– Lusaka Water and Sewerage Company Office Manager for George Compound

‘The major [challenge] is vandalism by community members. If I take you around the
school you will find water is just running from the taps. Taps are bought 10 times or
more in a year.’
– Guidance Teacher at Kanyama Primary School

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
36
Well-being
For Well-being, we find a negative impact in George (-3.44). The indicator for Health (-4.45)
indicates a significant negative difference, largely because intervention group interviewees rated
their overall health lower and reported more often that they in the past month they had felt so
unwell that it interrupted their usual activities. The impact for Well-being (-4.26) is also negative
in Chawama, but largely due to a large significant negative impact for Life Satisfaction (-12.50).

Table 6.4.6: Impact of the project on the Well-being dimension

George Compound Chawama Compound
Intervention group mean 59.86 67.55
Comparison group mean 63.29 71.81
Difference (Impact) -3.44** -4.26**
Standard error (1.74) (2.12)
Observations (intervention group) 150 151
Observations (total) 585 662
* p < 0.1, ** p < 0.05, *** p < 0.01; PSM estimates are bootstrapped with 1,000 repetitions.

Well-being was a less common theme in the KIIs, except for education in terms of school
absenteeism. Representatives from the schools explained that, in the cases of erratic water
supply, girls and boys may stay away from school to collect water for household needs.
Additionally, boys often miss school to gather water for vendors (to earn money). They said that
even having relatively good facilities as the schools does not stop students from missing school
due to water and sanitation issues in the community.

SCHOOL ABSENTEEISM

‘There is erratic water supply in the community. For two or three days, some locations
have dry taps. Pupils would be sent by their guardians, by their parents, to go fetch
water from alternative sources. When they are sent, they could stay away from school
on that day because they are tending to chores that day. Cases of absenteeism
cannot be attributed to lack of water in the school because we usually have water. But
it could be due to challenges they have in the community where they live. From
experience, what I have seen, usually it is the boys who would want to stay away from
school because some of them would go out to do some piece work. They draw water
for [vendors]. They pay boys to go and draw water for them. It is usually on Mondays
and Fridays when we have the open market days in Chawama. There are so many
people that want to take advantage of this day, this open market. Some of our boys go
to make some money.’
– Guidance Teacher at Chawama Primary School

‘Absenteeism, our school is small compared to any school in our area. We have
around 1,000 plus, in the morning we have around 700 and 700 in the afternoon.
Absenteeism is around 10%, in a week. Challenges include distance, they are sent for
businesses by their parents. The reasons are similar for both boys and girls. I’m
inclined to say that absenteeism is increasing because of the population growing.’
– Guidance Teacher at Kanyama Primary School

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
37
The Environment
For The Environment in George (-8.77), the overall dimension is negative, along with two
indicators – Water Safety (-11.86) and Waste Management (-14.40). Interviewees in the
intervention group were less likely to report being aware of any of the following in their
community: water safety plans, drainage plans, sanitation system monitoring, solid waste
management plans or water use planning for domestic and productive demands.

In Chawama, we do not see a significant impact for The Environment overall, but Water Safety
(-7.81) and Waste Management (-7.25) do show significant negative impacts. In the intervention
group, interviewees were less likely to say their community has sanitation system monitoring or
water use planning for multiple needs, such as domestic and productive uses.

Water safety is not a prominent theme in the KIIs, but there is some recognition that the current
sanitation issues pose a threat to groundwater protection. Representatives from all three
compounds did share details on water quality monitoring – recall that the project supported
environmental health technicians (EHTs) in water quality testing, through training and provision
of test kits – including working with equipment and protocols. On this point, we note a possible
spillover effect – in the comparison area, they said they work together with George and
Chawama for water quality monitoring because in Kanyama they do not have their own test kits.

Table 6.4.7: Impact of the project on the Environment dimension

George Compound Chawama Compound
Intervention group mean 32.67 42.53
Comparison group mean 41.43 46.88
Difference (Impact) -8.77** -4.35
Standard error (3.72) (3.11)
Observations (intervention group) 150 151
Observations (total) 585 662
* p < 0.1, ** p < 0.05, *** p < 0.01; PSM estimates are bootstrapped with 1,000 repetitions.

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
38
 WATER SAFETY AND WATER QUALITY MONITORING

‘We are mindful that eventually there may be some form of contamination. For us, with
the incoming of Lusaka Sanitation, we are so happy. Because at least we’ll be able to
protect our groundwater.’
– Lusaka Water and Sewerage Company Office Manager for George Compound

‘We do [water quality monitoring] in all the zones through engagement with the
community, we routinely carry out random water point sampling for bacteriological and
chemical analysis. We take the samples to the food and drugs lab for analysis…the
results take at least two weeks to get back because they do analysis for the whole of
Zambia [everyone takes their samples there, so we must wait in line]. We do have the
[equipment here at the clinic], but the challenge we’ve had to do our own analysis
here is due to the media, the reagent. Chemical analysis is firstly pH and residual
chlorine; sometimes we also check for contaminants such as zinc and lead.’
– Environmental Health Technician at George Health Clinic

‘To start with, water quality monitoring, what is done at the moment. We usually go in
the community visiting the water sources that we have, where people have access to
water. Then, the monitoring that is done there, we have gadgets that we usually use,
we have a digital one which just measures chlorine residual. Then we use the
Lovibond also to also check for chlorine residual as well as the pH. For suspected
water points that could have been contaminated…we also use the H2S kits...[if it is
turned, we do further analysis]. Even before we receive the results, there are actions
that are put in place that we take. We sensitise [those] accessing the water point,
giving them health education, not only that, we provide chlorine to them [and]
emphasise the point whereby in a situation where they don’t have chlorine they have
to boil water.’
– Environmental Health Technician at Chawama Health Clinic

‘We don’t have [our own laboratory]. But then we have sister facilities, like George and
Chawama, that have portable labs…we can go and test within their clinic just to see if
water is safe. This we do routinely. Because we cannot know at what point water will
be contaminated. So, it’s more like a routine.’
– Environmental Health Technician at Kanyama Health Clinic

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
39
6.5 DIFFERENTIAL IMPACTS BY SUBGROUP
We used PS-weighting with an interaction term to check for differential impacts by subgroup, to
understand who may have experienced the effects of the project more or less (more details are
provided in Appendix 3). In this section, we look at impact in terms of the Sustainable Water and
Sanitation Index for three subgroups – (1) by interviewee gender, (2) by whether households
have disabled or chronically ill members and (3) by home ownership status. For each subgroup,
summary results are provided by compound below. Differential impacts are highlighted in blue.

First, we looked at the results by gender – do women and men experience impacts differently?
In both George and Chawama, 51% of interviewees were women. In Table 6.5.1, we see that
women have lower average Sustainable Water and Sanitation scores than men, irrespective of
the project. In George, the impact of the project was significantly different for men (-12.10) and
women (-6.34; not shown in table), mainly due to significantly more negative impacts for men in
terms of Water Security and Well-being. In Chawama, we do not see a significant difference
between women and men for the Sustainable Water and Sanitation Index overall, but we do for
Operations, for which men did not experience a significant impact but women did (-6.35).

Table 6.5.1: Impact of the project by interviewee gender

Sustainable Water and Sanitation Index George Compound Chawama Compound
Effect of being a woman in the comparison group -3.87* -2.19
(2.11) (2.38)
Project impact for men -12.10*** 1.81
(1.92) (1.95)
Differential project impact for women and men 5.76** -1.58
(2.67) (2.93)
* p < 0.1, ** p < 0.05, *** p < 0.01; PSM estimates are bootstrapped with 1,000 repetitions. Standard errors are in
parentheses.

Next, we reviewed how vulnerable groups, namely those with disabilities or chronic illnesses,
have been impacted by the project. Our data are insufficient to consider only households that
reported this directly because the subgroup is too small (less than 2% of households). Instead,
we formed subgroups based on households reporting that (1) 80% or more members are ‘fit for
work’ or (2) fewer than 80% are. ‘Fit for work’ is indicated by interviewees for all household
members over the age of five when asked whether they are ‘fit and able to do domestic or
livelihood work now if they wanted to?’ We consider ‘not fit for work’ as a proxy for disability,
recognising it an imperfect measure.

In George 38% of interviewees reported less than 80% of household members are fit to work,
while in Chawama 44% did. In Table 6.5.2, we do not find differences between the two
subgroups in the comparison group for Sustainable Water and Sanitation Index scores. We do
see a significant negative impact in George and positive impact in Chawama, among
households having members with a disability. In George, the negative impact is not significantly
different between subgroups. However, in Chawama, we do find a significant differential impact.
It shows that the impact among households that do not have members with a disability
experienced a negative impact (-2.66) for Sustainable Water and Sanitation. The results
suggest this is largely due to negative impacts for Water Security, Institutions and Well-being for
this subgroup.

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
40
Table 6.5.2: Impact of the project by whether any household members have a disability
Sustainable Water and Sanitation Index George Compound Chawama Compound
Effect of having household members with a 0.81 2.13
disability in the comparison group (1.99) (2.04)
Project impact among households having -11.02*** 5.63***
members with a disability (1.98) (2.18)
Differential project impact for households with or 3.11 -8.29***
without members with a disability (2.52) (2.81)
* p < 0.1, ** p < 0.05, *** p < 0.01; PSM estimates are bootstrapped with 1,000 repetitions. Standard errors are in
parentheses.

Finally, we looked for differential impact by home ownership status. We expected that
homeowners could experience their water and sanitation situation differently than tenants (e.g.,
renters, caretakers), largely due to differing levels of autonomy with regard to altering their
water connections and sanitation facilities. The KIIs described further some of the differences
between homeowners and tenants, highlighting them as key subgroups in this context.

HOMEOWNERS AND TENANTS

‘Most people here are tenants with landlords. The landlords stay longer, but the
tenants come and go.’
– Ward Development Committee Representative in Chawama Compound

‘At a household- or property-level, we are having a situation now where at a plot you
have more than one household. That’s a typical arrangement. Initially, of course, we
would have two, three, and the like. But this time you will find so many rooms. And all
those want water, individually. So, there’s a growing demand for water, and also the
infrastructures remain the same. Of course, we’ve added two boreholes, but that
cannot really meet the growth in population and subsequently the demand for water.’
– Lusaka Water and Sewerage Company Office Manager for George Compound

In George 53% were tenants, while in Chawama 50% were tenants. In George, we see a
significant positive difference for homeowners relative to tenants for Sustainable Water and
Sanitation in the comparison group. We also see a negative project impact for tenants, but this
impact is not significantly different from that of homeowners. In Chawama, we find no significant
difference, impact or differential impact.

Table 6.5.3: Impact of the project by home ownership

Sustainable Water and Sanitation Index George Compound Chawama Compound
Effect of being a homeowner in the comparison 3.53* -0.38
group (1.98) (1.86)
Project impact among tenants -8.11*** -0.34
(1.71) (1.83)
Differential project impact for tenants and -2.21 2.83
homeowners (2.53) (2.55)
* p < 0.1, ** p < 0.05, *** p < 0.01; PSM estimates are bootstrapped with 1,000 repetitions. Standard errors are in
parentheses.

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
41
6.6 ON THE CHOLERA OUTBREAK
In this section, we further examine possible implications the 2018 cholera outbreak described in
Section 4.1 for the results of this evaluation. Despite efforts to account for the outbreak and
subsequent response throughout every stage from design to data gathering to analysis, we
cannot rule out the possibility that the results may be biased. It is clear that both before and
after the project, the water and sanitation situation in George was worse than that of Kanyama,
which was in turn worse than Chawama.

In George, people were much more likely to say that the 2018 cholera outbreak affected them
more than previous cholera outbreaks, while those in Chawama were more likely to say that it
affected their household the same or less. The comparison area, Kanyama, was somewhere in
between. This is illustrated in Figure 6.6.1. Not surprisingly, this trend correlates with what we
noted regarding the water and sanitation situations in these communities. These data mostly
align with information reported by WHO Africa and the CDC (see Section 4.1), both in terms of
the number of cases observed (more in George and Kanyama) and vaccination rates (higher in
Chawama and Kanyama).

To some extent, the severity of the cholera outbreak in each intervention community links to the
sustainability of any impacts achieved by the end of the project in 2017 – more sustainable in
Chawama, less sustainable in George. The KIIs also confirm the severity of the outbreak, with
more challenges being described by institutional representatives in George and Kayama,
relative to those in Chawama.

Figure 6.6.1: How much households were affected by the 2018 cholera outbreak
compared to previous outbreaks (descriptive only, before matching)

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
42
 IN THE WAKE OF THE CHOLERA EPIDEMIC

‘In the wake of the cholera epidemic, we purchased an online chlorinator, it’s installed,
it’s working. We purchased hand washing facilities, which are dotted around the
school. We have integrated health hygiene into our lessons, a deliberated
programme.’
– Guidance Teacher at Chawama Primary School

‘As you are aware, in January we had an outbreak. This year we had an outbreak
which almost hammered us too much. I would say that outcome was being seen
because of activities which were being undertaken, because we had a lot of cases
from January and February. But as we continued with the same activities we were
doing with the community, you could see the decline of cases. They declined up to the
declaration that there was no outbreak of cholera any more. As of now, I can say that
we’ve got a rise again. There’s that rise which is now spiking…you might give the
health education, again it goes back to the community – if they see [the cholera
outbreak is over], everyone now becomes reluctant, but us we have not relaxed, we
have still continued with the same impact.’
– Environmental Health Technician at George Health Clinic

‘Remember we had the dry spell for five years, from 2010 up until 2016. The last
outbreak was in 2010 or 2011. So, after that it was dry, there was no cholera case
recorded in Lusaka. And so, people got used to drinking water from shallow wells and
the situation from the taps worsened. There was a situation where people would line
up at 6:00 and after 10:00 the taps were closed. Then they come and open again at
14:00, by 16:00 or 17:00 it’s closed. What of those who go for working? Where will
they get the water from? They needed water. So, a lot of people opted for these
[shallow] wells.’
– Environmental Health Technician at Kanyama Health Clinic

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
43
7 CONCLUSIONS
7.1 CONCLUSIONS
Overall, the project had mixed impacts that varied by compound, gender and disability. The
project had a negative impact in George Compound for the Sustainable Water and Sanitation
Index (-8.62, p<0.01; worse for men than women) and Water Insecurity (HWISE; 6.34, p<0.01);
no significant impact on water and sanitation access is observed. On the other hand, in
Chawama, we find the project had a positive impact on access to piped water on premises
(4.42, p<0.1) and improved sanitation (8.95, p<0.01). Despite this, we do not see a significant
impact on the Sustainable Water and Sanitation Index overall (although several significant
impacts are observed among the dimensions and indicators, and there is a significant positive
impact among households having members with a disability) and we find a significant negative
impact on Water Insecurity (HWISE; 2.03, p<0.05).

Throughout this report, tangible and intangible impacts are combined in the indicators and
dimensions of the Sustainable Water and Sanitation Index (see Table 7.1.1). Both the aspects
are important for measuring this concept, to capture differences ranging from access to
infrastructure to personal experiences and opinions. Physical infrastructure alone does not tell
the whole story because that system may not work for everyone all the time. Personal
experiences and opinions are key to understanding how people are really affected and what
factors are contributing the sustainability of water and sanitation systems.

Table 7.1.1: Example tangible and intangible aspects of the Sustainable Water and
Sanitation Index dimensions and indicators
Dimension Tangible Aspects Intangible Aspects
Water Security Water service interrupted or limited (low Worrying, feeling angry, ashamed,
water pressure, less water than excluded, stigmatized because of
expected), not enough water for problems with water
drinking, food preparation, washing
(hands, body, clothes, etc.)
Equity Women and men equally share Services are perceived to be equally
responsibilities for water collection and inclusive for diverse groups (for women
unpaid care work related to water (not and men, young and old, people with
measured in this evaluation) disabilities)
Institutions Access to information, taking action to Confidence in holding the government
hold the government and service and service providers to account,
providers to account for water and transparency of decision-making
sanitation systems and services process, trust in institutions
Operations Infrastructure access (type and Feeling safe accessing infrastructure,
location), existence of a water and confidence in the water and sanitation
sanitation committee, calculation of committee to carry out duties (O&M,
household expenses on water and repairs, replacements), opinions
sanitation service pricing and affordability
Well-Being Incidence of health problems Life satisfaction, feeling stressed,
(diarrhoea, fever, cholera, etc.), school subjective health
attendance of children

The Environment Existence of water safety and waste Implementation and awareness of
management plans (not measured in community water safety and waste
this evaluation), household water management plans
conservation and source protection
practices

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
44
This distinction between the tangible and intangible also helps with interpreting the impact of the
project. Most negative impacts are related to the intangible – worry and anger due to problems
with water, opinions about equity, confidence, satisfaction, trust, opinions on affordability and
fairness of pricing, feeling safe and subjective health. Meanwhile, we see little to no impact on
tangible aspects – water fee structure (including options for those who cannot pay), proportion
of monthly expenses going to water, sanitation and health, types of water and sanitation
services and systems used, physical terrain between the household and the water source,
hours of access per day, incidence of specific health issues, school attendance and measures
taken to conserve and protect water resources. This distinction is more apparent in George,
where we see more negative impacts, than in Chawama, where we see more positive impacts.

The KIIs highlight the role of specific project activities, such as water quality monitoring and
CBEs. For example, there is evidence that water quality monitoring is continuing and that the
comparison area – Kanyama – is also benefiting by occasionally borrowing the water quality
monitoring kits provided in George and Chawama. As for the CBEs, some are still active, but
not as active as they used to be during the project. Some CBEs are mentioned as still working
with the Lusaka Water and Sewerage Company. At the same time, concerns were raised about
solid waste management CBEs, some of which were said to not be disposing of waste properly.
Overall, water systems and services were described as making good progress, with plans
already underway to resolve issues such as intermittent water supply. However, sanitation
remained an immense challenge, with clarity on how to set up safe, equitable and effective solid
waste management systems still lacking.

In Chawama, the results are relatively straightforward – the project achieved several positive
impacts, although alongside some negative impacts. On the other hand, in George, the mostly
negative results may reflect increased awareness of water and sanitation issues among
residents and subsequent frustration with seeing little progress made to remedy those issues. It
is possible for mobilization to lead to negative results if residents are not listened to, or if those
listening have little power to make real changes in their communities – a power analysis would
help to illuminate such disconnects between the residents and duty bearers, extending beyond
the compound to district and national levels as well (Hopkins, Brady & Brownlie, 2014).

7.2 PROGRAMME LEARNING

CONSIDERATIONS
Consider conducting a power analysis during programme
design and regularly revisit it during implementation
Based on the mixed impacts found through this evaluation– the same approach worked
relatively well in Chawama but not in George – future programmes could use a power analysis
as the basis for further tailoring project designs to specific populations and local structures. A
power analysis should be done for each community, to understand the of centres of power both
inside and outside that community (e.g., district and national level) and how these power
relations might alter project effectiveness. Regular reflections on this analysis should be done
with a broad cross-section of the community. Based on these reflections, the power analysis
can evolve (and strategies may need to change) over time as power structures (hopefully)
change throughout the course of the project.

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
45
Based on the findings in this evaluation, reflect on mobilization
strategies
While the project activities were the same in George and Chawama, mobilization appears to
have been more successful in Chawama because the project was able to work through
established local-level structures. In George, local-level structures had to be set up as part of
the project (e.g., Ward Development Committees – WDCs). The maturity of these local
structures likely influenced the impact of the project in each compound.

Also, based on the low levels of project exposure seen in this evaluation, the project could have
looked for additional ways to connect with more community members more often. Linking to the
first recommendation above, mobilization can also be part of the power analysis. For example,
look for ways to further develop mobilization strategies that account for how project
implementation may be affected by differences in existing community structures.

When shifting from rural to urban WASH programming, build in

training opportunities for staff to adapt their skills
This project was a first attempt at urban WASH programming in Lusaka. Together, Oxfam and
Village Water (implementing partner) had successfully carried out various rural WASH projects
in the past. Transitioning this work to an urban environment was both a new and different
challenge, as well as a learning experience. Communities within peri-urban Lusaka can have
very different experiences when it comes to water and sanitation systems and services.

Understanding this reflects a need for further training for staff when transitioning from rural to
urban WASH programming, to improve knowledge and skills for working in such a different
context. With shifts from rural to urban accelerating in the sector, driven at least in part by
increasing urbanization, strategies are often being adapted from rural to urban contexts. In this
process, ensure that training opportunities for staff are built in early on so they can adapt and
learn about aspects that differ between rural to urban WASH programming.

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
46
APPENDIX 1: DETAILED INDICATORS, QUESTIONS, AND THRESHOLDS
The following set of tables provide the detailed indicators, questions, and thresholds for each dimension of the Sustainable Water and Sanitation Index. The table also
shows if each indicator is directly (highlighted in green) or indirectly (highlighted in yellow) linked to the Urban WASH project’s Theory of Change (ToC).

Table A1.1: Water Security dimension = Water Security indicator

Indicator Variable Question Threshold ToC link?
In the last month, how frequently did you or anyone in your household worry you would not have enough water for all of your
hwise1_i
household needs?
In the last month, how frequently has your main water source been interrupted or limited (water pressure, less water than
hwise2_i
expected, dried up)?
hwise3_i In the last month, how frequently has there not been enough water to wash clothes?
In the last month, how frequently has you or anyone in your household had to change schedules or plans due to problems with
hwise4_i your water situation? Activities that may have been interrupted include caring for others, doing household chores, agricultural
work, income generating activities, etc.
In the last month, how frequently have you or anyone in your household had to change what was being eaten because there
hwise5_i
were problems with water (for washing foods, cooking)?
In the last month, how frequently have you or anyone in your household had to go without washing hands after dirty activities
hwise6_i
(defecating or changing diapers, cleaning animal dung) because of problems with water?
Water Security
In the last month, how frequently have you or anyone in your household had to go without washing their body because of Never = 1
(Average of 15 hwise7_i Indirect
problems with water (not enough water, dirty, unsafe)? Rarely to Always = 0
variables x 100)
In the last month, how frequently has there not been as much water to drink as you would like for you or anyone in your
hwise8_i
household?
hwise9_i In the last month, how frequently did you or anyone in your household feel angry about your water situation?
In the last month, how frequently have you or anyone in your household gone to sleep thirsty because there wasn’t any water to
hwise10_i
drink?
hwise11_i In the last month, how frequently has there been no usable or drinkable water whatsoever in your household?
In the last month, how frequently have problems with water caused you or anyone in your household to feel ashamed, excluded,
hwise12_i
and/or stigmatized?
hwise13_i In the last month, how frequently have you or anyone in your household drunk water that looked, tasted, and/or smelled bad?
hwise14_i In the last month, how frequently have you or anyone in your household drunk water that you thought was unsafe?
hwise15_i In the last month, how frequently have you or anyone in your household been unable to access the water that you preferred?

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
47
Table A1.2: Equity dimension = (Gender Inclusive indicator + Inclusive of Vulnerable Users indicator)/2
Indicator Variable Question Threshold ToC link?
Which of the two statements do you agree with most:
Gender 1 – A woman can be a leader in making decisions for water and sanitation in my community. Option 1 = 1
womanleaders
Inclusive 2 – Men are by nature better leaders than women when it comes to making decisions about water and sanitation in my Option 2 = 0
community. Direct
(Average of 3
variables x 100) equalwat_gender The water needs of people of all genders are being met. True = 1
equalsan_gender The sanitation needs of people of all genders are being met. False = 0
Which of the two statements do you agree with most:
1 – A person with a disability can be a leader in making decisions for water and sanitation in my community. Option 1 = 1
pwdleaders
Inclusive of 2 – A person with a disability is not able to be a leader when it comes to making decisions about water and sanitation in Option 2 = 0
Vulnerable my community.
Users equalwat_disability The water needs of people with disabilities and/or chronic illnesses are being met. Direct
(Average of 5
variables x 100) equalsan_disability The sanitation needs of people with disabilities and/or chronic illnesses are being met. True = 1
equalwat_children The water needs of children are being met. False = 0
equalsan_children The sanitation needs of children are being met.

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
48
Table A1.3: Institutions dimension = (Accountability indicator + Transparency indicator + Enabling Environment indicator + Trust indicator)/4
Indicator Variable Question Threshold ToC link?
What do you think you can you do to hold the government and service providers accountable for water At least one idea = 1
holdacc_idea
and sanitation services? Don’t know/Not sure = 0
Accountability
How confident do you feel about your ability to hold the government and service providers accountable Very/somewhat confident = 1
(Average of 3 holdacc_conf Direct
for water and sanitation services? Not confident = 0
variables x 100)
Have you done anything in the past year to try to hold the government and service providers accountable Yes = 1
holdacc_yn
for water and sanitation services? No = 0
info_yn Do you get any information about the management of your water and/or sanitation systems?
Do you know who you can go to if you have questions or concerns about your water and/or sanitation
concern_yn
systems?
watdecision_who Do you know who makes the decisions about your water services/systems? Yes = 1
No = 0
Transparency watdecision_how Do you understand how decisions are made for your water services/systems?
(Average of 8 sandecision_who Do you know who makes the decisions about your sanitation services/systems? Direct
variables x 100) sandecision_how Do you understand how decisions are made for your sanitation services/systems?
Overall, how satisfied are you with how decisions are made for your water and sanitation Very/Somewhat satisfied = 1
decision_satisfaction
services/systems? Very/Somewhat dissatisfied = 0
How much do you think that your needs and opinions are accounted for in decisions about your water A lot/Somewhat = 1
decision_yourneeds
and sanitation systems? Not at all/Just a little = 0
Household reported that they earned income from their own business during the past 12 months (1
businessowner
Enabling year).
Environment Household reported that they earned income from their own business during the past 12 months (1 Yes = 1
businessstarter Direct
(Average of 3 year), but they did not report earning any income from this source in 2012. No = 0
variables x 100) Have you or any member of your household made any big investments of more than 2,000 kwacha in the
investment
past year (for example, to build a latrine or buy a booth)?
trust1 How much would you say you trust: Local Government/Lusaka City Council (LCC)?
trust2 How much would you say you trust: Ward Development Committee (WDC).
trust3 How much would you say you trust: Central Government (national).

Trust trust4 How much would you say you trust: Water and sanitation service providers.
A lot/Somewhat = 1
(Average of 9 trust5 How much would you say you trust: Property owner (landlord/landlady). Direct
Not at all/Just a little/No opinion = 0
variables x 100) trust6 How much would you say you trust: Non-governmental organizations (NGOs)
trust7 How much would you say you trust: Community-based organizations (CBOs)
trust8 How much would you say you trust: Community-based enterprises (CBEs).
trust9 How much would you say you trust: Area Councillor.

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
49
Table A1.4: Operations dimension = (Asset Management indicator + Financial Viability indicator + Accessibility indicator)/3
Indicator Variable Question Threshold ToC link?
Structure known = 1
committeestructure How are water and sanitation committees structured in your community?
Don’t know/N/A = 0
How confident are you in the capacity of this committee or group to manage routine operation and
confident1
maintenance of water systems within your community?
How confident are you in the capacity of this committee or group to manage routine operation and
confident2
maintenance of sanitation/sewerage systems within your community?
How confident are you in the capacity of this committee or group to manage major repairs to water systems
confident3
when broken down within your community?
How confident are you in the capacity of this committee or group to manage major repairs to
confident4
sanitation/sewerage systems when broken down within your community? Very/somewhat confident = 1
How confident are you in the capacity of this committee or group to manage a full replacement of water Not confident or No opinion = 0
Asset confident5
Management systems if the current systems cannot be repaired within your community?
Direct
(Average of 12 How confident are you in the capacity of this committee or group to manage a full replacement of
confident6
variables x 100) sanitation/sewerage systems if the current systems cannot be repaired within your community?
How confident are you in the capacity of this committee or group to manage management of water systems
confident7
through a crisis, shock or disaster within your community?
How confident are you in the capacity of this committee or group to manage management of
confident8
sanitation/sewerage systems through a crisis, shock or disaster within your community?
What water source does your household use most often for drinking water when your primary drinking water Piped/improved = 1
waterbd_alt
source is not functioning or is inaccessible? Unimproved = 0
How often does it happen that you cannot use your main toilet/facility/place because it does not function Never/Almost never = 1
sanbd
properly? Annually to Daily = 0
What type of sanitation facilities do members of your household use most often when your main Closed pit/flush toilet = 1
sanbd_alt
toilet/facility/place is not functioning or is inaccessible? Open pit/none = 0
watfeestruct What is the fee structure for your primary drinking water source? Fixed or variable = 1
sanfeestruct What is the fee structure for your primary sanitation services (such as pit emptying costs)? Irregular or free = 0
watfeeopt_yn Do you know of any options for those who are not able to pay (for water)? Yes = 1
Financial sanfeeopt_yn Do you know of any options for those who are not able to pay (for sanitation)? No = 0
Viability watpayment How difficult is process of paying for water? Not difficult at all = 1 Indirect
(Average of 13
sanpayment How difficult is the process of paying for your sanitation facility/toilet/service? Very/somewhat difficult = 0
variables x 100)
Calculation of how much the household spends on water (fees, bottled water, chlorine, etc.), as a
expense_wat
percentage, based on an accounting of monthly expenses. <=5% of total = 1
Calculation of how much the household spends on sanitation and personal hygiene (toilet, soap, etc.), as a >5% of total = 0
expense_san
percentage, based on an accounting of monthly expenses.

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
50
 watafford In the past month, how difficult has it been for your household to come up with the money to pay for water?
Not difficult at all = 1
In the past month, how difficult has it been for your household to come up with the money to pay for Very/somewhat difficult = 0
sanafford
sanitation?
watpricing Do you think your water service is fairly priced? Completely/Somewhat fair = 1
sanpricing Do you think your primary sanitation facility/toilet/service is fairly priced? Completely/Somewhat unfair = 0
Are you planning and/or saving up now to upgrade/improve your sanitation facility/toilet/service in the next 6 Yes = 1
sanplans
months? No = 0
2 or more improved sources = 1
watsources Can you please list all the water sources that your household uses for all purposes during the DRY season?
1 or 0 improved sources = 0
waterdry How do you primarily access water for drinking now during the dry season?
Piped/Improved = 1
waterwet How do you primarily access water for drinking now during the wet season?
Unimproved = 0
waterdom How do you primarily access water for other domestic purposes during the dry season?
Closed pit/flush toilet = 1
sanfac What type of sanitation facilities do members of your household use most often now?
Open pit/none = 0
watsourceofficial Is your primary drinking water source during the dry season an official/government service? Yes = 1
sanofficial Do you use any official/government sanitation facilities or services? No = 0
watsource_sp Is your primary drinking water source during the dry season private, only for your household, or is it shared? Private = 1
san_sp Is the main toilet/facility/place you use private, only for your household, or is it shared? Shared = 0
watsourceloc During the dry season, where is your primary drinking water source located? In house/On plot = 1
Accessibility sanloc Where is your main toilet/facility/place located? Off plot = 0
(Average of 20 Indirect
<=10 minutes = 1
variables x 100) watsourcetime How long does it take to collect water from this source, including walking and any time spent waiting?
>10 minutes = 0
<=5 minutes = 1
santime_walk How long does it take to walk (to your main sanitation facility)?
> 5 minutes = 0
Easy/N/A = 1
watsourceterrain During the dry season, what is the terrain like on the way to and from your primary drinking water source?
Very/Somewhat difficult = 0
During the dry season, when everything with the water system is functioning properly, how many hours per
wateraccess
day are members of your household able to access your main drinking water source? 24 hours = 1
When everything with the toilet/facility is functioning properly, how many hours per day are members of your <24 hours = 0
sanaccess
household able to access it?
watsourcesafegoing Do you feel safe and secure when you are going to and from this water source?
watsourcesafeusing Do you feel safe and secure while you are at this water source (while you are using it)? Yes/ N/A = 1
sansafegoing_day Do you feel safe when you are going to and from your main toilet/facility/place during the day? No = 0
sansafegoing_night Do you feel safe when you are going to and from your main toilet/facility/place at night?

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
51
Table A1.5: Well-being dimension = (Life Satisfaction indicator + Health indicator + Education indicator)/3
Indicator Variable Question Threshold ToC link?
Life Satisfaction
Very/Somewhat satisfied = 1
(lifesatisfaction x lifesatisfaction Overall, how satisfied are you with life these days? Indirect
Very/Somewhat dissatisfied = 0
100)
Very good/Good = 1
health Overall, how would you describe your state of health now?
Very poor/Poor = 0
stomachpain In the last 30 days (1 month), have you had any stomach pain (excluding menstrual pain, for women)?

fever In the last 30 days (1 month), have you had a fever?

Health
diarrhoea I’m sorry to have to ask this, but in the last 30 days (1 month), have you had any diarrhoea? No = 1
(Average of 7 Direct
Yes = 0
variables x 100) In the last 30 days (1 month), have you felt so unwell that you had to interrupt your normal daily activities
unwell
(excluding menstrual pain, for women)?
Thinking back to the most recent cholera outbreak earlier this year, were any members of your household
cholera
directly affected?
Less/The same = 1
choleracompare How much did this cholera outbreak affect your household compared to other outbreaks in the past?
More = 0
Yes = 1
Education schoolnow Are all school-aged children in the household attending school? (calculation)
No = 0
(Average of 2 Direct
variables x 100) Have any school-aged children in the household missed school in the past 1 month due to water and sanitation No = 1
absencewatsan
related issues? (calculation) Yes = 0

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
52
Table A1.6: The Environment dimension = (Water Safety indicator + Waste Management indicator + Water Conservation indicator)/3
Indicator Variable Question Threshold ToC link?
Water Safety environment1 The government or another committee monitors water systems in my community to ensure water is safe to drink.
True = 1
(Average of 2 The government or another committee has water safety plans to protect drinking water sources in my community Direct
environment2 False/Don’t know = 0
variables x 100) from contamination.

environment3 The government or another committee has drainage plans to protect my community from flooding.
Waste Management
True = 1
(Average of 3 The government or another committee monitors sanitation systems in my community to ensure wastewater/sewage Direct
environment4 False/Don’t know = 0
variables x 100) is safely managed.
environment5 The government or another committee has solid waste management plans to keep my community clean from refuse.
The government or another committee has plans that account for the various uses of water, including domestic uses True = 1
environment6
(within the household) and productive uses (agriculture, businesses, factories). False/Don’t know = 0
Water Conservation
Any technique = 1
(Average of 3 waterconserve What techniques does your household use to reduce the amount of water you use? Indirect
N/A = 0
variables x 100)
Any technique = 1
waterprotect What techniques does your household use to protect your primary water source from contamination?
N/A = 0

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
53
APPENDIX 2: SUMMARY STATISTICS BEFORE
MATCHING
For reference, Table A2.1 and Table A2.2 below show various summary statistics for George and
Chawama, respectively. These data represent the sample before propensity score matching (PSM), but
after filtering (data from households that had moved to the community less than two years before the
survey were excluded, i.e., toward the end of the project or after it was over). In each table, the difference
column indicates several significant differences between the intervention and comparison group before
matching. The purpose of PSM is to balance these differences during analysis (see Appendix 3).

Table A2.1: Summary statistics before propensity score matching for George Compound
(Intervention) and Kanyama Compound (Comparison)
Variable Intervention Comparison Difference Standard
group mean group mean error
(George) (Kanyama)
Interviewee is a woman (%) 50.64 53.21 -2.57 4.53
Interviewee age (years) 34.69 33.98 0.71 1.13
Interviewee completed secondary school (%) 38.46 48.57 -10.11* 4.51
Interviewee is married (%) 59.62 53.57 6.04 4.51
***
Interviewee participated in a group in 2012 (%) 35.90 56.07 -20.17 4.47
HH head is a woman (%) 20.51 23.93 -3.42 3.82
HH head age (years) 42.26 40.29 1.97 1.17
HH head completed secondary school (%) 44.23 58.39 -14.16** 4.48
**
HH head is literate (%) 82.69 91.61 -8.91 2.74
Number of HH members in 2012 4.18 3.64 0.54** 0.20
80%+ HH members over 5 years old fit to work (%) 81.16 74.99 6.17* 2.55
HH lived in the community in 2012 91.67 95.54 -3.87 2.03
HH owned their home in 2012 (%) 45.51 31.43 14.08** 4.28
*
HH earned income from salaried employment in govt in 2012 7.05 14.11 -7.06 2.99
HH earned income from salaried employment in private sector 20.51 28.04 -7.52 3.99
and/or NGOs in 2012 (%)
HH earned income from casual labour in 2012 (%) 33.97 35.18 -1.20 4.32
HH earned income from their own business in 2012 (%) 32.69 43.57 -10.88* 4.44
HH earned income from farming, agriculture, husbandry in 2012 (%) 11.54 9.82 1.72 2.74
HH earned income from remittances in 2012 (%) 10.90 16.79 -5.89 3.27
HH earned income from cross-border trading in 2012(%) 3.21 3.57 -0.37 1.66
**
HH was in the lowest 20% of wealth distribution in 2012 (%) 28.29 18.10 10.19 3.66
HH was in the second lowest 20% of wealth distribution in 2012 (%) 19.08 21.15 -2.07 3.71
HH was in the second highest 20% of wealth distribution in 2012 (%) 18.42 20.43 -2.01 3.66
HH was in the highest 20% of wealth distribution in 2012 (%) 16.45 18.10 -1.65 3.50
+ ***
Improved water source (dry season) in 2012 (%) 83.97 95.54 -11.56 2.27
Improved+ water source (wet season) in 2012 (%) 83.33 95.18 -11.85*** 2.33
Piped on premises (dry season) in 2012 (%) 19.23 33.04 -13.80*** 4.12
***
Piped on premises (wet season) in 2012 (%) 19.23 33.21 -13.98 4.13
Improved sanitation facility in 2012 (%) 38.71 71.43 -32.72*** 4.18
***
Affected more by 2018 cholera outbreak than previous outbreaks 92.95 61.96 30.98 4.04
(%)
Observations 716
+
Improved water includes piped water on premises. * p < 0.1, ** p < 0.05, *** p < 0.01. Variables dated 2012 are estimates, based on
recall data. Construction of the wealth index is described in Section 4.4.

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
54
Table A2.2: Summary statistics before propensity score matching for Chawama Compound
(Intervention) and Kanyama Compound (Comparison)
Variable Intervention Comparison Difference Standard
group mean group mean error
(Chawama) (Kanyama)
Interviewee is a woman (%) 50.00 53.21 -3.21 4.46
*
Interviewee age (years) 36.53 33.98 2.55 1.16
Interviewee completed secondary school (%) 43.83 48.57 -4.74 4.46
Interviewee is married (%) 50.00 53.57 -3.57 4.46
Interviewee participated in a group in 2012 (%) 61.73 56.07 5.66 4.41
HH head is a woman (%) 26.54 23.93 2.61 3.84
*
HH head age (years) 43.21 40.29 2.92 1.16
HH head completed secondary school (%) 48.77 58.39 -9.63* 4.42
**
HH head is literate (%) 83.95 91.61 -7.66 2.68
Number of HH members in 2012 3.94 3.64 0.30 0.20
**
80%+ HH members over 5 years old fit to work (%) 65.63 74.99 -9.36 2.86
HH lived in the community in 2012 90.06 95.54 -5.47** 2.06
HH owned their home in 2012 (%) 51.85 31.43 20.42*** 4.22
HH earned income from salaried employment in govt in 2012 12.35 14.11 -1.76 3.07
HH earned income from salaried employment in private sector 38.89 28.04 10.85** 4.09
and/or NGOs in 2012 (%)
HH earned income from casual labour in 2012 (%) 43.21 35.18 8.03 4.30
HH earned income from their own business in 2012 (%) 45.68 43.57 2.11 4.43
HH earned income from farming, agriculture, husbandry in 2012 (%) 12.96 9.82 3.14 2.74
HH earned income from remittances in 2012 (%) 34.57 16.79 17.78*** 3.56
HH earned income from cross-border trading in 2012(%) 4.32 3.57 0.75 1.69
HH was in the lowest 20% of wealth distribution in 2012 (%) 14.29 18.10 -3.81 3.38
HH was in the second lowest 20% of wealth distribution in 2012 (%) 15.53 21.15 -5.62 3.57
HH was in the second highest 20% of wealth distribution in 2012 (%) 20.50 20.43 0.07 3.61
HH was in the highest 20% of wealth distribution in 2012 (%) 31.06 18.10 12.96*** 3.62
+ **
Improved water source (dry season) in 2012 (%) 100.00 95.54 4.46 1.62
Improved+ water source (wet season) in 2012 (%) 98.77 95.18 3.59* 1.75
***
Piped on premises (dry season) in 2012 (%) 53.09 33.04 20.05 4.26
Piped on premises (wet season) in 2012 (%) 54.32 33.21 21.11*** 4.26
Improved sanitation facility in 2012 (%) 73.46 71.43 2.03 4.02
Affected more by 2018 cholera outbreak than previous outbreaks 29.01 61.96 -32.95*** 4.28
(%)
Observations 722
+
Improved water includes piped water on premises. * p < 0.1, ** p < 0.05, *** p < 0.01. Variables dated 2012 are estimates, based on
recall data. Construction of the wealth index is described in Section 4.4.

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
55
APPENDIX 3: PROPENSITY SCORE MATCHING
METHODOLOGY
The results presented in Section 6 of this report have been estimated using propensity score matching
(PSM). PSM is a statistical technique that allows the effect of an intervention to be estimated by
accounting other factors that predict receiving the intervention, or ‘treatment’. The idea behind PSM is to
match households in the intervention group to similar households in the comparison group, based on
baseline characteristics. After each participant is matched with a non-participant, the average treatment
effect on the treated (those who benefited from the intervention) is equal to the difference in average
outcomes of the intervention and the comparison groups after project completion. This appendix describes
and tests the specific matching procedure employed in this evaluation. The approach follows the guidance
provided by Caliendo and Kopeinig (2008).

Estimating propensity scores

Finding an exact match for treated households, based on various baseline characteristics, is very hard to
implement in practice. Rosenbaum and Rubin (1983) demonstrate that a ‘propensity score’ can
summarize all this information in one single variable. The propensity score is defined as the conditional
probability of receiving the intervention given background variables. Specifically, propensity scores are
calculated using a statistical probability model (e.g., probit or logit) to estimate the probability of
participating in the project based on a set of characteristics.

Table A3.1 shows the variables used to estimate the propensity score in this report, alongside marginal
effects at the mean, standard errors and p-values by compound. Note that the propensity score could not
be calculated because of one or more missing values for 14 households (six comparison, eight
intervention – four each from George and Chawama). Following Caliendo and Kopeinig (2008), only
variables that influence the participation decision, but which are not affected by participation in the project,
have been included in the matching model. In the table, the dependent variable corresponds to whether
the household received the intervention (i.e., it is equal to 1 if the household participated in the project,
and 0 otherwise). The coefficients in the table correspond to the marginal effects, which are the change in
the probability of receiving the intervention if the independent variable is increased by one.

Defining the region of common support

After estimating the propensity scores, it is necessary to verify that potential matches exist for the
observations in the intervention group with those from the comparison group – checking that there is
common support. The area of common support is the region where the propensity score distributions of
the intervention and comparison groups overlap. The common support assumption ensures that each
‘treatment [intervention] observation has a comparison observation “nearby” in the propensity score
distribution’ (Heckman, LaLonde & Smith, 1999). Figure A3.1 shows the propensity score density plots for
both groups. It can be observed that, although the distributions of propensity scores are clearly different
between the intervention and comparison groups in each case, there is a reasonably good area of overlap
between the groups. However, in constructing the model for household-level outcomes for George, two
intervention and 119 comparison observations were dropped for lacking suitable matches. Similarly, in the
matching process for Chawama, seven intervention and 43 comparison observations were dropped.

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
56
Table A3.1: Variables used for propensity score matching with marginal effects, standard errors, and p-values, by intervention compound
Variable Intervention Group: George Compound Intervention Group: Chawama Compound
Marginal Standard p-value Marginal Standard p-value
effect error effect error
Interviewee is a woman 0.00 0.03 0.98 -0.01 0.03 0.77
Interviewee age (years) -0.00 0.00 0.44 -0.00 0.00 0.89
Interviewee completed secondary school 0.02 0.04 0.59 -0.02 0.04 0.61
Interviewee is married 0.03 0.04 0.40 0.06 0.04 0.11
HH head is a woman -0.02 0.04 0.71 -0.01 0.04 0.78
HH head age (years) 0.00 0.00 0.95 0.00 0.00 0.72
HH head completed secondary school -0.03 0.04 0.44 -0.05 0.04 0.28
HH head is literate -0.08 0.06 0.17 -0.10 0.06 0.12
Number of HH members in 2012 0.02* 0.01 0.02 0.01 0.01 0.29
** **
HH lived in the community in 2012 -0.28 0.10 0.01 -0.24 0.09 0.01
HH owned their home in 2012 0.09* 0.04 0.02 0.11** 0.04 0.01
80%+ HH members over 5 years old fit to work (%) -0.00 0.03 0.92 0.04 0.03 0.17
**
HH earned income from salaried employment in govt in 2012 -0.02 0.05 0.64 -0.10 0.03 0.01
HH earned income from salaried employment in private sector and/or NGOs in 2012 -0.02 0.03 0.61 0.11** 0.04 0.01
HH earned income from casual labour in 2012 -0.03 0.03 0.41 0.03 0.03 0.39
HH earned income from their own business in 2012 -0.04 0.03 0.22 -0.00 0.03 0.96
HH earned income from farming, agriculture, husbandry in 2012 -0.00 0.05 0.96 0.01 0.05 0.90
HH earned income from remittances in 2012 0.01 0.05 0.77 0.14** 0.05 0.00
HH earned income from cross-border trading in 2012 0.10 0.11 0.37 -0.01 0.07 0.89
HH was in the lowest 20% of wealth distribution in 2012 0.05 0.05 0.38 0.07 0.06 0.29
HH was in the second lowest 20% of wealth distribution in 2012 -0.03 0.04 0.56 -0.02 0.05 0.72
HH was in the second highest 20% of wealth distribution in 2012 0.03 0.05 0.48 0.00 0.05 0.92
HH was in the highest 20% of wealth distribution in 2012 0.09 0.06 0.17 -0.02 0.05 0.72
*
Interviewee participated in a group in 2012 -0.08 0.03 0.01 0.00 0.03 0.99
Main drinking water source (dry season) in 2012 (ordinal 0 - 2) 0.06 0.11 0.56 -0.04 0.10 0.70
Main drinking water source (wet season) in 2012 (ordinal 0 - 2) -0.13 0.11 0.22 0.17 0.09 0.08
***
Improved sanitation facility in 2012 -0.20 0.04 0.00 -0.00 0.04 0.98

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
57
 2018 cholera outbreak effect vs. previous ones (ordinal 0 – 2) -0.13*** 0.02 0.00 0.13*** 0.02 0.00
Observations 706 712

* p < 0.1, ** p < 0.05, *** p < 0.01. Variables dated 2012 are estimates, based on recall data. Construction of the wealth index is described in Section 4.4. The dependent variable is binary, taking 1 for project
participants, and 0 otherwise.

Figure A3.1: Common support histograms of propensity scores for intervention (Treated) and comparison (Untreated) households, by intervention
compound with George and Kanyama on the left and Chawama and Kanyama on the right

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
58
Matching intervention households to comparison households
Following Rosenbaum and Rubin (1983), households are matched based on propensity scores using a kernel matching algorithm. Kernel matching assigns more
weight to the closest comparison group observations that are found within a selected ‘bandwidth’. Thus ‘good’ matches are given greater weight than ‘poor’ matches.
The psmatch2 module in Stata (Leuven & Sianesi, 2003) was used with a bandwidth of 0.06 and analysis was restricted to the area of common support. When using
PSM, standard errors of the estimates were bootstrapped using 1,000 repetitions to account for additional variation caused by the estimation of the propensity scores.

Checking balance
For PSM to be valid, the intervention group and the matched comparison group need to be balanced. In other words, the intervention and comparison groups need to
be similar in terms of their observed characteristics. The most straightforward method of doing this is to test whether there are any statistically significant differences in
baseline covariates between both groups in the matched sample. The balance of each of the matching variables after kernel matching is shown by intervention
compound in Table A3.2. There are no statistically significant differences between intervention and comparison in the matched sample for any of the matching
variables (for either intervention compound). For all of these variables, the p-values for the difference in means tests are large, with the lowest value being 0.25 and
most being more than 0.70. It can therefore be concluded in each case that a satisfactory match has been found for the intervention group in the sample, according to
this set of matching variables.

Table A3.2: Variable balance check after propensity score matching

Variable Intervention Group: George Compound Intervention Group: Chawama Compound
Intervention Comparison p-value Intervention Comparison p-value
group mean group mean group mean group mean
Interviewee is a woman 0.51 0.55 0.41 0.51 0.53 0.73
Interviewee age (years) 34.58 34.53 0.97 35.91 34.5 0.37
Interviewee completed secondary school 0.38 0.34 0.5 0.44 0.44 1
Interviewee is married 0.59 0.65 0.25 0.50 0.46 0.42
HH head is a woman 0.21 0.17 0.33 0.28 0.29 0.84
HH head age (years) 42.15 41.3 0.57 42.87 41.83 0.51
HH head completed secondary school 0.44 0.42 0.67 0.50 0.51 0.83
HH head is literate 0.83 0.77 0.31 0.85 0.85 0.98
Number of HH members in 2012 4.19 4.1 0.70 3.95 3.87 0.75
HH lived in the community in 2012 0.93 0.95 0.25 0.91 0.91 0.91
HH owned their home in 2012 0.47 0.42 0.42 0.50 0.44 0.26

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
59
 HH has at least 80% members over 5 years old fit to work 0.62 0.65 0.53 0.56 0.55 0.82
HH earned income from salaried employment in govt. in 2012 0.07 0.07 0.95 0.13 0.13 0.97
HH earned income from salaried employment in private sector and/or NGOs in 2012 0.21 0.18 0.45 0.38 0.39 0.76
HH earned income from casual labour in 2012 0.35 0.34 0.86 0.42 0.43 0.84
HH earned income from their own business in 2012 0.33 0.3 0.56 0.46 0.5 0.43
HH earned income from farming, agriculture, husbandry in 2012 0.11 0.08 0.48 0.13 0.14 0.79
HH earned income from remittances in 2012 0.11 0.09 0.52 0.33 0.37 0.49
HH earned income from cross-border trading in 2012 0.03 0.02 0.54 0.05 0.05 0.77
HH was in the lowest 20% of wealth distribution in 2012 0.28 0.31 0.60 0.15 0.14 0.83
HH was in the second lowest 20% of wealth distribution in 2012 0.19 0.19 0.99 0.17 0.18 0.79
HH was in the second highest 20% of wealth distribution in 2012 0.19 0.21 0.62 0.21 0.21 0.96
HH was in the highest 20% of wealth distribution in 2012 0.16 0.13 0.37 0.30 0.32 0.79
Interviewee participated in a group in 2012 0.37 0.35 0.72 0.60 0.59 0.79
Primary drinking water source (dry season) in 2012 (ordinal 0–2) 2.05 2.05 0.96 2.52 2.53 0.82
Primary drinking water source (wet season) in 2012 (ordinal 0–2) 2.04 2.04 0.95 2.52 2.54 0.71
HH had an improved sanitation facility (e.g., toilet) in 2012 0.39 0.38 0.88 0.75 0.78 0.61
2018 cholera outbreak effect vs. previous ones (ordinal 0–2) 1.13 1.14 0.82 2.25 2.25 0.98
Observations 585 662

* p < 0.1, ** p < 0.05, *** p < 0.01. Variables dated 2012 are estimates, based on recall data. Construction of the wealth index is described in Section 4.4. The dependent variable is binary, taking 1 for project
participants, and 0 otherwise.

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
60
Propensity score weighting
We use propensity score (PS) weighting for difference-in-difference estimates and subgroup analysis. The
PS-weighted models used in this report are described below.

PS-weighted OLS regression to estimate project impact using difference-in-difference for

improved water, piped water and improved sanitation

Where data for both recall and now are available, we used PS-weighted OLS regression to estimate
project impact on the difference in each outcome – improved water, piped water, improved sanitation. We
calculated this difference by subtracting the 2012 (recall) value from the 2018 (now) value. All three of
these outcomes are binary, meaning the difference variables can take the following three values: -1 (the
situation got worse), 0 (the situation stayed the same), 1 (the situation got better). The results of the
regression tell us the difference in this difference, controlling for the PSM matching variables and with PS-
weighting.

PS-weighted OLS regression with interaction term to estimate differential impacts for subgroups

The average overall effect was estimated using PSM. To test for differential impacts, we used PS-
weighted OLS regression together with an interaction term (all matching variables used for PSM are also
included as control variables). Based on how each subgroup variable is defined, those equal to 1 can be
considered as the ‘group of focus’ and those equal to 0 as the ‘reference group’. In this report we
considered three subgroups with interaction terms as follows:

1. Gender: the interaction term multiplies a variable for interviewee gender (equal to 1 if a woman, 0
if a man) by the intervention variable (equal to 1 if in the intervention group, 0 if in the comparison
group). Therefore, we define women as the group of focus and men as the reference group.
2. Disability of household members: the interaction term multiplies a variable indicating household
members’ disabilities (equal to 1 if 80% of members over five years old are ‘fit to work’, 0 otherwise)
by the intervention variable (equal to 1 if in the intervention group, 0 if in the comparison group).
Therefore, we define households with fewer members with disabilities as the group of focus and
households with more with disabilities as the reference group.
3. Home ownership status: the interaction term multiplies a variable indicating home ownerships
status (equal to 1 if the household owns their own home, 0 otherwise – i.e., tenants) by the
intervention variable (equal to 1 if in the intervention group, 0 if in the comparison group). Therefore,
we define homeowners as the group of focus and tenants as the reference group.

The tables in Section 6.6 provide the results of PS-weighted regressions, each with the relevant
interaction term. Table A.3.3 provides tips for interpreting these results.

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
61
Table A3.3: Example results and interpretation of PS-weighted regression with interaction term
Sustainable Water and Example Interpretation
Sanitation index results
Effect among the group of 2.50** The first row shows the coefficient, standard error and significance
focus in the comparison (2.10) for the subgroup variable (i.e., gender, disability, home ownership
group, relative to the status). It indicates the difference between the group of focus and
reference group reference group, irrespective of the intervention. The sign of the
coefficient tells us whether the average is higher (if positive) or
lower (if negative) for the group of focus, relative to the reference
group, while the size of the coefficient tells us how large this
difference is (e.g., between women and men). The standard error
and significance indicate to what extent this difference is statistically
meaningful.
The example results would mean that the average value for the
group of focus is 2.50 higher than it is for the reference group for the
Sustainable Water and Sanitation Index, and this difference is
statistically significant (p<0.05).
Project impact for the 5.09*** The second row shows the coefficient, standard error and
reference group (4.02) significance for the intervention (i.e., project impact) for the
‘reference’ group (i.e., men, households with fewer members with
disabilities and tenants). The sign of the coefficient tells us whether
the project impact is positive or negative, while the size of the
coefficient tells us how large this impact is for the ‘reference’ group.
The standard error and significance indicate to what extent this
difference is statistically meaningful.
The example results would mean that the project had a statistically
significant (p<0.01) positive impact of 5.09 for the reference group
for the Sustainable Water and Sanitation Index.
Differential project impact 1.08 The third row shows the coefficient, standard error and significance
for the group of focus and (1.02) for the interaction term. The sign and size of the coefficient can be
the reference group interpreted together with the two other coefficients for the subgroup
and intervention variables to calculate the project impact for the
group of focus. The standard error and significance indicate to what
extent the difference in impact between the group of focus and
reference group is statistically meaningful.
The example results would mean that the project also had a positive
impact for the group of focus for the specified outcome, and that
impact is slightly larger (e.g., 1.08 + 5.09 = 6.17). However, this
difference in impact between the group of focus and the reference
group is not statistically significant (i.e., there is no differential
impact by gender for the Sustainable Water and Sanitation Index).
* p < 0.1, ** p < 0.05, *** p < 0.01; PSM estimates are bootstrapped with 1,000 repetitions. Standard errors are in parentheses.

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
62
APPENDIX 4: RISK OF BIAS
Not all quasi-experimental impact evaluations are the same. Choices made during sampling, selection of the comparison group, and at the analysis stage are crucial in
assessing overall confidence in the results. Table A4.1 uses our standard framework to assess the risk of bias against ten predetermined parameters for this
evaluation. This framework is specifically for ex-post quasi-experimental impact evaluations. Lower overall risk provides higher confidence in the results.

Table A4.1: Risk of Bias table

No. Title Description Assessment Description
Sampling
1 Random sampling Score LOW risk if: MEDIUM Interviewees were randomly sampled within each of
Sampling is conducted using probability random sampling methods on a clearly the three compounds (2 intervention compounds
established sample frame. and 1 comparison compound) using a random walk
protocol (see Section 4.2 Household Surveys for
details).
Score MEDIUM risk if:
Sampling is conducted using probability random sampling methods at
geographical level (e.g., village level), and use random sampling to select
interviewees within the geographical area.

Score HIGH otherwise.

2 Representativeness Score LOW risk if: LOW Project activities were largely conducted at the
of project participants Project participants have been involved for the entire duration of the project and community level and there were no participant lists
have been involved in the project with the same level of exposure. at the household level. It is expected that different
Project participants have been exposed to a variety of different activities, some households were exposed to different activities
may have dropped out from some activities, but sampling is conducted on the directly (e.g., attending a roadshow, seeing a
entire list of project participants. poster), but that all community members were
affected indirectly (e.g., through infrastructure and
governance improvements).
Score MEDIUM risk if:
Project participants have been exposed to a variety of different activities.
Sampling is conducted only among those project participants that have been
enrolled for the entire duration of the project or that have been enrolled in all the
activities. These are not less than 80% of the entire list of project participants OR
it is clear the results apply only to a particular group of project participants.

Score HIGH otherwise.

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
63
 3 Selection survey Score LOW risk if: LOW All community members were included in the
interviewees Identification of survey interviewees is not determined by project participation sampling frame and equally likely to be selected for
(the same protocol to identifying the interviewee(s) within the household is an interview.
applied in intervention and comparison groups).
The resulting selection of survey interviewees is not affected by project
participation (based on observables).

Score MEDIUM risk if:

Identification of survey interviewees is not determined by project participation
(the same protocol to identifying the interviewee(s) within the household is
applied in intervention and comparison group).
The resulting selection of survey interviewees is affected by project participation
(based on observables).

Score HIGH otherwise.

Selecting comparison group
4 Potential for Score LOW risk if: LOW The comparison group is a compound with the
contamination The units for comparison group are selected in geographical areas where it is same community characteristics and household
(spillovers) not reasonable to expect for the project to have had spillover effects. compositions as the intervention group compounds.
The project also implemented some activities (which are not considered the All three compounds are located approximately the
most relevant under analysis) which are expected to have had an impact also in same distance from the city centre, with the two
the comparison group (e.g., the project implemented campaigns using radio and project compounds being to the west and the south
other digital media, but these are only a minor component of the activities and the comparison compound being to the south-
implemented). The report makes clear which impact is assessed (added-value of west. However, these compounds do not border
other components, taking into account exposure to those minor components). each other and are separated by
industrial/commercial areas.

Score HIGH risk if:

The project did implement some minor activities,
Units for the comparison group are selected within the same geographical area including support to the Lusaka District Health
as the intervention group, and it is reasonable to expect that project activities Office and radio programming, that may have also
had spillover effects (e.g., comparison observations within the same village, for had an impact in the comparison group.
awareness-raising projects).
5 Self-selection of Score LOW risk if: LOW Project activities were largely conducted at the
project participants The comparison group is exploiting an experiment or natural experiment. community level and there were no participant lists
Units are randomly selected at community level both in the intervention and at the household level. The comparison group is the
comparison group. only other compound in Lusaka, in addition to the
two intervention compounds, that was part of
The selection process for the comparison group is mimicking the same selection Oxfam’s humanitarian response to the 2018 cholera
process used by the project. outbreak. To isolate the impact of the project, the
comparison group selected was the only possibility.

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
64
 Score MEDIUM risk if:
If the self-selection is corrected during the matching procedure (e.g., controlling
for group participation at baseline).

Score HIGH risk if:

Project participants were selected or self-selected based on idiosyncratic or
unobservable characteristics, and the selection of comparison interviewees is
done randomly from neighbouring geographical sites.
6 Other interventions in Score LOW risk if: MEDIUM The institutions the project worked with, both
the comparison group There are no other actors in the area (e.g., INGOs, NGOs, governmental governmental and non-governmental, are involved
programmes). in various ongoing efforts to improve water and
Other actors are conducting activities which are not linked to the project’s theory sanitation facilities and services in the peri-urban
of change. areas of Lusaka, including in the intervention and
comparison compounds. The project provided
system-level support in line with these ongoing
Score MEDIUM risk if: activities for the intervention group only.
Other actors are conducting similar activities linked to the project’s theory of
change in both the intervention and the comparison group.

Score MEDIUM-HIGH risk if:

Other actors are conducting similar activities linked to the project’s theory of
change in the comparison group only, but the evaluation purposefully chooses to
compare these activities to the intervention making it clear that the impact is
compared with these other activities (e.g., as a natural experiment).

Score HIGH risk if:

Other actors are conducting similar activities in comparison communities only.
Other actors are conducting activities in the comparison communities, which are
not the same, but are partially related to the project’s theory of change.
Analysis
7 Representativeness Score LOW risk if: LOW During analysis for George Compound, 4% of the
During analysis or matching procedure less than 10% of the sample in the sample in the intervention group was excluded; for
intervention group is excluded. Chawama Compound, 7% of the intervention group
was excluded.
Score HIGH risk if:
During analysis or matching procedure more than 10% of the sample in the
intervention group is excluded.

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
65
 8 Robustness checks Score LOW risk if: LOW Different econometric model specifications – PSM
Magnitude and statistical significance of the results are approximately consistent and PS-weighted OLS regression – produced
with different econometric models. approximately consistent results.

Score HIGH risk if:

Results are not consistent with different econometric models.
9 Triangulation Score LOW risk if: LOW Challenges described in KIIs are reflective of the
Results are triangulated and consistent with other evaluation methods within the results seen from quantitative analysis.
same evaluation.
Results are triangulated and consistent with other data on the same project but
from different evaluations.

Score HIGH risk if:

Results are not consistent or triangulated with other evaluation methods.
10 Multiple hypothesis Score LOW risk if: LOW This evaluation drafted a pre-analysis plan prior to
testing Multiple hypothesis tests apply Benjamini or Bonferroni tests. data analysis and followed the plan.
The evaluation drafted a pre-analysis plan before analysis and followed the plan.

Score MEDIUM risk if:

The evaluation drafted a pre-analysis plan prior data analysis and significantly
altered the plan, but changes that are clearly justified.

Score HIGH otherwise.

11 Clustering Score LOW risk if: HIGH Clustering was not possible with only two project
Clustering is applied. areas and one comparison area.
Clustering was tested but rejected as providing higher standard errors than non-
clustering estimates.

Score HIGH otherwise.

Other
12 Other Any other issue reported by the evaluator. LOW Data gathered in the comparison area on or after 23
November may have been biased due to flooding.
During analysis, indicator variables were checked
but there were no systematic differences between
data gathered before versus after the flooding.

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
66
REFERENCES
Abadie, A. and Imbens, G.W. (2008). On the Failure of the Bootstrap for Matching Estimators.
Econometrica, 76(6), 1537–57.

Alkire, S. and Foster, J. (2011). Counting and Multidimensional Poverty Measurement. Journal of Public
Economics, 95(7–8), 476–87.

Banerjee, S. G. & Morella, E. (2011). Africa's water and sanitation infrastructure: access, affordability,
and alternatives. The World Bank.

Bartram, J., Brocklehurst, C., Fisher, M. B., Luyendijk, R., Hossain, R., Wardlaw, T., & Gordon, B.
(2014). Global monitoring of water supply and sanitation: history, methods and future
challenges. International journal of environmental research and public health, 11(8), 8137–8165.

Bishop, D., & Bowman, K. (2014). Still learning: a critical reflection on three years of measuring
women's empowerment in Oxfam. Gender & Development, 22(2), 253–269.

Bland, M. & Altman, D.G. (1997). Statistics Notes: Cronbach's Alpha. British Medical Journal,
314(7080), 572.

Bratton, M., & Gyimah-Boadi, E. (2016). Do Trustworthy Institutions Matter for Development?
Corruption, Trust and Government Performance in Africa.

Caliendo, M. & Kopeinig, S. (2008). Some practical guidance for the implementation of propensity score
matching. Journal of Economic Surveys, 21, 31–72.

CDC (2018). Morbidity and Mortality Weekly Report (MMWR): Cholera Epidemic – Lusaka, Zambia,
October 2017–May 2018. https://www.cdc.gov/mmwr/volumes/67/wr/mm6719a5.htm

Eubank, N. (2014). Zambian 2006 to 2010 Constituency and Ward Boundaries. Available at:
http://purl.stanford.edu/yc436vm9005

Filmer, D. & Pritchett, L.H. (2001). Estimating Wealth Effects without Expenditure Data – Or Tears: An
Application to Educational Enrollments in States of India. Demography, 38(1), 115–32.

Fuller, R. & Lain, J. (2015). Measuring resilience: Lessons learned from measuring resilience in
Oxfam’s large-N Effectiveness Reviews. https://policy-practice.oxfam.org.uk/publications/measuring-
resilience-lessons-learned-from-measuring-resilience-in-oxfams-large-583601

Giné-Garriga, R., & Pérez-Foguet, A. (2018). Measuring sanitation poverty: A multidimensional

measure to assess delivery of sanitation and hygiene services at the household level.

Godlonton, S., Hernandez, M., & Murphy M. (2018). Anchoring Bias in Recall Data: Evidence from
Central America. American Journal of Agricultural Economics, 100(2), 479–501.

Heckman, J., LaLonde, R., & Smith, J. (1999). The economics and econometrics of active labor market
programs, Handbook of Labor Economics, 3(A), 1865–2097.

Hughes K. & Bushell, H. (2013). A multidimensional approach to measuring resilience. https://policy-

practice.oxfam.org.uk/publications/a-multidimensional-approach-to-measuring-resilience-302641

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
67
Hopkins A., Brady, C., & Brownlie, A. (2014). The Right to be Heard Framework: A learning companion.
Oxfam GB: Oxford. https://policy-practice.oxfam.org.uk/publications/the-right-to-be-heard-framework-a-
learning-companion-254793

HWISE (2019). Household Water Insecurity Experiences (HWISE) Scale.

https://sites.northwestern.edu/hwise/

Kayser, G. L., Moriarty, P., Fonseca, C., & Bartram, J. (2013). Domestic water service delivery
indicators and frameworks for monitoring, evaluation, policy and planning: a review. International journal
of environmental research and public health, 10(10), 4812–4835.

Leuven, E. & Sianesi, B. (2018). PSMATCH2: Stata module to perform full Mahalanobis and propensity
score matching, common support graphing, and covariate imbalance testing.

Lombardini, S., Bowman, K., & Garwood, R. (2017) A 'How To' Guide To Measuring Women’s
Empowerment: Sharing experience from Oxfam's impact evaluations. https://policy-
practice.oxfam.org.uk/publications/a-how-to-guide-to-measuring-womens-empowerment-sharing-
experience-from-oxfams-i-620271

Medland, L. (2018). Outcomes-Based Programming for Water and Sanitation: Oxfam's vision and
monitoring framework. https://policy-practice.oxfam.org.uk/publications/outcomes-based-programming-
for-water-and-sanitation-oxfams-vision-and-monitorin-620573

Mizniak, J., Wildman, T., & Medland, L. (2017). A Global Water and Sanitation System That Works for
All: Oxfam GB’s Sustainable Water and Sanitation Strategy. https://policy-
practice.oxfam.org.uk/publications/a-global-water-and-sanitation-system-that-works-for-all-oxfam-gbs-
sustainable-w-620333

Nicola, F. & Giné, X. (2012). How Accurate Are Recall Data? Evidence from Coastal India. The World
Bank. Policy Research Working Paper 6009.

Porteous, D. (2006). The enabling environment for mobile banking in Africa.

Rosenbaum, P. & Rubin, D. (1983). The central role of the propensity score in observational studies for
causal effects. Biometrika, 70:41–55.

Shilling, F., Khan, A., Juricich, R., & Fong, V. (2013). Using indicators to measure water resources
sustainability in California. In World Environmental and Water Resources Congress 2013: Showcasing
the Future (2708–2715).

Thomas, E., Andrés, L. A., Borja-Vega, C., & Sturzenegger, G. (Eds.). (2018). Innovations in WASH
Impact Measures: Water and Sanitation Measurement Technologies and Practices to Inform the
Sustainable Development Goals. The World Bank.

UN (2018). Sustainable Development Goal (SDG) 6: Ensure availability and sustainable management
of water and sanitation for all. https://sustainabledevelopment.un.org/sdg6

Vonk, J. (2019). Going Digital: Privacy and data security under GDPR for quantitative impact
evaluation. https://policy-practice.oxfam.org.uk/publications/going-digital-privacy-and-data-security-
under-gdpr-for-quantitative-impact-eval-620884

WHO Africa (2018). Zambia cholera outbreak situation reports.

https://www.afro.who.int/countries/zambia/situation-reports/cholera

WHO/UNICEF (2017). JMP Methodology: 2017 Update & SDG Baselines.

https://washdata.org/sites/default/files/documents/reports/2018-04/JMP-2017-update-methodology.pdf

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
68
Wilbur, J., & Danquah, L. (2015). Undoing inequity: water, sanitation and hygiene programmes that
deliver for all in Uganda and Zambia – an early indication of trends. In 38th WEDC International
Conference, Loughborough University, UK.

Young, S. L., Collins, S. M., Boateng, G. O., Neilands, T. B., Jamaluddine, Z., Miller, J. D., Schuster, R.
C. et al. (2019). Development and validation protocol for an instrument to measure household Water
Insecurity across cultures and ecologies: the Household Water InSecurity Experiences (HWISE) Scale.
BMJ open, 9(1), bmjopen-2018.

Sustainable Water and Sanitation in Zambia: Impact evaluation of the ‘Urban WASH’ project
Effectiveness Review series 2018/19
69
Oxfam Effectiveness Reviews

For more information, or to comment on this report, email [email protected]

© Oxfam GB February 2021

This publication is copyright but the text may be used free of charge for the purposes of advocacy, campaigning,
education, and research, provided that the source is acknowledged in full. The copyright holder requests that all
such use be registered with them for impact assessment purposes. For copying in any other circumstances, or for
re-use in other publications, or for translation or adaptation, permission must be secured and a fee may be
charged. Email [email protected].

The information in this publication is correct at the time of going to press.

ISBN: 978-1-78748-728-4
DOI: 10.21201/2021.7284

Oxfam GB, Oxfam House, John Smith Drive, Cowley, Oxford, OX4 2JY, UK.

OXFAM
Oxfam is an international confederation of 20 organizations networked together in 67 countries, as part of a global
movement for change, to build a future free from the injustice of poverty. Please write to any of the agencies for
further information, or visit www.oxfam.org.

Oxfam America (www.oxfamamerica.org)

Oxfam Australia (www.oxfam.org.au)
Oxfam-in-Belgium (www.oxfamsol.be)
Oxfam Brasil (www.oxfam.org.br)
Oxfam Canada (www.oxfam.ca)
Oxfam France (www.oxfamfrance.org)
Oxfam Germany (www.oxfam.de)
Oxfam GB (www.oxfam.org.uk)
Oxfam Hong Kong (www.oxfam.org.hk)
Oxfam IBIS (Denmark) (http://oxfamibis.dk/)
Oxfam India (www.oxfamindia.org)
Oxfam Intermón (Spain) (www.oxfamintermon.org)
Oxfam Ireland (www.oxfamireland.org)
Oxfam Italy (www.oxfamitalia.org)
Oxfam Mexico (www.oxfammexico.org)
Oxfam New Zealand (www.oxfam.org.nz)
Oxfam Novib (Netherlands) (www.oxfamnovib.nl)
Oxfam Québec (www.oxfam.qc.ca)
Oxfam South Africa (http://www.oxfam.org.za/)
KEDV (www.kedv.org.tr)

www.oxfam.org.uk/effectiveness