WaSH MEL COMPENDIUM - Water Institute3.1. 3.2. 3.3. Lesson 1: Sustainability Requires...

C O M P E N D I U M O F B E S T P R A C T I C E S A N D L E S S O N S L E A R N E D

COMPENDIUM

A PROJECT FUNDED BY THE CONRAD N. HILTON FOUNDATION

of Best Practicesand Lessons Learned

M E LW a S H

2016O C T .

aa

aaaaaa

y β

xx

yβ Δ

Δ

x

xy

Δaa

Compendium

A PROJECT FUNDED BY THE CONRAD N. HILTON FOUNDATION

of Best Practicesand Lessons Learned

M E LW a S H

AuthorsMike Fisher, Ryan Cronk, Allison Fechter, Pete Kolsky,Kaida Liang, Emily Madsen, Shannan George

EditorsDavid Fuente, Jamie Bartram

2016O C T .

A B O U T T H E W A T E R I N S T I T U T E

The mission of the Water Institute at UNC is to provide

global academic leadership for economically, environmentally,

socially and technically sustainable management of water, sanita-

tion and hygiene (WaSH) for health and human development and

to be a vibrant, interdisciplinary center that unites faculty, students

and partners from North Carolina and developed and develop-

ing nations worldwide. We develop solutions to improve water,

sanitation and hygiene for all. We make our work relevant and

practical by linking research with policy and practice. Since our

launch in 2010, we’ve shared new insights and knowledge that

have informed the work of local and national governments and in-

ternational aid organizations—including the World Bank, World

Health Organization, and UNICEF.

Our four main strategic functions are research, teaching and

learning, knowledge information management, and network-

ing and partnership development. Through research, we tackle

knowledge gaps that impede effective action on important WaSH

and health issues. We respond to the information needs of our

partners, act early on emerging issues and proactively identify

knowledge gaps. By developing local initiatives and international

teaching and learning partnerships, we deliver innovative, relevant

and highly accessible training programs that will strengthen the

next generation’s capacity with the knowledge and experience to

solve water and sanitation challenges. By identifying or develop-

ing, synthesizing and distributing relevant and up-to-date knowl-

edge and information on WaSH, we support effective policy and

decision-making that protects health and improves human devel-

opment worldwide, both predicting and helping prevent emerging

risks. Through networking and partnership development, we bring

together individuals and institutions from diverse disciplines and

sectors, enabling them to work together to solve the most critical

global issues in water and health.

i i iA C R O N Y M S A N D A B B R E V I A T I O N S

A C R O N Y M S A N D A B B R E V I A T I O N S

CNHF Conrad N. Hilton Foundation

CQI Continuous quality improvement

E. coli Escherichia coli

ICTs Information and communication technologies

IS Implementation science

JMP Joint Monitoring Programme

M&E Monitoring and evaluation

MEL Monitoring, evaluation and learning

MSTs Mobile survey tools

MDG Millennium Development Goal

NGO Nongovernmental organization

PIMS Post-implementation monitoring surveys

QA/QC Quality assurance / quality control

SDG Sustainable Development Goal

UNC The University of North Carolina–Chapel Hill

UNICEF The United Nations Children’s Fund

WaSH Water, sanitation and hygiene

WHO World Health Organization

WV World Vision

iv M E LW a S H C O M P E N D I U M O F B E S T P R A C T I C E S A N D L E S S O N S L E A R N E D

Ensure availabil ity and sustainable managementof water and sanitation for all

U N I T E D N A T I O N S

S U S T A I N A B L E D E V E L O P M E N T G O A L 6

Targets

6.1 By 2030, achieve universal and equitable access to safe and affordable

drinking water for all

6.2 By 2030, achieve access to adequate and equitable sanitation and hygiene

for all and end open defecation, paying special attention to the needs of

women and girls and those in vulnerable situations

6.3 By 2030, improve water quality by reducing pollution, eliminating

dumping and minimizing release of hazardous chemicals and materials,

halving the proportion of untreated wastewater and substantially

increasing recycling and safe reuse globally

6.4 By 2030, substantially increase water-use efficiency across all sectors and

ensure sustainable withdrawals and supply of freshwater to address water

scarcity and substantially reduce the number of people suffering from

water scarcity

6.5 By 2030, implement integrated water resources management at all levels,

including through transboundary cooperation as appropriate

6.6 By 2020, protect and restore water-related ecosystems, including

mountains, forests, wetlands, rivers, aquifers and lakes

6.a By 2030, expand international cooperation and capacity-building support

to developing countries in water- and sanitation-related activities and

programmes, including water harvesting, desalination, water efficiency,

wastewater treatment, recycling and reuse technologies

6.b Support and strengthen the participation of local communities in

improving water and sanitation management

vC O M P E N D I U M O F B E S T P R A C T I C E S A N D L E S S O N S L E A R N E D M E L : M O N I T O R I N G , E V A L U A T I O N A N D L E A R N I N G

MONITORING is data collection and analysis

to guide implementation and progress over a period of time.

Monitoring answers the questions:

Are we on track to deliver what we promised?

Where are we behind or ahead?

Where are the opportunities for improvement?

EVALUATION is the systematic and objective

appraisal of a project/program (usually by a third party) to

assess impact and guide future policy. Evaluation answers the

questions:

How did we get here?

How would we improve next time?

LEARNING is the systematic process by which

insights from monitoring and evaluation are applied to

improve programs and interventions.

What is MEL?Monitoring, evaluation and learning

vi M E LW a S H C O M P E N D I U M O F B E S T P R A C T I C E S A N D L E S S O N S L E A R N E D

T A B L E O F C O N T E N T S

INTRODUCTION1.

3.3 .1 .

3 .3 .2 .

3 .3 .3 .

3 .3 .4 .

3 .3 .5 .

The State of WaSH MEL among CHNF Grantees

The Data Dilemma

From M&E to MEL

A New Generation of Data

The Focus on Improvement

1.1.1.2.1.3.

Improving WaSH Programs by Monitoring, Evaluation and LearningSustainable Development Goal 6 and Improving WaSH Service DeliveryA Need for Suitable Data

HISTORY OF MEL2.

LESSONS LEARNED FROM MEL3.3.1.3.2.3.3.

Lesson 1: Sustainability Requires High-Performing Management Systems Lesson 2: Household Water Quality is a Widespread ChallengeLesson 3: Continuous, Targeted Improvements Are Rare and Require a Mindset Shift

3.4. Lesson 4: Complex WaSH Problems Can Be Solved Using CQI Methods in Ways that Would Never Be Possible with Traditional Monitoring

3.4 .1 .

3 .4 .2 .

3 .4 .3 .

3 .4 .4 .

3 .4 .5 .

3 .4 .6 .

Background

Compounding the Problems with a Project-Oriented Approach

CQI: A Systems and Process-Oriented Approach to Complex Problem Solving

CQI Enables Precise, Data-Driven Actions to Improve Outcomes

CQI is Adaptive

Effective, Targeted Improvements Require Data and Evidence

3.5. Lesson 5: Progress is Unequal Across Countries and Not Always Where Expected

WaSH MEL Best PracticesI

About the Water InstituteAcronyms and AbbreviationsUnited Nations Sustainable Development Goal 6What is MEL?

vi iC O M P E N D I U M O F B E S T P R A C T I C E S A N D L E S S O N S L E A R N E D T A B L E O F C O N T E N T S

PRINCIPLES OF MONITORINGFOR IMPROVEMENTPrinciples of Generating Fit-for-Purpose Data4.1.

4.

4.1 .1 .

4 .1 .2 .

4 .1 .3 .

4 .1 .4 .

4 .1 .5 .

4 .1 .6 .

4 .1 .7 .

4 .1 .8 .

4 .1 .9 .

4 .1 .10.

4 .1 .11.

Asking the Right Questions: Outputs, Outcomes and Process Indicators

Sampling and Sample Size Calculations

Measuring X and Y Variables

Methods of Data Collection (Measurement, Direct Observation and Direct Response)

Crafting Robust Survey Questions and Operational Definitions: Avoiding Bias, Jargon, Constructs and Other Pitfalls

Data Collection: Best Practices and Pitfalls

Selecting and Using Information and Communication Technologies

Hands-On Training

Quality Assurance / Quality Control and Reviewing Data

Regular Refresher Training

Proper Data Analysis

4 .3 .1 .

4 .3 .2 .

4 .3 .3 .

4 .3 .4 .

4 .3 .5 .

4 .3 .6 .

Output Emphasis and Lack of Adequate Outcome Metrics

Lack of Adequate Sampling and Sampling Size

Lack of Adequate Monitoring Tools

Bias and Errors

Absence of Quality Control

Problematic Assumptions

Common Mistakes and Pitfalls4.3.

Checklist for WaSH MEL Implementation4.2.

LEVERAGING MEL: TURNINGM&E FIT-FOR-PURPOSE DATAINTO IMPROVEMENTBacking Up Evidence with Action5.1.

5.

5.3 .1 .

5 .3 .2 .

5 .3 .3 .

Suitability of CQI for Addressing Complex WaSH Problems

Adaptation of CQI Methods to WaSH

Implementing CQI in WaSH Programs

An Improvement Mindset5.2.Review of CQI as a Method for Addressing Complex Problems5.3.

vi i i M E LW a S H C O M P E N D I U M O F B E S T P R A C T I C E S A N D L E S S O N S L E A R N E D

Tools, Evidence and Learningsfrom WaSH MEL 2012-2016

I I

APPENDICES 1–VIII

T A B L E O F C O N T E N T S ( C O N T . )

C O M P E N D I U M O F B E S T P R A C T I C E S A N D L E S S O N S L E A R N E D

WaSH MELBest Practices

IP A R T

2 M E LW a S H C O M P E N D I U M O F B E S T P R A C T I C E S A N D L E S S O N S L E A R N E D

INTRODUCTION

I M P R O V I N G W A S H P R O G R A M SB Y M O N I T O R I N G , E V A L U A T I O NA N D L E A R N I N G

1.1.

Access to safe water, basic sanitation and good hygiene (WaSH) are critical to human

health and development. In recognition of the fundamental importance of these services,

governments, bilateral and international aid organizations, philanthropic foundations

and nongovernmental organizations (NGOs) have prioritized investments in WaSH

programs in recent decades. Monitoring, evaluation and learning (MEL) techniques play

an important role in tracking and enhancing the impact of WaSH programs.

Earlier efforts focused on expanding access to basic services, which was reflected in

international targets for water and sanitation coverage as expressed in the Millennium

Development Goals and other international development agendas such as the International

Drinking Water Supply and Sanitation Decade (1981-1990). In recent decades, there is a

growing recognition of the need to improve the quality of WaSH services, maximizing

their widespread and continuous use (not just “access”) and ensuring continuity

and sustainability over time. In addition, there is a growing recognition that not all

investments in WaSH services are equally effective and that some programs achieve far

greater impact per dollar invested than others.

In light of these recognitions, WaSH implementers and funders seek to improve

their programs and investments to maximize impact and efficiency—more specifically,

to provide water that is safe and reliable in adequate quantities to meet users’ basic needs

and to do so on a sustainable basis. This implies the need to ensure adequate chemical and

microbial quality of water for consumption, both at the source and at the point of use, in

contexts where continuous piped water at the home is not regularly available. In terms of

sanitation, this means ensuring that populations have access to and regularly make use of

improved sanitation facilities that meet their needs (with respect to functionality, privacy,

safety, and accessibility), and that excreta are safely disposed of to minimize the likelihood

that they will subsequently contaminate the human environment. In terms of hygiene,

1.

aa

aaaaaa

y β

xx

yβ Δ

Δ

x

xy

Δ

aa

3C O M P E N D I U M O F B E S T P R A C T I C E S A N D L E S S O N S L E A R N E D 1 . I N T R O D U C T I O N

this means ensuring that populations have access to and make use of basic handwashing

supplies such as soap and water, a goal with a major behavioral component.

Implementers increasingly have prioritized funding of water, sanitation and hygiene

together in recent decades, recognizing the synergistic interactions among interventions

in these areas. With respect to all dimensions of WaSH, implementers and funders are

interested in improving not only the quality and coverage of services—and their continuity

and sustainability over time—but also the cost-effectiveness of these services and their

delivery, in order to maximize the impact of programs with finite resources. Funders and

donors also are increasingly seeking to achieve these improvements in collaboration with

local governments and to credibly document the impacts of their activities.

S U S T A I N A B L E D E V E L O P M E N T G O A L 6A N D I M P R O V I N G W a S H S E R V I C E D E L I V E R Y

1.2.

The Sustainable Development Goals (SDG), agreed upon in 2015 by the

United Nations General Assembly, include an emphasis on improving

the quality of WaSH service delivery, as well as on expanding access

to these services. Goal 6 (“Ensure availability and sustainable

management of water and sanitation for all”) calls for, among

other targets, several related to water, sanitation and hygiene (see box).

The World Health Organization/UNICEF Joint Monitoring

Programme for Water Supply and Sanitation has developed indicators and

operational definitions associated with these targets that call for safe water

to be free from microbial and priority chemical contaminants and to be

reliably available in adequate quantities close to home. These indicators

and definitions also call for adequate sanitation that hygienically separates

excreta from human contact and includes safe disposal.

Achieving these targets will require substantial increases in access to

basic WaSH services, especially as populations continue to grow; it will

also require WaSH implementers to deliver more services of higher quality

more effectively than ever before. Many program, project, national and

global monitoring initiatives are being improved and adapted to address

SDG priorities; and new monitoring initiatives are being developed.1 Data

experts expect billions of dollars of investment in monitoring initiatives

and new data collection during the SDG era.2

SDG Goal 6WaSH-related Targets

• By 2030, achieve universal and

equitable access to safe and

affordable drinking water for all

• By 2030, achieve access to adequate

and equitable sanitation and hygiene

for all and end open defecation,

paying special attention to the needs

of women and girls and those in

vulnerable situations

• By 2030, expand international

cooperation and capacity-building

support to developing countries

in water- and sanitation-related

activities and programs, including

water harvesting, desalination, water

efficiency, wastewater treatment,

recycling and reuse technologies

• Support and strengthen the

participation of local communities

in improving water and sanitation

management

1 WHO, UNICEF. 2015. Methodological note: Proposed indicator framework for moni-toring SDG targets on drinking-water, sanitation, hygiene and wastewater. Geneva: WHO.2 Espey, J. 2015. Data for development: A needs assessment for SDG monitoring and statistical capacity development. Sustainable Development Solutions Network.


A N E E D F O R S U I T A B L E D A T A1.3.

The dedicated SDG targets provide a useful framing and policy justification for actors

at the program, subnational, national and global levels to achieve universal coverage of

basic water and sanitation and to improve services. Achieving these targets and priorities

will require more and better evidence from project, program, subnational and national

monitoring data. Such evidence normally contributes to better service delivery outcomes,

but these data are often evaluated in a limited capacity where there is more added value

than present analyses derive and/or data are low quality. Improving the quality of

monitoring data and conducting service delivery research using monitoring data may lead

to greater insight and opportunities to improve water and sanitation services in the SDG

era. •

HISTORY OF MELIn 2010, the Conrad N. Hilton Foundation (CNHF) launched its five-year Strategy

for Sustainable Safe Water Access. Although its specific language was updated during the

strategy period, its broad aims remained constant—to provide sustainable safe water access

for one million people by 2015. As part of this strategy, CNHF sought to fund a greater

diversity of water programs (with more varied implementation approaches) in both West

Africa and other regions and to document the impacts of these programs more rigorously.

While the contribution of earlier programs such as CNHF’s West Africa Water

Initiative to the expanding coverage of water and sanitation services was likely

substantive, documentation of these efforts was limited and the implementation methods

used (i.e., boreholes with handpumps, etc.) were largely perceived by CNHF to be

conventional, in tension with the foundation’s desire to foster innovative new approaches

alongside proven existing methods.

In mid-2012, CNHF provided support to the Water Institute (WI) at the

University of North Carolina–Chapel Hill (UNC) to create a monitoring, evaluation

and learning (MEL) framework for its global safe water portfolio in response to a growing

need to document and enhance the impact of its programs. The WI’s proposal emphasized

the use of continuous quality improvement (CQI) methods to leverage monitoring and

evaluation (M&E) data for improvement in WaSH programs, something that had not

been done previously in the WaSH sector.

When the WI began reviewing CNHF’s water program, the foundation was

providing financial support through grants to eight implementing partners, plus several

knowledge and advocacy partners. Of the implementing partners working in the seven

countries receiving funding from the foundation, most were conducting varying levels

2.

Countries receivingCNHF funding

Burkina Faso

Ethiopia

Ghana

India

Mali

Mexico

Niger

5C O M P E N D I U M O F B E S T P R A C T I C E S A N D L E S S O N S L E A R N E D 2 . H I S T O R Y O F M E L

of monitoring and/or evaluation activities (e.g., post-implementation monitoring

surveys by WaterAid and the WaSH Bottleneck Analysis Tool by UNICEF). Grantees

were conducting monitoring and evaluation activities and programs that were designed

for internal purposes and would have been very difficult to export and scale to other

programs and projects.

At the September 2012 Hilton Foundation Grantee Convening in Accra, Ghana,

the WI gathered information from grantees about the top common challenges they were

facing: functionality, sustainability, urbanization, financing, water quality and the need for

systematic and targeted advocacy efforts and capacity building. In addition, a lack of data

from all project and program levels, including from the community level, was identified

as a common challenge. The lack of robust or credible data remains a problem across the

sector today.

Grantees indicated that monitoring at the community level is difficult

and that the traditional practice of M&E is conducted at the project level but not the

community level. Also, grantees agreed that existing monitoring tools are too

numerous and confusing to meet their needs. Often, different monitoring activities

were expected by multiple stakeholders (donors, government and other organizations),

leading to implementers simultaneously conducting multiple monitoring efforts using

different frameworks and methods to track similar projects in a given setting. Specific

challenges of monitoring include too many or too few indicators as

well as a lack of common operational definitions and methods for

measurement (e.g., how to count beneficiaries). Finally, the grantees focused on the

overall perception of M&E and its value to their work They stated that M&E exercises

often had little impact on programs and projects and that monitoring for

the sake of monitoring was of little value and often lacked purpose.

During the initial convening as well as in subsequent surveys, CNHF grantees

revealed that they wanted M&E processes built in from the start of projects and programs

so they could track progress from the beginning. They also requested a systematic way to

evaluate and learn from their WaSH projects with simple M&E tools and methods, which

would mean using a few critical indicators and cost-effective ways to measure and prove

causality. Finally, they requested that time, capacity building and funding be properly

allocated to allow for dedicated M&E.

After the convening of the grantees, the WI conducted a thorough literature review,

a review of grantee M&E frameworks and indicators, a sector-wide search for common

indicators and a consultation with WaSH experts and leaders. The core set of WaSH

indicators was presented to CNHF and its grantees at a closed meeting at the 2013 UNC

Water and Health Conference. The set of core indicators was designed to be simple (hence


only 21 indicators were chosen), measurable (the WI also presented working definitions

for each indicator) and relevant across all WaSH, not just water projects and programs. A

few individuals from grantee organizations felt the indicators were not tailored enough to

the needs of their projects and were vocally against the draft indicator set. •

LESSONS LEARNED FROM MEL

L E S S O N 1 : S U S T A I N A B I L I T YR E Q U I R E s H I G H - P E R F O R M I N GM A N A G E M E N T S Y S T E M S

3.1.

Sustainability of community water systems has long been a concern for WaSH-

implementing organizations. Breakdowns of water systems are known to occur frequently

in low- and middle-income countries, resulting in a discontinuity and/or absence of

services that can have implications for human health and development.

In seeking to increase access to water supply over the last 50 years, it has perhaps

been natural for international support agencies, NGOs and sector experts to focus on

hardware and technology. It was easy to see that drinking water from ponds, open streams

or unprotected shallow wells without treatment was dangerous, and it seemed that the

biggest obstacle to a safe water supply was the lack of boreholes, pumps, pipes and/or

treatment systems. Such a view of the problem also suggested that international aid had

3.Throughout the implementation of the WaSH MEL program by the WI in collaboration

with CNHF grantees, several lessons were learned from the monitoring data collected,

the experiences of WI staff and CNHF grantees, and from additional projects and tasks

requested by CNHF during the grant period. Five of the most salient are, in summary:

L E S S O N 1

Sustainability requires high-performing management systems.

L E S S O N 2

Household water quality is a widespread challenge.

L E S S O N 3

Continuous, targeted improvements are rare and require a mindset shift.

L E S S O N 4

Complex WaSH problems can be solved using CQI methods in ways that would never be possible with traditional monitoring.

L E S S O N 5

Progress is unequal across countries and not always where expected.

7C O M P E N D I U M O F B E S T P R A C T I C E S A N D L E S S O N S L E A R N E D 3 . L E S S O N S L E A R N E D F R O M M E L

L E S S O N 1 : S U S T A I N A B I L I T YR E Q U I R E s H I G H - P E R F O R M I N GM A N A G E M E N T S Y S T E M S

3 Fisher, M. B., et al. 2015. Understanding handpump sustainability: Determinants of rural water source functionality in the Greater Afram Plains region of Ghana. Water Resources Research 51(10): 8431–49.

great potential to solve the problem; foreign assistance could, with relative ease and speed,

purchase capital equipment and well-drilling expertise to drill a large number of boreholes

in the neediest areas of rural Africa and Asia. A frequent model for assistance in water

supply relied on an external support agency to provide some or all of the initial capital to

build the system, while leaving operations and maintenance to community management.

It was implicitly assumed that communities would recognize the enormous value of the

water supply and organize themselves to manage the system.

It has since become clear that people can be denied sustainable access to

safe water in at least two ways other than a lack of technology to assure

a good water source nearby. One is contamination between the water source and the

household, discussed further in Lesson 2. The second is perhaps the toughest challenge in

assuring sustainable and safe water supplies: frequent breakdown of existing systems due

to insufficient management capacity for the water supply at local levels. Villagers staring

at a broken handpump are no better off than villagers with no handpump at all—even if

thousands of dollars were provided to drill the borehole.

While management of a “simple” rural water supply may not require high levels of

technical expertise, it does have certain basic institutional and financial requirements that

have been widely neglected in practice. In addition to access to a competent mechanic,

sustainability requires such social assets as trust, accountability, incentives to maintain

and repair systems, and the ability to raise funds to cover running costs and effect repairs.

Increasingly, support agencies have come to recognize the importance of these “software”

issues and are giving more attention to local-level capacity building and support.

Recent research by the WI has explored the relationship between water system

sustainability and high-functioning management systems. A recently reported WI study

on handpump sustainability in rural Ghana3 where CNHF grantees had implemented

projects revealed a link between water source functionality and management determinants

(presence of identifiable management, access to tools and spare parts, savings, collection

of a tariff and external technical support). The study considered a wide range of

hydrogeological, technological and institutional variables calculated from data collected

from 1,509 water sources serving 570 communities in the Greater Afram Plains in Ghana.

Villagers staring at a broken handpump are no better off than villagers with no handpump at all—even if thousands of dollars were provided to drill the borehole.


Communities with identifiable management were more than twice as likely to have

functional water points than communities without an identifiable management system.

A Bayesian network model was developed to identify the determinants of

functionality. The authors found that nearly 80% of water points were functioning when

visited, which underlines the key role of functionality of existing systems in assuring

access. Eighty percent functionality suggests that 80% of the communities are enjoying

a functioning water supply, but it also means that one in five communities do not have

access to a functioning water supply at the moment, despite the large investments made

to provide one. Furthermore, in most cases, the damage is far from irreparable, leading to

a performance focus upon minimizing the time the system is out of service. It is perhaps

not surprising that management capacity is a key determinant of functionality, through

minimizing the overall downtime of the system.

The WI study found that a base starting functionality of 72% (i.e., nearly

¾ of the systems working at the time of a spot check) could be increased to 97%

with optimal management systems and available tools. The Bayesian

model suggests that effective tariff collection and a management team identifiable in the

community are both highly correlated with working systems. Both tariff collection and

availability of tools may be as significant as the model suggests or may be indicative of

other management factors that account for higher functionality.

In addition, WI researchers used qualitative research methods to investigate factors

that affect the sustainability and functionality of community-managed drinking water

systems. A WI researcher developed a set of rehabilitation pathways4 to examine the steps

in the process of initiating and completing a water system repair, in order to facilitate

the identification of weak links in the chain. Participatory field research was conducted

in World Vision (a CNHF grantee) communities in Kenya, Ghana and Zambia.

Key findings included the importance of directing management training at the entire

water management committee (not just one or a few water committee members) and

encouraging communities to mobilize resources proactively rather than reactively for

quicker repairs of breakdowns.

Adoption and documentation of such a framework during systems operation could

provide the first steps in local management CQI that can eventually yield the simple—yet

high-functioning—systems required to keep safe water flowing in rural communities.

Research examining the link between sustainability and successful management

systems, both within and outside of the WI, indicates that building and maintaining the

capacity of water management entities may be essential to maximizing the long-term

sustainability of water systems.3,4 It is important for WaSH implementers to recognize

this link and allocate the resources necessary to create and maintain highly capable water

management entities.

From the first identification of a breakdown to the completion of its repair, this research identified the actors involved, the constraints that slow rehabilitation and those factors that can prevent repair or rehabilitation from occurring altogether.


4 Klug, V. 2016. Water system breakdown typology and rehabilitation pathways in sub-Saharan Africa. Master’s Technical Report, Dept of Environmental Sciences and Engineering, Gillings School of Global Public Health, UNC.5 Montgomery, M. A., et al. 2009. Increasing functional sustainability of water and sanitation supplies in rural sub-Saharan Africa. Environmental Engineering Science 26.5: 1017–23.

L E S S O N 2 : H O U S E H O L D W A T E R Q U A L I T YI S A W I D E S P R E A D C H A L L E N G E

3.2.

From the very beginning of its involvement in WaSH, CNHF has focused upon drinking

water quality. The rigorous survey work and analysis done in this study strongly suggest

a consistent pattern of drinking water quality challenges faced by those served by CNHF

grantees, summarized in Table 1, Figure 1 and these two statements:

1. The majority of water sources developed by Hilton grantees in

sampled project areas offer water of acceptable or low risk. These

sources conformed with WHO guidelines for either microbial safety (0 bacteria/100

ml) or “low” microbial risk (1-10 bacteria/100 ml). In Ghana 55% of sources fell into

one of these two categories, while in Ethiopia and Burkina Faso about half did (57%

and 64% respectively).

2. The significant majority of water samples stored in houses in these

same areas represent water of dubious quality. They fell into WHO’s

categories of intermediate risk (11-100 bacteria/100 ml) or high risk (>100

bacteria/100 ml.) In Ghana 76% of households in project areas had water falling into

one of these two categories, while in Ethiopia and Burkina Faso, about 80% and 68%

did, respectively.

In short, half or more of the acceptable or low risk water available from the source is

significantly contaminated by the time it is available for home use.

Assessment of multiple CNHF grantees and data collection in six countries revealed

widespread water quality challenges. In Ghana, Burkina Faso and Ethiopia, data collected

in 477 communities where CNHF grantees had implemented projects revealed detectable

E. coli contamination in samples from 60%, 48% and 56% of community water sources,

Sustainability of sanitation and hygiene services is also a major concern and has

major behavioral components; effective community management may play a role in the

long-term sustainability of these services as well. Several factors potentially related to

community management have been linked to the sustainability of rural sanitation services,

including community demand, local financing, and dynamic operation and maintenance.5

Additional work to explore the role of community management in the sustainability of

sanitation and hygiene services is needed and is recommended as a critical research area in

support of countries’ efforts to maximize progress on SDG 6.


while levels of contamination corresponding to the high-risk category according to WHO

guidelines for drinking-water quality (>100 CFU/100 mL) were present in samples

from 36%, 24% and 34% of sources, respectively. In all cases, the proportion of samples

in the high-risk category decreased substantively when only improved sources such as

boreholes and piped sources were considered. By contrast, 80-90% of samples of stored

Table 1. Water Quality Data from Selected Communities Served by CNHF Grantees in Three Countries

Ghana Burkina Faso Ethiopia

Communities 224 95 158

Sources 926 987 155

Improved sources 388 686 115

Households 527 581 864

Detectable E. coli

High Risk* Detectable E. coli

High Risk Detectable E. coli

High Risk

Source water quality 60% 36% 48% 25% 56% 34%

Improved source water quality 45% 17% 24% 4% 48% 28%

Household water quality 85% 52% 83% 42% 90% 60%

Improved source household water quality 82% 46% 81% 33% 90% 60%

*(>100 CFU/100 mL)

Figure 1. Microbial risk of water from Ghana,6 Burkina Faso7 and Ethiopia.8

Ghana Burkina Faso Ethiopia

Sour

ce w

ater

Hou

seho

ld s

hare

d w

ater


water for consumption contained detectable E. coli in all three countries and 40-60% of

these samples were in the high risk category. Furthermore, stored water samples from

households using improved sources were not dramatically less likely to be contaminated

or in the high-risk category than were sorted water samples from households using

unimproved sources.

These findings suggest that across multiple contexts, microbial water quality remains

a serious concern, particularly at the household level, and that many of the water quality

benefits provided by access to improved community water sources may be undone by

secondary contamination during unsafe transport and storage. Subsequent work in other

countries and recent systematic reviews suggests that these results are not unique to

CNHF grantees but characterize rural and peri-urban water quality challenges across a

wide range of low- and middle-income countries. These challenges are rarely captured by

current M&E approaches implemented by many WaSH programs, which rarely collect

water quality samples from sources after initial implementation or from households in

program areas. In the future, WaSH programs should seek to prioritize monitoring and

improving water quality, with a particular emphasis on microbial water quality at the

household level.

These analyses of baseline data provided a starting point for CQI in Ghana, for

example, through an intervention to promote safe water storage containers that are not

susceptible to contamination as people dip their hands with cups or bowls into the water;

these containers are defined as those with a narrow mouth and tight-fitting lid, from

which water is extracted by pouring or dispensing from a tap9 (Figures 2a and 2b).

Among homes that adopted safe water storage containers, the percentage of households

with water that met WHO guidelines increased approximately 70% (from 17% to 29%).

Nevertheless, the challenge of household water quality remains substantial. Even

under the best circumstances in Ghana, where safe water storage was used for water

coming from an improved source, over half the samples were of intermediate or high risk.

The results strongly suggest a need to consider a clearer understanding of the full water

transport chain from well to storage container with a view towards safeguarding quality

along the entire route. Safe storage was correctly identified as one key factor in improving

water quality at the home, but the evidence strongly suggests it is not the only one.

6 Fisher, M., and Liang, K. WaSH MEL Ghana Pilot Interim Report. The Water Institute at UNC, Chapel Hill, NC.7 Williams, A.R. 2016. WaterAid Burkina Faso Baseline Study Report. The Water Institute at UNC, Chapel Hill, NC.8 Shields, K., et al. 2015. MWA Water Quality Study Report [Working Draft]. The Water Institute at UNC, Chapel Hill, NC.9 Mintz, E., et al. 2001. Not just a drop in the bucket: expanding access to point-of-use water treatment systems. American Journal of Public Health 91.10: 1565–70.

mike fisher

Figures 2a, 2b. A traditional water storage container (above) and a safe water storage container (below).


L E S S O N 3 : C O N T I N U O U S ,T A R G E T E D I M P R O V E M E N T S A R E R A R EA N D R E Q U I R E A M I N D S E T S H I F T

3.3.

The Water Institute’s 2013-2015 assessment of CNHF’s Sustainable Strategy for Safe

Water Access included a detailed review and assessment of CNHF grantees’ M&E

frameworks and data where available. It revealed that while many CNHF grantees

had strong programs with a substantive impact on improving access to improved water

sources, none had M&E systems in place that were capable of robustly documenting

their programs’ impact on sustainable access to safe water, as defined by CNHF. Very

few grantees were collecting accurate beneficiary numbers, no partners were consistently

collecting water quality data after installing water points or at the household level, and

very few were collecting data from follow-up visits or tracking service level provision

on a regular basis. Essentially, good or SMART10 monitoring was limited across CNHF

grantees and deficient across the sector in general. Traditional monitoring conducted by

nongovernmental organizations often involved collecting monitoring data for internal

reporting and donor facing progress reports, where outputs were the only metrics

being tracked and reported (e.g., number of water points constructed, the number of

beneficiaries, the number of meetings held with the government) and were usually based

on best estimates and averages. Very few grantees were reporting to or coordinating with

local or national governments.

Three widespread challenges faced by most CNHF grantees were:

1. documentation of the number of people their program served,

2. the quality of water and sanitation services those people received,

and

3. the continuity and sustainability of these services over time.

In most cases, programs used nominal estimates of program coverage, based on

assumptions (e.g., “one handpump serves 300 people,” or “the entire population of this

community will benefit from this new water source”) rather than on credible monitoring

data. In some cases, monitoring data were used but were not collected in a robust manner,

3.3 .1 . The State of WaSH MEL among CHNF Grantees

SMART: Specific, Measurable, Achievable, Relevant and Time-bound

S: Is the question well

defined?

M: How much or how

many of something?

A: Is it realistic?

R: Is it worthwhile to

measure?

T: Is it measurable over

a specific period?

Wherever the other opportunities lie to improve the safety of drinking water, the results

suggest that CNHF grantees and the WI have been correct to act on the obvious truth

that water quality at the household level, and not just at the source, is the key parameter

on which to focus for improvement—and the baseline data suggest the magnitude of the

problem across Africa.


leading to dramatic overestimation of program impact; for example, some grantees used

survey data to estimate the population served, but used methods which incorporated

considerable upward bias, leading to overestimation of total daily users of their programs

by an estimated 200-500%. In all cases, there was a tendency to assume that “proximity

equals access” (i.e., that all individuals living near a water source benefited from that

water source, regardless of the number of people actually using the source, whether it was

working and whether it was producing safe water).

In the case of water quality, many programs conducted water quality analyses at some

point in time, but few grantees had well-designed monitoring systems capable of detecting

chemical and microbial hazards in source and household water. Specifically, grantees

implementing groundwater systems frequently performed a one-off test of chemical

(and sometimes microbial) water quality on a new well before installing a handpump but

rarely conducted water quality testing of existing systems to assess the safety of the water

produced. In no case did CNHF grantees conduct testing of the water consumed by

program beneficiaries at the household level.

Likewise, monitoring of the continuity and sustainability of services was insufficiently

robust in many cases. While some grantees collected data on the functionality and uptime

of the infrastructure they had implemented, many did not. Where data were collected,

the M&E methods used were often insufficient to provide an accurate picture of the true

number of hours per day or days per week that users received service. Likewise, reliable

data on the proportion of water and sanitation facilities that were functional at any given

time were often lacking.

These deficiencies in M&E systems appeared to be characteristic of

many implementers in the broader WaSH sector, rather than of CNHF

grantees in particular. While all grantees conducted some form of M&E and all

believed their activities to be adequate, few were in a position to credibly document

their contributions to CNHF’s Sustainable Safe Water Strategy. This suggests that

many WaSH implementers sectorwide are likewise poorly positioned to document their

contributions to the WaSH-related SDGs.

Furthermore, the assessment of CNHF grantees revealed high performance

in many areas but with a greater emphasis on compliance than

improvement. There may be value in further learning activities, such as basic training

in quality improvement methods, with the objective of fostering an improvement mindset

among CNHF grantees in particular, and the WaSH sector in general, in order to

promote the data-driven improvements needed to increase program quality and efficiency.

10 Doran, G. T. 1981. There’s a SMART way to write management’s goals and objectives. Management Review 70.11: 35–6.


Over the past three and a half years since the WaSH MEL project began, it has become

increasingly evident that many of the issues CNHF grantees identified as main challenges

in implementing robust M&E in their programs also present challenges for the WaSH

sector at large. The sector lacks a core set of validated methods and

standard indicators for WaSH M&E that can be used throughout the sector,

including by governments. When the WI began to build the WaSH MEL framework

for the CNHF and its grantees, researchers conducted an extensive literature review,

compiled a list of all grantee indicators in use at that time, reviewed indicators used by

other WaSH organizations sector-wide and consulted with sector experts and advisors on

best practices. The aim was to develop a parsimonious core set of indicators that would

be simple to measure but robust enough that when analyzed would provide valuable

insights into causes and correlations of multiple variables relevant to all WaSH projects. In

addition, the WI leveraged insight into discussions around the SDGs (not yet finalized at

the time) to align the core WaSH MEL indicators as closely as possible with the nascent

SDG targets related to drinking water and sanitation.

A shift in the WaSH mindset needs to occur as the grantees highlighted in the first

convening, as to “why we do M&E.” Traditional M&E simply provides a measure of

whether a project is succeeding or failing. Data was often collected in traditional M&E

without a clear purpose, which led to poor quality, unused, neglected data and a lot

of wasted resources. Adding learning to traditional M&E programs shifts

the focus to improving and increasing the value and impact of WaSH

investments. The learning component of MEL is key: Improvement is a continual

cycle that never stops.

3.3 .2 . The Data Dilemma

3.3 .3 . From M&E to MEL

The core set of WaSH MEL indicators is continually being refined and updated to

incorporate new SDG indicators and indicator guidance developed by the JMP and

others, as well as new insights from the field and from current research. These indicators,

together with tools and methods that include validated surveys, water quality testing

methods, quality assurance/quality control (QA/QC) protocols, and data analysis

methods, constitute a robust WaSH MEL framework for monitoring and evaluating

WaSH projects and programs. The toolkit includes the core indicators; community,

household, water point and institutional facility surveys designed to be compatible with

a variety of mobile survey tools currently used in the WaSH sector, including Akvo

3.3 .4 . A New Generation of Data


The learning component of MEL is key: Improvement is a continual cycle that never stops.

FLOW and mWater; a mobile water quality testing field kit; training manuals; in-person

enumerator training modules; QA/QC protocols and robust data analysis (incorporating

both basic summary statistics and more advanced regression analysis) and reporting

methods. The tools and the toolkit are not only meant to enable WaSH implementers

to assess performance and identify problems, but also to leverage advanced analysis in

order to provide valuable insights and knowledge about the root causes of problems and

the greatest opportunities for improving WaSH services at the project, program and

national levels. The MEL framework is designed to be usable by any WaSH

organizations and scalable for subnational and national systems, in

addition to being suitable for CNHF grantees.

3.3 .5 . The Focus on Improvement

Traditional M&E uses data for reports, presentations and grant proposals. A complete

MEL framework requires that data must be turned into improvement via action. The WI

introduced CQI to WaSH through the CNHF-funded MEL project. CQI provides a

systematic, data-driven, improvement-focused means for bringing about change. Instead

of relying on personal opinion, business as usual or peer pressure, CQI is a proven method,

and as the name suggests, a continuous commitment to improving outcomes. The shift in

the WaSH sector mindset needs to take place here, at the point where the data is collected

and analyzed, and the data need to provide fuel for the next opportunity for improvement.

Individuals and organizations can then take the necessary actions to target, improve and

sustain the outcomes.

L E S S O N 4 : C O M P L E X W a S H P R O B L E M SC A N B E S O L V E D U S I N G C Q I M E T H O D SI N W A Y S T H A T W O U L D N E V E R B E P O S S I B L E W I T H T R A D I T I O N A L M O N I T O R I N G

3.4.

Over the last 30 years, the WaSH sector has been characterized by its emphasis on

outputs and hardware without similar achievements in desired service delivery outcomes.

Monitoring and evaluation processes supporting implementation have been inadequate

to effectively measure and improve performance. The sector recognizes that “business as

usual approaches” will not achieve desired outcomes in terms of sustainability and that a

3.4 .1 . Background


new, systems-based approach is needed to improve WaSH program implementation and

performance. CQI methods and tools have proven to be powerful in addressing complex

problems in the manufacturing and health care sectors and have recently been shown to

improve outcomes related to drinking water supply and water quality in rural low-income

country settings.

3.4 .2 . Compounding the Problems with a Project-Oriented Approach

More than a decade ago, Lockwood, Bakalian and Wakeman11 characterized five

main groups of factors that significantly affected the post-project sustainability of

WaSH systems: technical, financial, community and social, institutional and policy,

and environmental. This work cited several prior sources that had identified similar

influences. These factors are widely agreed on in sector-wide sustainability conversations;

however, in practice, a project-vs.-services mindset continues, in which implementers

focus on maximizing outputs (completed projects) at the expense of tracking and

improving outcomes (sustained service delivery). This mindset may well have led to

widespread failure of WaSH systems, as recognized in statistics pointing to water point

or sanitation system access and use, or household water quality. One way to address this

project-based mindset is to change the means and measurements of success—not just for

“projects” but for implementing organizations themselves—thinking past the project and

hardware outputs to a systems mindset that seeks to ensure universal, sustained services by

improving the systems that deliver those services.

3.4 .3 . CQI: A Systems and Process-Oriented Approach to Complex Problem Solving

Applying CQI to the WaSH sector offers an opportunity to adapt to the task of solving

complex problems and delivering cost-effective improvements in service delivery. The

CNHF-funded MEL pilots in Ghana and Burkina Faso and the Water Quality Study

conducted in Ethiopia have applied CQI methods to the complex institutional, technical

and behavioral challenges of improving the uptake and impact of rural water, sanitation

and hygiene programs, something that has not been done previously in the WaSH sector.

CQI employs a systems- and process-centered approach to redesign interventions in real

time by tracking and analyzing performance data in a tight implementation feedback loop

to achieve given overall objectives. In doing so, CQI pays for itself by reducing the risk of

unsustainable outcomes and through more effective targeting of scarce resources. Of equal

importance, CQI empowers implementers and other local stakeholders to analyze and

address problems through MEL rather than surrendering these responsibilities to external

evaluators. Those closest to the problem quickly learn what does and does not work and

can use that knowledge to scale up high-impact improvements in their interventions.

CQI enhances sustainable operation of WaSH systems through short cycles of M&E that inform data-driven adaptations to program design and management.


3 .4 .4 . CQI Enables Precise, Data-Driven Actions to Improve Outcomes

Employing CQI requires robust data, next-level analysis and an improvement plan

that involves all levels of an organization. Without a data-driven system or approach,

individuals and organizations are left to take a best guess at how to solve complex issues.

Trends and percentages are not able to identify root causes and correlations. For example,

in Ghana the data revealed that the majority of household stored drinking water was

in the high-risk category based on WHO Drinking Water Quality Guidelines. The

implementing organization used this information to identify drinking water quality at the

household as a target issue for improvement.

To address the issue of water quality at the household, the WI and World Vision

Ghana started by designing a safe storage container with a stand. The design of the storage

container and the stand evolved as data was gathered from consumers about container

preference and usability. The design as currently distributed prevents hands from being

able to reach into the storage container (hands being a large contributor to contamination),

reaches above waist level to allow water to be poured into the top with limited back-

bending movements and sits above the ground in a stand to prevent goats and children

from knocking the container over. Subsequent monitoring rounds and data indicate

that household water quality was improved with the use of the newly designed storage

container and fewer households were categorized in the high-risk category for drinking

water quality.

Issues surrounding sustainability, water quality and community WaSH committee

management are often complex. Sometimes the problems are simple. If a part on a specific

type of handpump continually breaks, replacing it or repairing the part seems simple

and obvious. However, complex issues where solutions are not clear require a process-

focused approach to identify root causes and relationships to other variables in order to

make an improvement. Thousands of dollars are spent rehabilitating handpumps without

understanding or knowing the root cause of breakdowns and how to reduce water-

point downtime. Several tools used as part of CQI enables implementers to identify root

causes and identify relationships and pathways and impact on outcomes. CQI focuses

on breaking down each process and re-building the process to gain

efficiencies and improve the outputs and outcomes. CQI is a way to put data

into action to improve impact and outcomes.

11 Lockwood, H., et al. 2003.Assessing sustainability in rural water supply: The role of follow-up support to communities. Literature review and desk review of rural water supply and sanitation project documents. Washington, DC: World Bank.


3.4 .5 . CQI is Adaptive

Lessons learned from the CQI pilot project in Ghana were adapted to the Burkina

Faso context. After the WI worked with WaterAid, a CNHF partner working in

Burkina Faso, to conduct WaSH MEL monitoring, household water quality emerged

as an issue here as well. WaterAid chose to focus on addressing water quality. Using the

Ghana storage container as a starting point, WaterAid conducted further testing of the

container in rural communities in Burkina Faso. The storage container was adapted

to local preferences and distributed in households in monitoring areas. Using data and

evidence from other WaSH projects can accelerate learning in a new place. However,

adaptation relies on robust data and analysis. Without data and evidence, individuals and

organizations are not able to target improvement efforts and resources.

3.4 .6 . Effective, Targeted Improvements Require Data and Evidence

CQI is a way to immediately take action on both simple and complex issues and drive

improvements to increase impact and outcomes for everyone and ultimately the end user

and beneficiaries.

Data and evidence are necessary for targeting. If data and evidence do not exist,

lessons are hard to extract and improvements are nearly impossible to measure. Without

improvement, the same story remains: broken handpumps, abandoned water points,

unused toilets and open defecation. Investment in rehabilitation efforts and hardware

will be endless and cyclical without the ability to understand the causes and correlations

behind the WaSH bad-news narrative. Many individuals and organizations are very good

at problem diagnosis, but rarely are they able to provide evidence or solutions for complex

and even simple WaSH problems.

The WaSH MEL experience has demonstrated that implementation science methods

such as CQI can be used to enhance the impact of WaSH programs. Implementation

science (IS) methods, including CQI, have proven highly effective for addressing complex

challenges in manufacturing,12 service industries,13 health care14 and financial services.15

At the core of IS methods is the recognition that all work is done through systems and

all systems can be improved. Methods such as CQI rely on the systematic use of data

to improve processes. Briefly, quality improvement teams develop focused problem/

opportunity statements related to desired areas for improvement, collect and analyze

high-quality data on outputs and process indicators, and then develop improvement

packages based on the results of this analysis. These improvements are implemented on

an iterative basis to achieve adaptive solutions that maximize the quality and efficiency

of programs. CQI is different from conventional M&E because of its systematic


approach and its emphasis on analyzing data to identify and address the root causes of

complex performance issues, rather than simply tracking and quantifying outputs and/or

performance over time.

12 Maani, K., et al. 1994. Empirical analysis of quality improvement in manufacturing. International Journal of Quality & Reliability Management 11(7): 19-37.13 Ramaswamy, R. 1996. Design and Management of Service Processes: Keeping Customers for Life. Addison-Wesley.14 Nicolay, C., et al. 2012. Systematic review of the application of quality improvement methodologies from the manufacturing industry to surgical healthcare. British Journal of Surgery 99(3): 324–35.15 Leseure, M., et al. 2010. The implementation of lean Six Sigma in financial services organizations. Journal of Manufacturing Technology Management 21(4): 512–23.

At the core of implementation science methods isthe recognition that all work is done through systemsand all systems can be improved.

Despite the successful application of IS and CQI methods to multiple sectors, these

approaches had not been previously applied to complex problems in global WaSH. The

WI and CNHF sought to explore the potential of “systems thinking” and IS tools in

general, and CQI methods in particular, to improve outcomes in WaSH systems affected

by complex challenges and problems. The first WaSH CQI pilot projects were conducted

in Northern Ghana, in partnership with World Vision (WV), a Hilton Foundation

grantee. The Water Institute trained a team of WV staff in CQI methods and a team

of enumerators to collect high-quality monitoring data in 230 communities where WV

had previously implemented water programs. The CQI team reviewed baseline data from

these communities and identified water source functionality and household water quality

as two areas they wished to target for improvement.

Using CQI methods, the WV team and WI staff analyzed the data and identified

root causes of poor household water quality, as well as the factors most strongly associated

with nonfunctionality of rural water sources. The CQI team used these results to

identify an improvement package to address poor water quality in stored household

water samples, as well as to increase the functionality of water systems in the target

communities. The package included safe water storage containers and training on their

proper use (to improve stored household water quality) and a combination of refresher

training for WaSH committees (committees responsible for managing water sources in

rural communities) and handpump repair tools (to improve water source functionality).

These improvement packages were implemented in a randomly selected half of the 230

communities in an iterative manner. Briefly, improvements were implemented in three

to six communities, follow-up data were collected, and the package was refined based


on revealed issues with the uptake or performance of the improvements. In this manner,

refined versions of the improvement package were developed and implemented in over

100 communities and more than 500 households. Significant improvements in household

stored water quality were observed in the households that were using a safe water storage

container at follow-up. Improvements in water source functionality were also observed,

although these were significant at the 90% (but not the 95%) confidence level. With

additional time, trends in water source functionality may become more pronounced.

Overall, this project demonstrated that IS methods such as CQI can be

successfully adapted from sectors such as manufacturing and health care

to WaSH, in order to leverage monitoring data to improve program outcomes. Work is

currently underway to adapt the safe water storage improvements from Ghana to Burkina

Faso, and WV and the WI are planning to launch additional WaSH CQI projects across

multiple contexts and multiple dimensions of water, sanitation, and hygiene.

L E S S O N 5 : P R O G R E S S I S U N E Q U A LA C R O S S C O U N T R I E S A N D N O T A L W A Y SW H E R E E X P E C T E D

3.5.

Safe, sufficient sanitation and drinking water are important for human health, well-being

and development. Water and sanitation are recognized as human rights that are important

for addressing inequalities. The principle of progressive realization of human rights

requires that each government take steps “to the maximum of its available resources, with

a view to achieving progressively the full realization of the rights.” The United Nations

General Assembly’s 2010 Resolution on the Human Right to Water and Sanitation16 calls

upon governments “to scale up efforts to provide safe, clean, accessible and affordable

drinking water and sanitation for all.” Water and sanitation are also recognized in human

development policy—prominently in the Millennium Development Goals (MDGs) and

now in the SDGs.

Although coverage of the use of improved water and sanitation continues to increase

globally, there is much work to be done. Global coverage of drinking water and sanitation

is lower when accounting for the quality of services delivered to people. When accounting

for water quality, more than 1.8 billion people drink from a water source containing

fecal contamination.17 An additional 1.2 billion people use sources at an elevated risk of

Inequalities are considered the “unfinished business”of the MDGs and are explicit in the SDGs.


16 United Nations General Assembly. 2010. Resolution on Human Right to Water and Sanitation. New York: United Nations.17 Bain, R., et al. 2014. Global assessment of exposure to faecal contamination through drinking water based on a systematic review. Tropical Medicine & International Health 19.8: 917–27.18 Onda, K., et al. 2012. Global access to safe water: accounting for water quality and the resulting impact on MDG progress. International Journal of Environmental Research and Public Health 9.3: 880–94.19 Baum, R., et al. 2013. Sanitation: a global estimate of sewerage connections without treatment and the resulting impact on MDG progress. Environmental Science & Technology 47.4: 1994–2000.20 Pullan, R. L., et al. 2014. Geographical inequalities in use of improved drinking water supply and sanitation across sub-Saharan Africa: mapping and spatial analysis of cross-sectional survey data. PLoS Med 11.4: e1001626.21 WHO, UNICEF. 2015. Progress on Sanitation and Drinking-Water: 2015 Update and MDG Assessment. Geneva.22 Jordanova, T., et al. 2015. Water, sanitation, and hygiene in schools in low socio-economic regions in Nicaragua: a cross-sectional survey. International Journal of Environmental Research and Public Health 12.6: 6197–217.23 Fehr, A., et al. 2013. Sub-national inequities in philippine water access associated with poverty and water potential. Journal of Water Sanitation and Hygiene for Development 3.4: 63–45.24 United Nations General Assembly. 2015. Transforming our World: The 2030 Agenda for Sustainable Development. A/RES/70/1, 21 October.

contamination.18 An estimated 4.1 billion people lack sanitation that is treated before it is

discharged into the environment.19

Further, global and national-level coverage estimates mask subnational inequalities.20

Analysis by the WHO/UNICEF Joint Monitoring Programme shows substantial

inequalities exist in many countries and many disadvantaged populations have been left

behind.21 Studies confirm that inequalities exist in other nonhousehold settings such as

schools and health-care facilities.22,23 Inequalities are considered the “unfinished business”

of the MDGs and are explicit in the SDGs. The MDGs did not include specific language

to prioritize inequalities.

Sustainable Development Goal 6 calls for the “availability and sustainable

management of water and sanitation for all” by 2030.24 The goals and targets for water

and sanitation seek to correct for the unfinished business of the MDGs. Many external

support actors, such as multilateral agencies, NGOs and foundations, are committed to

reducing inequalities and providing services to the “poorest of the poor.” For example, the

language in the CNHF Strategy for Sustainable Safe Water Access mentions “reaching

the vulnerable and ultrapoor.”

Achieving universal coverage and reaching the vulnerable and ultrapoor will require

innovation, thoughtful strategy, substantial improvements in monitoring and investment

targeting. To make improvements, it is important to understand the characteristics of

unserved populations in order to identify the best ways to provide them services. People

without WaSH are an increasingly small, disparate and diverse population, often living

on the margins of society in low- and middle-income countries. In the SDG era

(Figure 3), people most in need of WaSH services will be located in small

geographic pockets in hard-to-reach locations. These people may be located


where “sustainable interventions” and “market-based approaches” do not apply. As

indicated by the WaSH Performance Index,25 progress is unequal across countries and not

in the countries where people think it is. Specifically, some countries that currently have

low rates of coverage may be achieving faster-than-average progress with respect to the

rate of change in coverage levels, while some countries with high current coverage may

be achieving slower-than-average progress with respect to their peers. Thus, for example,

several countries in sub-Saharan Africa and South Asia, such as Niger and Pakistan,

appear to be making rapid progress relative to their peers at comparable current levels of

coverage (Figure 4). The challenge of the SDG era will be to determine

where these populations are and how governments and external support

agencies can use data to target them and provide them with services

effectively and efficiently.

National-level data should be further disaggregated to measure inequalities in

coverage among groups such as minorities and populations in rural areas. However,

organizations within countries (e.g., NGOs, external support agencies) must conduct

more targeted monitoring of people and populations at a subnational level. More

monitoring of nonhousehold settings is needed to gain further insight into the sufficiency

of WaSH in these settings. If proposed SDG targets with respect to reducing or

eliminating inequalities in coverage of water and sanitation are to be achieved, rigorous

monitoring, evaluation and learning will be necessary to identify characteristics of these

54% accessto water

87% accessto water

1990s Today

Easiest to reach

Easiest to reachModerately hard to reach

Moderately hard to reach

Hardest to reachHardest to reach

Uns

erve

d

Uns

erve

d

Serv

ed

Serv

edFigure 3. Progress on drinking water access in Ghana and hypothesized ease of reaching the remaining unserved.


Figure 4. WaSH Performance Index values by country.

unserved populations so that existing and new projects can ensure services to the most

disadvantaged are provided.

Two stages of unserved targeting are needed. The first should focus on population-

level spatial characteristics. Stakeholders should use existing data to identify and prioritize

countries with low access and water scarcity. The WaSH Performance Index can be used

to identify low performing countries. Subnational data should be used where possible20

to identify areas that lack safe water and sanitation. Donors should consider funding

external support in these particular areas. The second stage should focus on household

and individual characteristics. Core wealth index indicators should be included to ensure

the ultrapoor and unserved are reached with services. Monitoring strategies also should

include indicators for other disadvantaged populations such as disabled and elderly people.

Rigorous monitoring, evaluation and learning will be necessaryto identify characteristics of these unserved populations.

It is also important that WaSH interventions reach nonhousehold settings such as

schools and health-care facilities to ensure universal coverage of services is achieved and to

ensure that certain populations are not marginalized. Following these approaches will help

ensure that the CHNF and partners are serving the unserved and helping to fulfill the

Human Right to Water and Sanitation. •

25 Luh, J., et al. 2016. Assessing progress towards public health, human rights and international development goals using frontier analysis. PloS One 11.1: e0147663.


PRINCIPLES OF MONITORINGFOR IMPROVEMENTP R I N C I P L E S O F G E N E R A T I N GF I T - F O R - P U R P O S E D A T A

4.1.

4.

Asking the Right Questions: Outputs, Outcomes and Process Indicators

Sampling and Sample Size Calculations

Measuring X and Y Variables

Methods of Data Collection (Measurement, Direct Observation and Direct

Response)

Crafting Robust Survey Questions and Operational Definitions: Avoiding Bias,

Jargon, Constructs and Other Pitfalls

Data Collection: Best Practices and Pitfalls

Selecting and Using Information and Communication Technologies

Hands-On Training

Quality Assurance / Quality Control and Reviewing Data

Regular Refresher Training

Proper Data Analysis

4 .1 .1 . Asking the Right Questions: Outputs, Outcomes and Process Indicators

In addition to being used to report the status, trends and levels of water and sanitation

services, monitoring data can be used to answer policy- and program-relevant research

questions.26 Relevant research questions can be developed in part by consulting findings

of prior monitoring efforts, prior evidence and theory of change models. Questions can

be tailored to the specific context, program or country and might explore, for example,

how water and sanitation interventions vary by setting, what processes are involved in

improving water and sanitation outcomes and what are the most important determinants

associated with higher levels of a given outcome.27 “Systems thinking” is important

to develop appropriate theoretical models for analysis, as environmental problems and

interventions are complex—with social, managerial, cultural, environmental and policy

determinants.28

Well-designed survey questions are also needed to answer policy- and program-

relevant research questions. Survey questions need to be scientifically robust, useful,

relevant, cost-effective and need to reduce bias where possible.29 While standard indicator

and survey question evaluation criteria are not available specifically for WaSH, use of the

SMART10 criteria may help ensure survey data is more reliable.30

4.1 .2 . Sampling and Sample Size Calculations

Many WaSH programs conduct M&E activities without considering the implications

of sampling and sample size. This oversight can threaten the validity and efficiency of

M&E activities in several ways. Generally, there are only two ways to obtain reliable

and representative information about a population of individuals, communities or

facilities: survey the entire population (i.e., go to every community, water source, latrine

and household) or survey a representative sample. High-quality monitoring of WaSH

programs generally requires multiple components such as in-person site visits, structured

observations and surveys at the community, household, and water and sanitation facility

levels, and water quality sampling and testing. Conducting these activities for an entire

population of communities, households or facilities would be cost-prohibitive for most

WaSH programs. If such an exhaustive census were proposed, it could only be conducted

a few times each decade, at most, to avoid consuming the entire budget of a typical WaSH

program, leading to a lack of timely data.

25C O M P E N D I U M O F B E S T P R A C T I C E S A N D L E S S O N S L E A R N E D 4 . P R I N C I P L E S O F M O N I T O R I N G F O R I M P R O V E M E N T

By contrast, a representative sampling approach can provide M&E data representative

of an entire program area, even if a far smaller number of communities, households

and facilities are surveyed. To do this, the units (i.e., communities, facilities, and/

or households) to be sampled must be selected at random from the entire population.

Random sampling ensures that each unit has an equal chance to be selected. As a result,

the data obtained are representative of the entire population.

Sampling can be done in several ways. In the stratified random sampling approaches

typically used, program areas are divided into communities or enumeration areas and

several of these larger units are selected at random. Within each of the larger units,

individual facilities, households, etc., may also be selected at random. In this way, a large

population can be sampled without the need to visit a household in district A, then travel

50 km to district B to sample the next household, thereby increasing resource efficiency

while maintaining the validity of the sampling approach. When conducting any type of

random sampling, it is important to obtain a sample size that is large enough to achieve

the objectives of the monitoring activity, but not so large as to be prohibitively expensive.

In order to do this, sample size calculations must be performed, in order to obtain the

smallest sample needed to obtain M&E data fit-for-purpose with an adequate degree of

precision. Examples of methods for sampling and sample-size calculation for WaSH M&E

activities are provided on the WaSH MEL Virtual Learning Center’s modules on these

subjects: Sampling (http://www.washmel.org/module-4-relaunch/) and Sample Size

Calculation (http://www.washmel.org/module-6-relaunch/).

4.1 .3 . Measuring X and Y Variables

Another challenge many WaSH implementers face is the selection of the right variables

and indicators to monitor. In many cases, implementers will monitor only outputs (for

example, number of boreholes drilled, number of hygiene promotion workshops held,

number of community-led total sanitation triggering session participants). Monitoring

only outputs is problematic because it does not provide information on whether those

outputs achieved their intended objectives (improving the quantity and quality of water

available, reducing open defecation, etc.), and thus impact cannot be measured. In other

cases, programs measure only outcomes (water quantity per person per day, household

water quality, proportion of households practicing open defecation, etc.). Robustly

26 Zachariah, R., et al. 2009. Operational research in low-income countries: What, why, and how? The Lancet Infectious Diseases 9(11): 711–17. 27 Hales, S., et al. 2016. Reporting guidelines for implementation and operational research. Bulletin of the World Health Organization 94.1: 58–64.28 Pidd, M. 2009. Tools for Thinking; Modelling in Management Science. John Wiley and Sons Ltd.29 Choi, B. C., and Pak, A. W. 2005. A catalog of biases in questionnaires. Preventing Chronic Disease 2(1): A13.30 Schwemlein, S., et al. 2016. Indicators for monitoring water, sanitation and hygiene: a systematic review of indicator selection methods. International Journal of Environmental Research and Public Health 13.3: 333.


monitoring outcomes is valuable in that it enables programs to measure their impact.

However, tracking outcomes without measuring the intermediate process variables and

mediating factors associated with those outcomes can be problematic because it does

not provide information as to how to improve outcomes. Specifically, if an organization

determines that the functionality of water sources it has implemented is low, it will be

unable to address the problem without also collecting information on the determinants

of low water source functionality. Is functionality low everywhere or only in certain

regions? Are some source types more likely to be functional then others? Is functionality

associated with the presence or quality of management entities, or the availability of tools

and spare parts? These key determinants can provide important clues as to what actions

an implementer can take to improve a given outcome. In this respect, it is useful to think

of M&E data collection as a process of collecting both “Y” variables (outcomes) and

“X” variables (potential determinants of those outcomes). If implementers are able to

effectively collect both those Y variables most closely related to their program objectives

and the X variables that determine those outcomes, they are much more likely to obtain

data which will enable them to improve those key outcomes over time.

4.1 .4 . Methods of Data Collection

(Measurement, Direct Observation and Direct Response)

Several methods can be used to collect WaSH data, each with its own strengths and

weaknesses. Data can be collected at the household, community and facility levels and can

be obtained by direct observation or direct response.

The level of data collection selected for monitoring—either at the household,

community or water system level—will reveal different insights about systems and

water availability. For example, most nationally representative household surveys ask

respondents to identify the main source of drinking water used by the household, which

measures source use. If a nearby handpump is broken for an extended amount of time

and people use a protected spring a kilometer away, they will report that they use a

protected spring as their main source and indicate that the source is functional, thus

introducing a “survivorship bias” with respect to source type and functionality. While this

answer provides some information about availability, it does not provide representative

information about the functionality of all systems available to the user. Alternatively,

household surveys that ask about the reliability of the main water source used provide a

more accurate representation of the actual availability of water for the populations being

studied but still may be subject to survivorship bias.

Water system surveys provide information about particular water systems, which can

be valuable to actors who need to make decisions about how to invest in infrastructure


and track the status of assets. A limitation of such surveys is that they do not provide

reliable estimates of the population using water systems.31

31 Fisher, M. 2015. Core WaSH MEL Indicators: Monitoring for Continuous Program Quality Improvement. The Water Institute at UNC, Chapel Hill, NC.

Direct observation typically provides more reliable data, but direct response costs much less and is much less time consuming.

Direct observation typically provides more reliable data. Trained enumerators can

consistently evaluate water source types, sanitation facility types and other technical

details about infrastructure and services. Householders cannot always provide reliable

responses to questions about which they do not have appropriate expertise or knowledge—

such as the cause of a water system breakdown. Direct response, however, costs much

less and is much less time consuming for data collectors. Direct response also can reveal

information about people’s behaviors and habits that are important for informing service

delivery, revealing, for example, which water source a respondent used most recently.

4.1 .5 . Crafting Robust Survey Questions and Operational Definitions:

Avoiding Bias, Jargon, Constructs and Other Pitfalls

Crafting Survey Questions

Once indicators and variables have been established, suitable survey questions can be

adapted and developed. Survey questions should support the measurement of variables,

which should support the tracking of indicators to obtain data that meet survey objectives.

In addition to survey questions that directly support variable measurement, surveys

usually also include questions that capture metadata, stratifying variables, shadow

indicators and quality assurance / quality control (QA/QC) checks. Metadata (e.g., the

date the survey was conducted) and QA/QC questions (e.g., asking for a water point ID

a second time in order to ensure it is entered properly) should be included in all surveys.

Questions that capture stratifying variables should be included if there is a need to stratify

data across different regions or countries.

Standard questions that have already been validated should be used whenever

possible. Using validated standard questions increases the validity and comparability of

data. The following are recommended resources for standard WaSH survey questions:

• WHO/UNICEF JMP: Core Questions on Drinking-water and Sanitation for

Household Surveys

• USAID Demographic and Health Surveys

• UNICEF Multiple Indicator Cluster Surveys

• World Bank Living Standards Measurement Study


Surveys that already have been conducted either regionally or nationally may include

questions appropriate to the local cultural context. Examples of these surveys include:

• National censuses

• Demographic and health surveys

• Multiple indicator cluster surveys

• Living standards measurement studies

Although standard survey questions are a great resource and should be used

whenever possible, it will likely be necessary to create new survey questions as well. New

survey questions should be created with these guidelines in mind:

1. Use the simplest language possible.Avoid using language that is more complex than necessary. For example, asking “Is

there a particular spot in this house where people go to wash their hands?” is preferable

to “Is there a fixed location in the dwelling where hygiene activities are performed?”

2. Phrase similar ideas consistently between questions.Consistent phrasing helps make the survey clearer for the respondent. Asking one

question about the main source of drinking water in the dry season and then asking the

next season about the main source of drinking water in the wet season is much clearer

than first asking “What is the main source of drinking-water for members of your

household in the dry season?” followed by “Where do you most frequently go to fetch

water during the months when it rains a lot?”

3. Ask questions respondents will likely be able to answerEstimating liters per capita per day may be a study objective, but most people will not

know the answer to the question: “On average, how many liters of water does your

household use per person per day?” This information may be better collected by asking

a series of questions in order to determine the number of household members, the

number of trips each household member made to fetch water the previous day and and

the size of the container(s) used to fetch water.

4. Ensure logical flow of questions.Group questions about similar topics together. Organizing the survey into sections can

help facilitate this grouping.

5. Put sensitive questions later in the survey when participants’ comfort level will be greater.A question such as “The last time [name of youngest child] passed stools, what was

done to dispose of the stools?” should not be asked at the beginning of the survey.


6. Avoid jargon in survey questionsSurvey questions are not an appropriate platform for WaSH jargon. Common everyday

language should be used to avoid confusion, and great care should be taken when

translating surveys to ensure that questions are achieving their desired meaning in the

local language and context.

7. Avoid constructs in survey questions.Constructs are ideas or theories considered to be subjective and not necessarily

universally familiar or based on observable reality. Examples of constructs in the

WaSH field include “productive uses of water,” or “opportunity cost,” etc. Asking

questions about these constructs is problematic because the concepts may be unfamiliar

to respondents, and therefore respondents may not be able to give meaningful answers

in relation to these constructs. Constructs may also contain implicit assumptions about

WaSH practices or human behavior, and these can likewise prove problematic if

unfounded. For these reasons, survey questions should rely on concepts that are directly

linked to observable reality wherever possible. Instead of asking about “productive uses

of water,” it may be better to ask about “water for gardens, crops or businesses,” etc.

Avoiding Bias in Survey Question Design

In all aspects of M&E, it is desirable to minimize bias and error. Error is defined as any

monitoring result that differs from the actual (“true”) result. Errors can be systematic or

random. Random errors are unpredictable errors that are equally likely to occur in either

direction. Systematic errors are more likely to occur in one direction than the other and

are often due to inadequacies in the measurement system. Bias is anything that introduces

systematic error in survey data. It is important to minimize when developing survey

questions. Several types of bias are relevant to survey development. Three of the most

common are recall bias, socially desirable response bias and acquiescence bias:

Recall bias occurs when respondents misreport events that happened a long

time ago. The best way to avoid recall bias is to ask questions about events that have

happened somewhat recently. For example, asking detailed questions about events

that happened in the past five years would likely result in poor quality data due to

recall bias. Asking about whether a water point has broken down in the past year or

even the past two weeks would likely result in much more accurate responses.


Socially desirable response bias occurs when respondents give false information to

avoid embarrassment or to impress the enumerator. Questions about sensitive topics such as open

defecation practices are more likely to provoke socially desirable response bias. Careful wording

of questions can help to eliminate this bias. Asking about open defecation practices with a question

that normalizes several different types of sanitation behavior may result in less bias. For example,

the question “Where do adults in this household defecate?” may produce more bias than this type of

wording: “Some people prefer to defecate in the bush or the open, some prefer to defecate in a latrine

or toilet and some prefer other places. What are the places that adults in this household defecate?”

It’s also important to consider the enumerator’s organizational affiliation with when attempting to

prevent socially desirable response bias. For example, if an NGO gives a safe water storage container

to a household, and then the same organization sends an enumerator to that household to ask “Is

your household using the safe water storage container?” then the respondent may be more likely to

say “yes” regardless of whether the household is using the container. In this case, socially desirable

response bias could be avoided by including a direct observation question in the survey so that the

enumerator observes whether the container is being used instead of (or in addition to) asking the

respondent.

Acquiescence bias can occur when a respondent is asked a leading question. Leading questions

tend to guide a respondent toward a given answer, causing the respondent to agree, even if this is not

accurate. “Do you always wash your hands before eating?” is a leading question. A better way to ask

about hand washing practices is “When do you usually wash your hands?” Acquiescence bias can be

avoided by developing balanced questions, rather than asking leading assertions for respondents to

disagree with or affirm.

The Importance of Operational Definitions

Operational definitions are clear and detailed definitions that relate to the variables being

measured. For example, many WaSH surveys may ask questions related to frequency of

diarrhea in children under the age of five. Each respondent may have a slightly different

definition of “diarrhea.” Including an operational definition in the survey question such as

“three or more loose or liquid stools within 24 hours” can help ensure consistency across

respondents. Operational definitions can be included in survey questions as “hints” or

“tips” listed next to survey questions or can be listed separately in an enumerator manual.

4.1 .6 . Data Collection: Best Practices and Pitfalls

Collecting M&E data fit-for-purpose in the field cannot occur unless adequate sampling,

survey tools and data collection methods are all in place. Even when these prerequisites

are present, however, M&E activities can fail catastrophically if data collection is not well

planned and managed in the field. In many cases, M&E efforts fail to produce

usable data, or to achieve their intended outcomes, due to one of the

following pitfalls:


• Insufficient training of field staff

• Failure to adequately pretest, pilot and validate surveys and instruments

• Inadequate language skills of enumerators and/or inadequate translation of survey tools

• Inadequate collection and/or transport of water quality samples

• Inadequate supervision of field staff

• Lack of QA/QC procedures

Best Practices to Avoid Pitfalls

1. Ensure adequate length of training.Robust M&E activities capable of producing data fit-for-purpose can be quite involved,

requiring the ability to identify specified households, communities and facilities; conduct

structured surveys in a reproducible manner, employing a variety of precise operational

definitions and observation protocols; collect and process water quality samples; and accurately

complete and submit survey forms. Training new or existing staff to accurately perform all of

these tasks can be an intensive process (Figure 5). Enumerators can learn to collect data in

just one or two days but often require one or two weeks to learn to collect accurate data. After

the first day or two of training, many field staff are able to complete every question of a survey,

even if the responses are frequently inaccurate and do not reflect a good grasp of the applicable

LYNDA OLiViA reYFigure 5. A continuous quality improvement training workshop in Ghana.


protocols and operational definitions. A training period of one to two weeks or more

ensures that staff can effectively collect high-quality data while adhering to protocols,

correctly applying operational definitions and avoiding contaminating water quality

samples. If WaSH implementers are not carefully observing enumerators and the data

collected and are not performing systematic quality checks, it may be easy to conclude

that a few days’ training is sufficient. But given that the cost of training enumerators

may represent only 5-10% of the cost of all M&E activities for a given year and that

inadequate training of field staff can destroy 100% of the value of any data collected,

adequate training or even overtraining may be a worthwhile investment for many

WaSH implementers.

2. Pretest and pilot all survey instruments.While some WaSH implementers may use existing, validated survey instruments,

many prefer to create their own surveys and M&E tools or to contract a third party

to produce these tools. However, even when highly experienced WaSH profession-

als create such instruments, they inevitably develop some questions and methods that

simply do not work in the field. Pretesting and piloting these instruments before they

are deployed provides an opportunity to identify and correct such deficiencies. Failing

to do so virtually guarantees that some elements of the survey tools will not work as

intended. Frequently, the WaSH implementer may not know which elements these are

and thus cannot have confidence in any data obtained.

3. Confirm written translation and the language ability of field staff.In many contexts where WaSH M&E work is conducted, multiple local languages

may be spoken. Typically, M&E surveys and tools will be developed in one language,

such as English or French, and then translated into one or more local languages when

the tools are used in the field. This can either be done in written form, where with

tools are professionally translated in advance, or it can be done via translation-on-the-

fly, in which enumerators translate surveys to the local language while conducting

the survey. The former approach is preferable when the target language has a readily

understood written form; the latter may be necessary when such a written form does

not exist or is not widely known. In all cases, it is essential to confirm that any written

translations are correct, and that enumerators are truly able to speak the necessary

languages fluently. In many cases, enumerators may exaggerate their ability to speak

one or more of the necessary local languages. In some cases, they may count on their

ability to communicate in a different local language, which may serve as a lingua

franca in that local context. However, performing a survey in a language that is not

well understood by the respondent can compromise the validity of any data obtained.


Thus, it is essential to confirm the accuracy of all written translations (often by back-

translation and/or third-party review) and to confirm the language abilities of all field

staff (often by an oral test with a native speaker of each required language).

4. Ensure proper technique in sampling procedures.Many WaSH implementers collect and analyze water quality samples (Figures 6a,

6b and 7). Often this is done through an external laboratory or consultant. In many

settings, it is common practice to use improvised containers for water quality sampling,

such as empty soft drink bottles that have been rinsed out. Similarly, in many cases

such samples are collected without particular attention to sterile technique (a method

of avoiding contamination during sample collection), cold chain and holding time

requirements (microbial samples) or sample preservation requirements (chemical

samples). Such oversights can compromise the validity of any water quality data

obtained. Thus, when conducting water quality testing, whether directly or through a

third party, it is essential to ensure that appropriate sampling procedures are observed.

5. Supervise field staff regularly.WaSH implementers typically attract highly professional and motivated staff.

However, without adequate supervision, field enumerators and/or survey consultants

LYNDA OLiViA reY

Figures 6a, 6b. Enumerators in Ghana collect samples using a compartment bag test (left) and test samples in the field (above).


Figure 7. Enumerators in

Ghana test water source flow rate

using a collapsible bucket from the

field test kit.

can be prone to unintentional and intentional errors that can compromise data quality.

Common issues can include accidentally visiting the wrong communities, cutting

corners on water quality sample collection procedures to save time, neglecting proper

informed consent or failing to survey households or water sources that are far away due

to a reluctance to walk long distances. In extreme cases, enumerators or contractors

may even disregard random sampling designs (instead visiting communities or

households that are most convenient), or falsify data, fabricating surveys altogether.

While these problems may be rare, without adequate supervision they may not be

detected until it is too late, leading to wasted resources and loss of valuable M&E data.

Furthermore, if mistakes occur in only a small percentage of surveys or enumerators

but it is not known which are valid, the good data also will be compromised. To avoid

such issues, it is important to have regular supervision of field staff.

6. Implement consistent QA/QC procedures.In addition to supervision, structured QA/QC checks can quickly detect errors so that

they can be addressed and corrected. This is discussed in greater detail in section 4.1.8.

LYNDA OLiViA reY


4.1 .6 . Selecting and Using Information and Communication Technologies

Information and communication technologies (ICTs) such as mobile survey tools (MSTs)

can dramatically improve the quality and efficiency of M&E data collection. Selecting

a suitable MST can enhance not only the field-level data collection process but also data

management and analysis. Some key MST functions include allowing users to gather and

transmit field data in real time, standardizing data storage and management, automating

routine analyses and visualizing data.32 A number of different MSTs are available on the

market, and this section offers some guidance on selecting and using an appropriate one.

When selecting an MST, different organizations may prioritize different factors.

Price may be the most important consideration for an organization with a small budget,

while usability may be the most important for an organization with little to no technology

experience. It could also be useful in the long term to consider selecting an MST that is

sufficiently well-established and will be available in years to come.

In a recent review,32 WI staff developed an

evaluation framework for testing MSTs. In order

to validate this framework, WI staff conducted

an illustrative evaluation of seven different

MSTs across five characteristics: relative cost,

performance, ease of use, learning curve and speed.

The seven reviewed MSTs were Akvo FLOW

(Akvo Foundation, Amsterdam), Open Data Kit

(ODK, open source), Magpi (Magpi, Washington,

DC), iFormbuilder (Zerion Software, Herndon, VA); Fulcrum (Spatial Networks, Inc.,

St. Petersburg, FL, USA), mWater (mWater, New York) and Survey CTO (Dobility,

Inc., Cambridge, MA, USA). Additional MSTs are available on the market, but the

illustrative evaluation was limited to these seven particular MSTs to limit the scope of the

validation exercise. The illustrative evaluation was performed in 2014-2015, and several

of the MSTs evaluated have been updated since. Results should be interpreted with this in

mind.

For each MST, both the application and the online dashboard were evaluated. Based

on five factors (cost, ease of use, performance, learning curve and speed), Fulcrum and

mWater received the best scores (i.e., lowest overall scores) at the time of the evaluation,

with Fulcrum performing best overall and mWater performing second-best overall

and best among the free MSTs evaluated (see Table 2 for more details). Fulcrum and

mWater were also rated the easiest to use, requiring the least amount of time to learn,

32 Fisher, M. B., et al. 2016. Evaluating mobile survey Tools (MSTs) for field-level monitoring and data collection: development of a novel evaluation framework, and application to MSTs for rural water and sanitation monitoring. International Journal of Environmental Research and Public Health 13.9: 840.

Overall rankingof seven tested MSTs

1. Fulcrum

2. mWater

3. iFormbuilder

4. Magpi, Akvo FLOW (a tie)

6. Survey CTO

7. ODK

Table 2. Equal-Weighted Composite Score and Overall Rank31

MSTOverallComposite Score*

OverallRank

Akvo Flow 20 4

ODK 22 7

Magpi 20 4

iFormbuilder 18 3

Fulcrum 9 1

mWater 14 2

Survey CTO 21 6*Lowest composite score is best


Table 3. General Characteristics of Tested Mobile Survey Technologies (as of 2015)31

Parameter Akvo FLOW ODK Magpi iFormbuilder Fulcrum mWater Survey CTO

Mobile platform compatibility

Android Android Android, iOS, Nokia

iOS Android, iOS Android, iOS Android

Mobile platforms tested

Android Android Android, iOS iOS Android, iOS Android, iOS Android

Mobile devices tested

Samsung Galaxy S II Skyrocket, Samsung Galaxy Stellar, Huawei Impulse

Samsung Galaxy S II Skyrocket, Huawei Impulse

Samsung Galaxy S II Skyrocket, Samsung Galaxy Stellar, iPhone 5

iPhone 4s, iPhone 5,iPhone 5s

Samsung Galaxy S II Skyrocket, Samsung Galaxy Stellar, iPhone 5, iPhone5s

Samsung Galaxy S II Skyrocket,Motorola Droid Mini, iPhone5

Samsung Galaxy S II Skyrocket, Motorola Droid Mini

Web browsers used to test dashboard

Chrome, Firefox, Inter-net Explorer

Chrome, Firefox, Safari






OS used to test dashboard

Windows 7, Mac OS X

Windows 7, Mac OS X

Windows 7, Mac OS X

Windows 7, Mac OS X

Windows 7, Mac OS X

Windows 7, Mac OS X

Windows 7, Mac OS X

Does app function offline?

Yes Yes Yes Yes Yes Yes Yes

Does dashboard function offline?

No No No No No No No

Cost (USD) Variable, approx. $10,000 for one instance with set up and training

Free $5,000/year for 10,000 uploads, $10,000/year for 20,000 uploads

Smart Enterprise $100, Exploring $2,000, Growing $5,000, Emerging $10,000*

Variable, $29/mo for 1 user, $99/mo for 5 users, $399/mo for 25 users, $749/mo for 50 users

Free Variable, $99/mo for 10 users, $399/mo for unlimited users

Est. cost for 10 users and 10,000 uploads in a year

$10,000 0 $5,000 $5,000 $4,788 0 $1,188

Cost rank(1=Lowest)

4 1 3 2 2 1 2

while Survey CTO and ODK took the most time to master. A standard survey took the

least amount of time to create and complete using Fulcrum and iFormbuilder and the

longest with Survey CTO and Magpi. Information regarding additional characteristics

to consider when choosing an MST, such as mobile platform compatibility, is included in

Table 3.

*These represent different usage levels.


4.1 .7 . Hands-on Training

Hands-on training is an important aspect of ensuring high-quality data collection.

While training manuals and written protocols are helpful aids in training enumerators to

collect M&E data, these are not suitable substitutes for hands-on practice and in-person

training and observation (Figure 8). This supervised deliberate practice provides an

opportunity for enumerators to cement practical skills such as flow rate measurement,

water quality sampling and/or analysis, and sanitary inspections of water and/or sanitation

facilities. In addition, it provides an opportunity for enumerators to cement and calibrate

operational definitions of functionality, water source types and latrine types, and other

critical distinctions that tend to improve with hands-on practice. It is recommended that

such hands-on training be supervised by a staff member who is experienced in field data

collection and familiar with the surveys and tools being used. This can greatly improve

the quality of data collected and is often overlooked by WaSH implementers who may not

have firsthand experience of the impact of supervised hands-on training on M&E data

quality.

LYNDA OLiViA reYFigure 8. Testing samples in the field for arsenic in Ghana.


4.1 .8 . QA/QC and Reviewing Data

Systems for reviewing and verifying the quality and accuracy of field data collected

during M&E activities include the collection of QA/QC samples for water quality

analysis, including field blank and duplicate samples, as well as the use of photo

verification to enable supervisors to confirm data such as water source type and sanitation

facility type, etc. This can be done by supervisors using a structured QA/QC protocol

that systematizes the data review process. In some cases, simple checks can be performed

for all records (e.g., checking that IDs and community names are recorded correctly). In

other cases, QA/QC checks are more painstaking and should be performed for a random

subset of records (e.g., collecting field blanks and duplicates for 5-10% of water samples

or inspecting verification photos for a random 5-10% of water sources to verify source

type). These QA/QC checks enable WaSH implementers to rapidly catch and address

data quality problems related to issues that were not clearly explained in training or not

fully understood by some or all enumerators. Furthermore, they enable implementers to

detect quality control issues such as accidental contamination of water quality samples

during collection and storage. They also provide a safeguard against deliberate falsification

of M&E data, as QA/QC checks can be designed to detect most common types of

falsification, making such behavior prohibitively difficult for those very rare enumerators

or contractors who might consider it. Finally, data quality tends to improve when

enumerators and contractors know the data will be reviewed.

4.1 .9 . Regular Refresher Training

It is helpful to retrain data collectors on a regular basis to ensure that WaSH organizations

continue to collect high quality data and enumerators’ skill levels do not deteriorate. Up

to one week of refresher training each year is recommended. Retraining is most critical for

organizations that have experienced staff turnover, but even data collectors who have gone

through formal training in the past will benefit from refresher training. Although good

QA/QC protocols and time spent reviewing data go a long way in ensuring data quality,

refresher training helps to ensure that all data collectors are on the same page and keeps

their skill levels high and consistent.

Refresher training is an opportunity for enumerators to ask questions about parts of

the data collection process or specific survey questions that may be unclear. It is a good

opportunity to revisit key operational definitions to ensure consistency. Refresher training

should consist of a review of all survey questions and procedures for sanitary inspections,

water quality testing and other hands-on processes, with plenty of opportunities for data

collectors to ask questions or raise concerns. The refresher training should include a field


component (Figure 9) so that data collectors have an opportunity to practice under the

supervision of an instructor. Time should be set aside to debrief after each field practice

day so that data collectors have an opportunity to ask questions and the instructor can

address any issues that were observed.

LYNDA OLiViA reY

Figure 9. Enumerators with a field kit in Ghana.

4.1 .10. Proper Data Analysis

Achieving the SDGs will require more and better evidence from project, program,

subnational and national monitoring data, but at the moment these data are often

evaluated in a limited capacity. Making better use of monitoring data will require

improvements to data collection and analysis such as developing relevant and appropriate

survey questions, using standard definitions, ensuring data are reliable, analyzing data


using appropriate methods to find important relationships and using the results to generate

policy, programming, and practice recommendations. More efficient and effective service

delivery monitoring increases the potential for insight from studies using these data.

Clear, consistent language and standard definitions are needed to enable meaningful

comparison of data.33 In reports and publications, clearly reported methods that follow

appropriate reporting standards such as the STROBE statement or WHO reporting

guidelines for implementation and operational research allow others to replicate the data

collection or study in different settings or contexts.27, 34 In addition to standard definitions,

good practice reporting includes clear descriptions of the data, which allows other people

to use the data for future analysis and meta-analysis.

Understanding the determinants of improved service delivery is a challenge, as

service delivery is nested within complex social, political, technical and environmental

systems.35 Data analysis in the SDG era will require the use of different analysis

techniques—used in novel ways—to solve problems and develop improvement solutions.36

Harnessing the power of “big data” generated through remote sensing (e.g., satellite

imagery), systematic meta-analysis and other techniques will provide cost-effective ways

to incorporate additional determinants into service delivery analyses.37

C H E C K L I S T F O R W a S H M E LI M P L E M E N T A T I O N

4.2.

The checklist below and Figure 10 provide an overview of the steps required to

implement high-quality M&E activities and to use the resulting data to drive continuous

quality improvement.

Preplanning1. Decide to implement a WaSH MEL or WaSH MEL CQI program.

2. Define MEL project scope and objectives (e.g., “track impact of our WaSH programs with respect to national objectives and the SDGs,” or “improve sanitation uptake in the program area,” etc.).

3. Identify internal champions who will help drive the program and external partners who can facilitate the development of a MEL framework.

4. Allocate/secure the necessary funding and personnel (as needed).

Checklist for WaSH MEL ImplementationCategories coveredby this checklist

• Preplanning

• MEL planning

• Procurement, training and piloting

• Data collection

• Data analysis and reporting

• CQI

• Ensuring ongoing monitoring quality

(cont.)


33 Remme, J. H. F., et al. 2010. Defining research to improve health systems. PLoS Med 7.11: e1001000. 34 Von Elm, E., et al. 2014. The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. International Journal of Surgery 12.12: 1495–9.35 Amjad, U. Q., et al. 2015. Rethinking sustainability, scaling up and enabling environment: a framework for their implementation in drinking water supply. Water 7.4: 1497–1514.36 Griggs, D., et al. 2013. Policy: sustainable development goals for people and planet. Nature 495.7441: 305–7.37 Lu, Y., et al. 2015. Policy: five priorities for the UN Sustainable Development Goals-comment. Nature 520.7548: 432–3.

Monitoring Planning Phase

Develop a MEL Framework Identify the core indicators and variables Develop survey tools Develop a sampling plan Develop a data analysis plan Develop QA/QC protocol Identify an appropriate ICT platform Hire data enumerators as needed Order data collection supplies and equipment

Pre-Planning Phase

Decide, as an organization, to implement a WaSH MEL CQI program Identify key players within the organization who will help drive the program Secure necessary funding and personnel as needed

CQI Planning Phase

Establish a CQI team within your organization Hold a CQI training Encourage the CQI group to identify WaSH and/or or-

ganizational issues for CQI

CQI Cycle

Training and Piloting Phase

Train data collectors Pilot survey tools Revise survey tools as needed

MEL CQI Checklist

Data Collection Phase

Commence data collection QC data regularly Share feedback with enumerators

based on QC Hold refresher training as needed

Intervention Cycle

Pilot identified intervention

Undergo uptake monitoring

Adjust intervention as needed

Figure 9. An overview of the WaSH MEL CQI process.


MEL Planning1. Develop a MEL framework.

a. Develop core indicators and associated variables. These should relate to the project scope and objectives described above

b. Develop survey tools to track these indicators and capture the related variables (all survey tools should be validated before use). Develop suitable informed consent forms and procedures if identifiable personal information will be collected during monitoring.

c. Translate survey tools and consent forms to applicable local languages and verify all translations. Note that where local languages do not have commonly used written variants, on-the-fly translation can be used, but standard verbal translations for all key terms and phrases will still need to be developed and agreed upon among enumerators to ensure the consistency and accuracy of translations.

d. Select suitable direct measurement methods, such as water quality testing methods.

e. Develop a sampling plan.

f. Develop/adopt appropriate training materials.

g. Develop a data analysis plan, specifying which indicators and statistics will be tracked and which advanced analyses will be performed.

2. Develop and implement QA/QC protocols.

3. Select data collection method (paper-based or appropriate MST, the latter being strongly recommended).

4. Develop terms of reference for field enumerators and supervisors (whether these personnel are existing staff, new hires or third-party contractors). Ensure that the necessary language skills for the monitoring area(s) are specified.

Procurement, Training and Piloting1. Procure all necessary equipment and supplies:

a. Mobile devices for data collection, if a mobile survey tool is used

b. Supplies for water quality testing (such as the DEL AGUA kit, Wagtech Potalab, or WaSH MEL Field Kit [Appendix II.3])

c. Coolers, gloves, hand sanitizer (for collecting water quality samples) and any other miscellaneous items

2. Recruit/designate/contract field enumerators and supervisors. Ensure that all enumerators have the necessary language skills for the area(s) in which they will work.

3. Conduct thorough training of enumerators and supervisors using appropriate training materials (a minimum of two weeks of training is recommended for high-quality MEL,

Checklist (cont.)


including a minimum of one week of supervised data collection in the field [deliberate practice using all tools and methods under the supervision of experienced trainers]).

4. Once enumerators and supervisors are fully trained, pilot all MEL tools in five to 10 communities outside of the designated sample that are as similar as possible to sampled communities.

5. Refine survey tools and methods as needed.

Data Collection1. Collect data in all sampled communities.

2. Perform weekly QC checks of all collected data.

3. Quickly resolve all data quality issues through refresher training or other changes as needed.

4. Document all substantive mistakes in collected data for subsequent data cleaning.

5. Ensure data are stored in such a way as to protect the privacy of all survey participants.

Data Analysis and Reporting1. Review all collected data and calculate summary statistics for all key variables and

indicators specified in the data analysis plan.

2. Track key indicators and statistics for each round of monitoring and over time.

3. Conduct all analyses and regressions specified in the data analysis plan.

4. Conduct additional analysis on an as-needed basis, depending on the nature of the M&E data and any unanticipated needs related to program monitoring and improvement.

5. Incorporate data analysis results into regular reports; in addition, some data and analysis may be viewable in real-time through a secure online dashboard, if a data aggregation and analysis platform supporting such features is used.

6. While protecting the confidentiality of any identifiable personal data, disseminate data and/or reports to key stakeholders:

a. Internal stakeholders within your organization

b. External stakeholders such as funders and/or collaborators

c. National and/or local government agencies

d. International organizations with an interest in WaSH M&E

Continuous Quality Improvement1. Determine whether the organization will implement CQI as part of its MEL initiative.

2. Establish a CQI team within the organization. This can be one or more standing teams or ad-hoc teams specifically created for each improvement project.

3. Hold an initial CQI training to familiarize the team with CQI methods.


Checklist (cont.)

4. Identify an improvement challenge to be addressed by the project.

5. Develop a charter to set the scope and objectives of the improvement project.

6. Specify key outcomes and potentially associated x variables to be measured in order to drive improvement.

7. Incorporate these outcomes and indicators into the MEL data collection activities described above.

8. Collect and analyze data.

9. Review data analysis to identify root causes of problems and target the largest opportunities for improvement.

10. Develop an improvement package based on these data analysis results.

11. Pilot the identified improvement at a limited scale.

12. Conduce uptake monitoring to assess the impact of the improvement package.

13. Iteratively refine the improvement as needed.

14. Sustain and scale successful improvements, and translate them into organizational standard operating procedures.

15. Continue additional improvement cycles on this project, or launch the organization’s next improvement project.

Ensuring Ongoing Monitoring Quality1. Regularly revisit M&E framework, tools and methods. Update as needed.

2. Regularly review data quality, and update QA/QC checks to address any major outstanding issues.

3. Periodically review mobile survey tools and data management/information security protocols. Update as needed.

4. Periodically review list of stakeholders receiving data and/or reports. Update as needed.

C O M M O N M I S T A K E S A N D P I T F A L L S4.3.

4.3 .1 . Output Emphasis and Lack of Common Output Metrics

The improved source indicator was used to monitor the MDGs and it has been criticized

for inadequately representing water and sanitation safety, quality and level of service.

Much of local and national government and project and program monitoring, including

much of that done by CNHF partners, has focused on measuring outputs rather than

outcomes and services.


The WHO/UNICEF Joint Monitoring Programme (JMP) for Water Supply

and Sanitation, the organization responsible for monitoring water and sanitation,

has identified several important priorities in the SDG era: progressively eliminating

inequalities, achieving universal coverage of basic water and sanitation services, improving

sustainability, improving the levels of service so that water and sanitation services are

safely managed and improving water and sanitation in nonhousehold settings (e.g., schools

and health care facilities).1, 21, 38, 39

38Cronk, R., et al. 2015.Monitoring drinking water, sanitation, and hygiene in non-household settings: priorities for policy and practice. International Journal of Hygiene and Environmental Health 218.8: 694-703. 39 WHO/UNICEF. 2015. Water, sanitation and hygiene in health care facilities: status in low and middle income countries and way forward. Geneva.

4.3 .2 . Lack of Adequate Sampling and Sampling Size

As noted earlier, adequate sampling is critical to obtaining accurate M&E data in a cost-

effective manner. Some WaSH implementers explicitly conduct sampling as part of their

M&E activities, selecting a subset of communities or water sources to monitor, while

others may opt to monitor all communities in which they work and/or all water sources

they have constructed (typically such exhaustive monitoring of communities and/or

sources cannot be done every year for budgetary reasons). However, whether or not it is

recognized, virtually all WaSH programs that conduct monitoring at the household level

necessarily conduct sampling to identify a subset of households to survey, since it would

be virtually impossible to visit every household in each community to collect monitoring

data. In many cases, this household monitoring is not done via a robust random sampling

approach but is left up to the discretion of enumerators or staff in the field, who may

look around and select a given number of households in each community arbitrarily,

without using a robust randomization method. Such an approach tends to lead to

convenience sampling, rather than true random sampling; specifically, it may tend to bias

surveys towards households that are more centrally located and against households that

are inconvenient to survey due to location, the respondent’s primary language, absence

of a respondent on the first visit (WaSH MEL training manuals instruct enumerators

to return at least once more if no one is home on the first visit) or other factors. These

biases can tend to skew the results of monitoring data collected in important ways. For

example, households that are centrally located may tend to have greater access to water

and sanitation facilities, which may in turn influence the quantity and/or quality of water

the household consumes as well as their sanitation practices. Centrally located households

may also tend to be wealthier, which may influence a number of other variables and

outcomes of interest. Finally, if there is no documented method of identifying households,


it is difficult to assess the degree of bias that may be present in resulting data. These issues

are similarly applicable to programs that conduct convenience sampling when selecting

communities and/or water sources to monitor.

In addition, where sampling occurs (either via convenience or random sampling),

the sample size used can have a large impact on the results obtained. Many WaSH

implementers who conduct sampling at the community, facility and/or household

level may not perform sample size calculations when determining how large a sample

to monitor. Failure to use adequate sampling methods may result in

collecting too little data to answer key monitoring questions (e.g., how are

our programs performing and how can we improve them?) or to make meaningful

comparisons between program areas, time periods or treatment conditions (e.g., those

who have received intervention A vs. those who have not yet received it). Inadequate

sample sizes may also lead to false conclusions when making such comparisons

(e.g., a WaSH implementer may conclude that sanitation access is higher in Region A

than in Region B because the average coverage rates are higher for Region A, even if these

two values are not significantly different). Basing program changes on such a spurious

conclusion (e.g., shifting funding from one region to the other, or emulating program

elements from Region A), could thus lead to unintended consequences. It should also be

noted that, even with adequate sample sizes, comparisons over time can be problematic

because multiple factors other than program activities may affect outcomes of interest

(e.g., changes in climate, wealth, other demographic shifts, unrelated government

or NGO programs in the area). Unless these other potential confounding factors are

controlled for, changes in outcomes over time cannot be attributed to program changes or

activities and any such attributions may lead to false conclusions. These false conclusions

may be particularly problematic since short-term increases or decreases in measured

program performance may be attributed to recent programmatic changes, even if the

performance shifts are unrelated (due to confounding) or nonexistent (i.e., the shifts may

not be statistically significant if a small sample size was used). Thus, failure to conduct

adequate sampling and use an appropriate sample size can, at best, reduce the value and

quality of M&E data and, at worst, lead to false and/or misleading conclusions.

4.3 .3 . Lack of Adequate Monitoring Tools

Even with appropriate sampling methods, high-quality M&E data cannot be collected

without appropriate tools. Some WaSH implementers may conduct monitoring activities

using survey tools that do not contain properly designed questions (e.g., surveys may

include leading questions, confusing or double-barreled questions, questions with

problems due to recall bias or socially desirable response bias, or questions that respondents


may not know the answers to). In other cases, questions may be adequately designed

but may not have adequate operational definitions and standards for the terms and

concepts they refer to. For example, without clear operational definitions of terms like

“functionality,” two enumerators may visit the same handpump that produces a low flow

after 20 pump strokes but may come to different conclusions about whether that source is

functional. Similarly, without precise operational definitions about what makes a latrine

accessible to those with limited mobility, inconsistent results may be obtained across

different enumerators and communities.

Finally, survey tools may have been adequately constructed, including adequate

operational definitions, but may not have been properly validated in the field. As a result,

survey questions may appear sound on paper, but in practice they may not produce

usable information in the local context due to issues with translations, culturally specific

constructs (e.g., the concepts of “water for productive uses” or “adequate water for

daily needs” may have very different meanings or no clear meaning in some contexts),

etc. Robust and validated tools will have addressed most of these issues so that the data

collected are far more likely to mean what WaSH implementers believe they mean.

However, while prior validation in one or multiple contexts improves the quality of

survey tools, it cannot guarantee that they will perform properly in a new context.

Piloting is always recommended. Nevertheless, the use of properly constructed survey

tools and operational definitions that have been validated in the field in one or more

countries can greatly improve the quality of M&E data.

4.3 .4 . Bias and Errors

Steps for minimizing the likelihood of bias and errors in survey questions are addressed

above, but bias can also be introduced during the design of M&E programs and survey

development phases. Sampling error deals with the precision of statistical estimates and

occurs when the sample is different from the population. Sampling bias occurs when the

sampling method favors the selection of some people over others. Convenience sampling is

a form of sampling bias. Careful sampling methods can reduce the likelihood

of sampling bias and errors. Training enumerators and other staff on the sampling

method and the importance of random sampling can help ensure that bias and error

prevention measures are respected.

Bias can also occur while the survey is being administered, for example, if the

enumerator attempts to answer the question for the respondent or if the enumerator asks

the respondent a leading question. Both can result in biased data, and it is important to

cover these possible sources of bias during the data collection training.


4.3 .5 . Absence of Quality Control

Although QA/QC is standard practice in most industries and in monitoring activities

conducted by many government agencies, such reviews are virtually nonexistent in the

WaSH sector. As a result, it is difficult or impossible to determine whether most WaSH

M&E data collected by WaSH implementers is credible. In the absence of adequate QA/

QC checks, such data should be interpreted with extreme caution, given the ease with

which bias and error can be introduced. While QA/QC checks do not need to be

elaborate or burdensome, their absence should be seen as a red flag with

respect to the credibility of M&E data and their introduction could greatly strengthen

most WaSH monitoring programs.

4.3 .6 . Problematic Assumptions

Operational research and evaluation of monitoring data are increasingly being discussed

by water and sanitation leaders as important methods of addressing water and sanitation

challenges and improving service delivery in lower middle income countries.21, 40, 41

Evidence generated from service delivery research can be used to help policymakers,

planners and practitioners make better decisions about how to manage infrastructure assets

and identify processes that lead to better service provision.

The many problems with data collection and the quality of water and service delivery

research studies are well documented.26, 42 Problems include data collected without a

clear objective or purpose, the use of poorly designed survey questions, and the use of

inadequate methods or inadequately documented methods. Data analysis may be limited

to descriptive statistics, with modeling methods infrequently used to examine relationships

between service outcomes and determinants.42 Results of service delivery studies are

infrequently published in peer-reviewed journals or translated into policy and practice

recommendations.26

Inadequate data collection and limited analysis of data wastes limited resources

and makes it difficult for decision makers to understand and synthesize evidence.

While improvements are desirable, a systematic approach has not previously been taken

to characterize the problems and challenges affecting service delivery data collection

Inadequate data collection and limited analysis of data wastes limited resources and makes it difficult for decision makers to understand and synthesize evidence.

49C O M P E N D I U M O F B E S T P R A C T I C E S A N D L E S S O N S L E A R N E D 5 . L E V E R A G I N G M E L

40 Department for International Development. 2012. Water, Sanitation and Hygiene Portfolio Review. London. 41 WHO. 2016. Water, Sanitation and Hygiene (WASH) in Health Care Facilities Global Action Plan.42 Royston, G. 2011. Meeting global health challenges through operational research and management science. Bulletin of the World Health Organization 89(9): 683–8.43 Foster, T. 2013. Predictors of sustainability for community-managed handpumps in sub-Saharan Africa: evidence from Liberia, Sierra Leone, and Uganda. Environmental Science & Technology 47(21): 12037–46. doi: 10.1021/es402086n.44 Atengdem, J., et al. 2013. Service level and sustainability of water supply in East Gonja Northern Region, Ghana. Triple-S working paper.45 Walters, J. P., and Chinowsky, P. S. 2016. Planning rural water services in Nicaragua: a systems-based analysis of impact factors using graphical modeling. Environmental Science & Policy 57: 93–100.

specifically in water and sanitation; document and synthesize examples of effective studies;

and present solutions and recommendations for improvement.

Studies on water and sanitation service delivery reveal a number of important policy

and practice findings, but they also have important methodological limitations. In some

studies, some survey questions could have been improved or modified to reveal greater

insight. For example, in three studies, data were removed from analysis due to concerns

about the reliability of survey questions.3, 22, 43 In these studies, researchers were not

involved in the design of the data collection instrument, design of the survey questions or

the data collection. In other examples, ethics approval was not reported, and in some cases,

data collection methods were not reported.44, 45 Data limitations and sources of bias were

often not documented.44, 45 Data sets were often not publicly available for further analysis

or meta-analysis.44 •

LEVERAGING MEL:TURNING M&E FIT-FOR-PURPOSE DATA INTO IMPROVEMENTB A C K I N G U P E V I D E N C E W I T H A C T I O N5.1.

5.

While there is considerable interest in WaSH monitoring and evaluation, collecting M&E

data for the purpose of reporting and publicity adds little value to the end users of WaSH

services. In contrast, the use of high-quality, fit-for-purpose data to drive improvements

in the quality and sustainability of WaSH programs can have a dramatic impact on the

lives of the populations those programs serve. Prior to the MDG period, it was common

for WaSH implementers to complete water and sanitation projects with little in the way

of monitoring and reporting of outcomes. In recent decades, routine M&E of WaSH

programs has become much more widespread, but the resulting reports often languish on

shelves and hard drives without prompting substantive action to learn from their findings

and leverage these learnings for improvement. Achieving the ambitious new water and

sanitation goals laid out by the SDGs will require WaSH implementers of all varieties to


improve the quality and efficiency of their programs and activities. Leveraging fit-for-

purpose WaSH M&E data to drive this improvement through systematic implementation

science methods may be one highly effective approach.

A N I M P R O V E M E N T M I N D S E T5.2.

One of the most important shifts needed for such evidence-based improvement to

occur is the proliferation of an improvement mindset among WaSH implementers and

practitioners. Currently, many WaSH implementing organizations, including nonprofit

organizations, bilateral aid organizations and national government agencies and ministries,

tend to view WaSH implementation through an output-based lens. Specifically, many

implementers perceive the need to deliver WaSH services but tend to view this task as a

process of converting unserved populations to served populations through the delivery

of a fixed suite of interventions. The idea of using M&E data to modify and improve

service delivery processes in order to achieve better outcomes is alien to many WaSH

implementers. However, when these ideas are presented and evidence-based quality

improvement tools are made available, implementers seem eager to embrace them,

provided it does not interfere with meeting existing output-driven deadlines. Thus, before

the quality and efficiency of WaSH service delivery can be improved, a mindset shift

may be required among WaSH implementers, specifically, a shift from focusing on

outputs to one that includes in addition to outcomes the empowerment

of implementers at all levels to modify and improve the processes

through which they deliver WaSH services. These mindset shifts, accompanied

by the availability of suitable WaSH quality improvement tools and methods, have the

power to considerably improve quality and efficiency in WaSH service delivery.

R E V I E W O F C Q I A S A M E T H O DF O R S O L V I N G C O M P L E X P R O B L E M S

5.3.

Implementation science (IS) methods such as CQI are predicated on the core assumptions

that all work is made up of processes and every process can be improved. As an example,

even a task as simple as making tea follows a defined process: water must be collected and

boiled, then tea must be retrieved and placed in a cup or pot. The hot water must then

be added to the tea. This mixture must be allowed to steep for a defined period of time

or until a desired strength of tea is achieved. While “making tea” may seem like a single

action—one often performed automatically—it is actually a process with multiple defined

steps. To improve the process, the individual steps can be studied to determine which is

the slowest, the most costly or the most prone to errors, and attention can be focused there.


While the application of these core assumptions to something as mundane as brewing

tea may seem trivial, these principles are equally applicable to a wide variety of processes

across virtually every sector of the global economy. Implementation science methods such

as LEAN, Six Sigma, total quality improvement, continuous quality improvement and

others have been used to revolutionize sectors as diverse as manufacturing,12 customer

service,13 health care,14 finance15 and others. These IS methods have differences but are

broadly similar in their functions; in all cases, the IS process involves clearly defining a

specific problem to be solved or opportunity to be seized, collecting and analyzing relevant

data, and using the findings from that analysis to conceive and implement improvements.

This cycle, in its most general form, has been described as the “Plan, Do, Study, Act,”

or PDSA cycle. Extensive evidence demonstrates the effectiveness of these methods in

improving processes and outcomes across each sector in which they have been applied.12-15

Implementation science methods are most suitable for solving complex process

problems in which a process that already achieves a desired outcome is to be improved.

Complex problems are defined as those for which an effective solution is not yet known

and is not intuitively obvious to those tasked with solving the problem. Examples

of complex problems include improving fuel economy in compact passenger cars or

reducing the waiting time in a hospital’s urgent care clinic. Each team member may have

preconceived ideas of how best to solve this problem, but it is not necessarily intuitively

obvious that any one idea will work, let alone which (if any) will be most effective. By

contrast, a problem that is not complex is one that has a known or obvious solution that

has not been implemented for some reason other than lack of knowledge about what to

do. Examples of noncomplex problems include a flat tire on a passenger car or a broken

elevator in an urgent care clinic. Furthermore, IS methods are generally better suited for

improving the quality or efficiency of a process rather than increasing the extent to which

that process is implemented or the level of investment in its implementation. For example,

IS methods are well suited for reducing the failure rate of water systems or the cost of

such systems but are less well suited for increasing the number of systems a government

or implementer chooses to implement. While these methods may not be universally

applicable to all problems, however, IS methods remain powerful tools for improvement

in a wide variety of complex systems across virtually all sectors.

The PDSA cycle

• Plan the desired improvement

• Do the monitoring

• Study the data

• Act to make an improvement based on the findings

5.3 .1 . Suitability of CQI for Addressing Complex WaSH Problems

Given the strength of IS methods such as CQI for solving complex problems, it seems

logical to apply these approaches to the complex challenges facing WaSH implementers

as they prepare to meet the ambitious targets laid out by SDG 6. Specifically, adapting

WaSH programs to improve the safety and reliability of water supplies are complex


problems without obvious or known solutions; doing so in a cost-effective manner will

be a priority for many implementers seeking to serve ever-increasing populations with

limited resources. Likewise, enhancing the quality of sanitation and hygiene services

and increasing sanitation and hygiene uptake are complex problems that may benefit

from systems thinking and CQI methods. The collection and analysis of high-quality

fit-for-purpose data may reveal the root causes of problems in these areas or highlight

opportunities to capture greater efficiencies.

5.3 .2 . Adaptation of CQI Methods to WaSH

WaSH programs have many similarities to manufacturing, service industries and

health care, which suggests IS methods could be applied to many WaSH challenges.

Specifically, WaSH programs typically use defined implementation processes. For

example, government and nonprofit implementers who install water supply technologies

such as boreholes and piped schemes will typically execute defined planning, siting,

implementation, commissioning and training activities in a reproducible process;

management and maintenance of these systems depend on additional defined processes.46

These processes rely on implicit or explicit standard operating procedures and are therefore

suitable for systematic quality improvement. Similarly, many WaSH implementers will

conduct sanitation interventions using standardized methods such as community-led total

sanitation, which also has a defined and standardized process that is open for adaptation

and improvement.47

Furthermore, like medicine, WaSH programs are designed to produce measurable

outcomes, including changes in health and wellbeing, in addition to satisfaction and

changes in other proxy variables (such as access to water and sanitation services,

water quality and water quantity), which may be easier to measure and more rapidly

sensitive to system improvements than changes in underlying health status and

livelihood outcomes. Unlike manufacturing and hospital-based medicine, most WaSH

programs are implemented in diverse and decentralized community settings rather

than centralized, controlled industrial or clinical environments. However, successful

examples of quality improvement methods in community-based medicine48 suggest that

community-based or community-managed WaSH programs may be similarly suitable

for systematic improvement. Furthermore, many WaSH programs, particularly those

related to sanitation and hygiene, require behavior change on the part of the end user

to be successful; likewise, behavior change is a component of some successful quality

improvement projects in health care as well.49


5 .3 .3 . Implementing CQI in WaSH Programs

The justification for the application of quality improvements methods to challenges

in the WaSH sector is ample. Despite many similarities, however, direct translation

of quality improvement methods from other sectors is not appropriate. Some unique

aspects of WaSH require adaptation of conventional quality improvement methods. For

example, while many water and sanitation interventions in urban areas may be managed

by municipal or private utility companies, community management of rural water and

sanitation systems is common in many lower middle income countries and presents

unique features and challenges with respect to implementing improvement projects.

Furthermore, the challenges of data collection in WaSH contexts may be much greater

than in clinical and industrial contexts, where trained clinicians and workers may be

readily available to collect data with minimal disruption to their ongoing activities. In the

case of WaSH CQI, the implementers driving improvement activities may be separated

from the communities and systems they wish to measure by greater magnitudes of time

and space, particularly where improvements in post-implementation outcomes of rural

WaSH programs are desired. This separation in time and space requires creative solutions

to collect timely, high-quality data in a focused and cost-effective manner.

Initial applications of CQI methods to WaSH programs have been undertaken in

West Africa with promising results. In Ghana, a CQI project demonstrated improvements

in household water quality and promising signs of improvements in water source

functionality. In Burkina Faso, safe water storage innovations from Ghana are being

adapted to the Burkinabe context with promising initial results. In Niger and Mali,

programs are underway to conduct CQI projects focusing on household water quality as

well, while in Ghana the next round of CQI is underway to identify opportunities for

improving sanitation outcomes in rural communities. In all cases, novel implementation

challenges have been identified and addressed. Thus, this work is not only building

an evidence base for the value of IS and CQI methods in WaSH, but also addressing

methodological questions as part of the process of establishing best practices in WaSH

CQI. Based on current activities and results, it seems likely that implementation science

methods may play a valuable role in improving service quality and contributing to the

achievement of WaSH-related SDGs. •

46 Harvey, P., and Reed, B. 2004. Rural Water Supply in Africa: Building Blocks for Handpump Sustainability. WEDC, Loughborough University.47 Crocker, J., et al. 2016. Impact evaluation of training natural leaders during a community-led total sanitation intervention: a cluster-randomized field trial in Ghana. Environmental Science & Technology 50.16: 8867–75.48 Fox, P., et al. 2007. Improving asthma-related health outcomes among low-income, multiethnic, school-aged children: results of a demonstration project that combined continuous quality improvement and community health worker strategies. Pediatrics 120.4: e902-e911.49 Fox, C. H., and Mahoney, M. C. 1998. Improving diabetes preventive care in a family practice residency program: a case study in continuous quality improvement. Family Medicine 30: 441–5.

WaSH MEL COMPENDIUM - Water Institute3.1. 3.2. 3.3. Lesson 1: Sustainability Requires...

Documents

Transcript of WaSH MEL COMPENDIUM - Water Institute3.1. 3.2. 3.3. Lesson 1: Sustainability Requires...