Editor's Note: In this post, Heloise Greeff, Doctoral Researcher, Water Programme, at the Computational Health Informatics Lab and Smith School of Enterprise and the Environment at the University of Oxford, discusses how we can pro-actively monitor the condition of handpumps and ensure that millions of people can access a reliable water source. This post originally appeared on the REACH program's site, the orginal article can be accessed here.
Predictive health monitoring is widely used in engineering applications to detect damage to infrastructure as early as possible. Forecasting failure rather than merely detecting failure once it occurs helps to reduce the downtime of systems. Ideally, predictive maintenance can be used to avoid downtime completely. With this approach already widely used in many fields from commercial and military jet engines to patient monitoring in health systems, it is now being extended to monitoring the condition of handpumps in rural villages.
The handpump remains a reliable and low-cost method to access groundwater, making it a critical component of rural water supply for around 200 million people in Sub-Saharan Africa. Community handpump models, such as the Afridev and India MK II, are designed to lift water from deeper sources than traditional rope and bucket systems which can only be used in shallow wells. However, high water demands result in continuous usage and frequent breakdowns. Unfortunately, practical challenges in the supply of spare parts combined with a lack of local skills and resources result in an estimated 30% of handpumps in Africa not working at any given time.
The use of predictive maintenance in handpumps has the potential to limit interruptions of weeks or more which are common across rural Africa. A broken handpump in a remote village can force women or girls to walk up to 20 kilometres to find alternative sources which may be contaminated or expensive. Reliability and sustainability of water supplies are important to ensure healthy communities, societies, and economies in all regions of the world.
The Oxford University ‘Smart Handpump’ was successfully introduced in 2012. Proof-of-concept for the remote monitoring of handpumps used a simple microprocessor, accelerometer and global system for mobile communications (GSM) components. The Smart Handpumps provide hourly data on usage.
In 2014 a preliminary analysis used high frequency accelerometer data to show that these patterns contain useful information. High-rate waveforms from the accelerometry data can be processed using robust machine learning methods that are sensitive not only to the dynamics of the whole system but also the subtle interaction between the user and the pump. The small changes in pump dynamics and the subtle reactions of the user become a prominent signal in determining the deterioration of pump mechanics and imminent failure. This same signal can also be used for monitoring the level of the shallow aquifer at the pump location.
By retrofitting a simple and inexpensive device to a standard pump handle, the Smart Handpumps are able to pro-actively monitor the condition of handpumps and ensure that millions of people can access a reliable water source.
In February 2016 we visited 33 different handpumps across the Kwale County in southeast Kenya. We recorded 103 different users extracting approximately 5,059 litres of groundwater using the handpumps. These data will be used to produce a low-cost predictive maintenance system that is scalable across large rural regions. The development of a prototype hardware system is being supported by UNICEF, funded through a competitive tender process, as part of their Product Innovation portfolio. Field testing will be conducted in partnership with UNICEF country programmes in Eastern and Southern Africa.
By monitoring the heartbeat of thousands of handpumps across Sub-Saharan Africa and South Asia, it is possible to give millions of people access to a reliable and secure water network. The handpump network has existed for many years and despite being neglected remains the most reliable method to access groundwater in remote locations as the world advances to achieving universal drinking water security.
Take a look at the video to see our work in Kwale County in Kenya in action:
Editor's Note: How can we assess the impact of a WASH investment? In this post, Guy Norman, Head of Evaluation, Research and Learning at Water & Sanitation for the Urban Poor (WSUP), gives us a snapshot of the findings from their recent pilot analysis of the impact of WSUP’s work in Madagascar.
Providing improved water, sanitation and hygiene (WASH) services to a community doesn’t just have an impact on health, it can be expected to have multiple positive impacts, including creation of livelihoods. For a slum-dweller, employment and a steady income are life-changing things! And jobs created are likely to have a ripple effect in the local economy—more jobs mean more money circulating around the community.
But assessing the total impacts of a WASH investment by an organisation like Water & Sanitation for the Urban Poor (WSUP) is far from straightforward. For example, we would expect an improved water supply to have positive impacts on a wide range of things including health, livelihoods, time required to collect water, local environmental quality, the water utility’s revenues, and indeed the national economy. What’s more, a given investment may also have negative impacts; for example, an improved water kiosk may reduce the profits of an existing water supplier. So any assessment of total impact needs to consider a wide range of potential impacts, and needs to “subtract” possible negative impacts from the positive impacts.
Moving towards achieving this, WSUP has recently contracted PricewaterhouseCoopers (PwC) to pilot their Total Impact Measurement and Management (TIMM) framework in Antananarivo, the capital of Madagascar. TIMM is a framework for doing precisely this type of analysis.
More detailed findings of this pilot analysis will be published soon, and this blog gives a “sneak peek” at findings around the impact of WSUP’s work in Antananarivo on employment.
The Antananarivo pilot of the TIMM approach
The pilot focused on a subset of WSUP’s activities in Antananarivo over the period 2012−2015:
1. Water kiosks and laundry blocks: support for construction, including creation of management arrangements;
2. Setting up small community-based organisations (known as RF2s) for street cleaning and drain clearance;
3. Helping the water utility (JIRAMA) reduce its levels of non-revenue water (NRW) (i.e. water for which no payment is received, either because it physically leaks from pipes or because it’s supplied but not paid for, as a result of inefficient billing for example)
Very briefly, the TIMM approach, as applied here, involved first generating a model of the impact pathways. This included many boxes and arrows (WSUP investments in boxes on the left, outcomes in boxes on the right, many intermediate boxes in the middle, and lots of “causality” arrows linking everything together). Data to “calculate” the model comprised existing WSUP data on project outcomes, including householder survey data, some limited additional data collection, and international data derived from a literature review.
The estimated net impacts of WSUP’s investments in the three activities indicated above are shown in the wheel schematic below. Each impact has been converted to a monetised value, but this in no sense implies that people’s lives are just about profit and loss! Rather, an analysis of this type requires a common unit of measurement of different types of impact, and a convenient unit is U.S. dollars. Note the dramatic health impacts (interesting, because previous analyses of this type have often suggested that WASH interventions have more of an impact on time savings than they do on health). Note also some negative impacts in terms of greenhouse gas emissions (garbage rotting on the streets produces less methane than garbage in a landfill). But as noted at the end of this blog, we ask you to take these results as preliminary; these findings may be a good indication of reality, but PwC, WSUP and others are still assessing to what extent we can consider these results to be accurate.
Job creation findings
WSUP-supported water kiosks and laundry blocks need builders to construct them and attendants to staff them; both also provide self-employment opportunities for laundry washers (almost exclusively women). The TIMM analysis indicated that kiosks and laundry blocks not only generate direct employment, but also create work opportunities for suppliers; much of this money is then spent in the local community on goods and services, supporting more jobs. The construction sector gets a demand boost, and the new connections between kiosks and the water network also increases profits and wages for those working at the water utility. Water resellers, who buy water in bulk from WSUP’s kiosks and then sell it door-to-door at a mark-up, are estimated to now have an annual income of around 300,000-480,000 Malagasy ariary.
RF2 teams sweep the streets and empty bins, and each team employs around eight people. These are self-sustaining organisations financed by user tariffs and local government support. The RF2 impact on livelihoods is estimated at almost US $1.5 million, and the spending of those wages generates an additional US $0.4 million of indirect economic benefit.
In order to help the water utility to reduce its level of non-revenue water, WSUP provided the initial funds for a team dedicated to NRW reduction and technical training. Not only did this create jobs within the utility, but the wages that they were paid are projected to have had wider economic effects within the city. WSUP’s work with the water utility led to an estimated increase in profits and wages of US $2 million, with wider economic benefits totalling approximately US $157,000.
The TIMM results show that livelihood impacts make up more than 13% of the total net impact of the WSUP projects analysed. This should have long-term positive effects; the total livelihoods impact for women and men across all interventions from 2013 to 2025 is worth approximately US $3.6 million and US $2.3 million, respectively. Laundry workers (who are 95% female) are the highest earners, and are projected to generate US $2.8 million in additional earnings by 2025.
Particular impacts on women’s livelihoods
Madagascar ranks somewhere in the middle of the global Social Institutions & Gender Index, and while women make up nearly half of the work force they are paid less and hold a lower proportion of skilled and managerial jobs compared to men.
This TIMM analysis indicates that about 70% of the total earnings from WSUP water kiosks and laundry blocks go to women. Each laundry washer earns an estimated 8,000 Malagasy ariary per day (around $490 per year), which is about three times more than before the laundry blocks were built. More earnings for women is socially relevant as women are more likely to invest their earnings in education, nutrition, and health than men.
And finally, a cautionary note
This has been a pilot application of the TIMM approach, based on careful analysis of likely impact pathways, careful collation of relevant data, and reasoned assumptions about likely magnitudes of effect along each of the impact pathways that we examined. But this is complex modelling, the input data is certainly not perfect, and the assumptions about magnitudes of effect are just that, best-estimate assumptions. Modelling of this type can generate very useful information for an organisation like WSUP, but at this stage the jury is still out on whether the current results are sufficiently reliable, and whether the data collection requirements for this type of analysis are manageable. We’ll be releasing more detailed publications soon, so if you’re interested please watch out for them.
Thanks to the PwC team led by Tom Beagent for this excellent work and to WSUP’s Rosie Renouf for help with the analysis underlying this blog, as well as to WSUP’s supporters DFID, TCCAF, and the Stone Family Foundation for their role in this project.
Plan International USA and the Water Institute at the University of North Carolina at Chapel Hill (UNC) have released new findings and results about rural sanitation behavior change processes using the Community-led Total Sanitation (CLTS) approach. Entitled CLTS Learning Series: Lessons from CLTS Implementation in Seven Countries, the research report identifies implications for practice and delivers policy recommendations based on a rigorous review of seven country case studies and their approach to CLTS implementation.
Covering experiences from Haiti, Uganda, Niger, Cambodia, Lao PDR, Nepal, and Indonesia, long-form, individual country reports are complemented by a meta-analysis of all case studies, as well as a shorter, executive summary style briefing paper for rapid review.
The reports present common features to CLTS implementation, identifies consistent bottlenecks and enabling conditions, and shares lessons relevant to scaling-up CLTS.
Copies of all reports from this work are available at the project website: https://waterinstitute.unc.edu/clts/
Editor’s note: This blog post was authored by Ryan Cronk, PhD student researcher at The Water Institute at the University of North Carolina-Chapel Hill, which focuses on producing practical and relevant sector knowledge by linking research with policy and practice. In his post, Ryan describes the development of the WaSH Performance Index, which compares progress on WaSH access and equity across countries, and highlights some insights that the 2015 Index reveals.
What is the WaSH Performance Index?
The WaSH Performance Index is the first index to rank countries based on water and sanitation performance and on implementation of the human right to water and sanitation. It is the first of its kind in that it compares countries of all sizes, water and sanitation coverage, and income levels. By assessing how countries are improving water and sanitation compared to best-in-class countries at similar levels of water and sanitation coverage, the Index provides a fair comparison of progress.
Why a performance index?
Water, sanitation and hygiene (WaSH) are essential to human health and development, and water and sanitation are recognized as human rights. Proposed global targets for the post-2015 Sustainable Development Goals (SDGs) call for universal access to WaSH and reducing inequalities in access. The forthcoming SDGs provide potential for convergence of human development and human rights policy.
Monitoring approaches to assess progress toward proposed goals have focused on the level of coverage of water and sanitation. New instruments are necessary to monitor and evaluate country performance on WaSH and to ensure progressive realization of the human right to water and sanitation.
Looking at coverage between countries, such as improved water access in the United Kingdom (100%) and Mozambique (49%), does not provide a meaningful comparison. An improved approach is to compare rates of change of coverage. Countries like Mali have been doing well (improving at 4.3% per year) while countries like Colombia have not been improving (-2.4% per year).
The challenge in comparing rates of change is that countries are at different levels of WaSH development. When comparing levels of coverage with rates of change, we tend to see rates increasing at low levels of coverage, plateau at intermediate levels of coverage, and slow as they approach 100% coverage. To compare countries fairly, we must compare country rates of change to best-in-class rates of change at different levels of coverage.
The WaSH Performance Index meets these needs by comparing country performance on increasing access and equity to best-in-class performance at different levels of water and sanitation coverage. The Index provides insight based on already-available water and sanitation data from the WHO/UNICEF Joint Monitoring Programme for Water Supply and Sanitation. It is designed to accommodate new types of data relevant to the SDGs, such as hygiene, water safety, and non-household settings, as they become available.
The WaSH Performance Index answers two policy questions:
- How quickly are countries improving access to improved water and sanitation relative to best-in-class performance?; and
- How quickly are countries improving equity in access to improved water and sanitation relative to best-in-class performance?
What are some of the interesting insights from the 2015 WaSH Performance Index?
- High-performing countries in the 2015 rankings are those that achieved significant improvement in recent years compared to their peers. These include El Salvador, Niger, Egypt, Maldives, and Pakistan. Low-performing countries are those that showed stagnation or decline in recent years compared to their peers, such as the Dominican Republic, the Gambia, Ghana, Samoa, and Timor-Leste.
- Despite persistently being the region with the lowest water coverage in the world (Figure 1), water access performance among countries in Sub-Saharan Africa varies widely, with both high and low performers (Figure 2). Identifying characteristics of high performing countries and learning from them may enable more rapid progress among countries.
- Among the most populated countries in the world, Pakistan, China, and Nigeria were top performers (ranked 5, 11, and 18 respectively), while Russia, the Philippines and India were bottom performers (ranked 72, 83, and 92). India’s ranking as a bottom-performer predates the recent launch of the “Clean India Mission” by Prime Minister Modi and suggests his initiative may be even more critical and urgent than originally thought
- Progress toward equity in sanitation is significantly associated with governance indicators including control of corruption, government effectiveness, regulatory quality, and rule of law. These results suggest the enabling environment for WaSH contributes to progress in sanitation equity.
- Despite the assumption that countries with higher Gross Domestic Product (GDP) will perform better in improving access to water and sanitation, GDP was not significantly correlated with performance. This means that even countries with limited resources can make great strides if they have the right programs in place. National governments, NGOs, and aid agencies can direct their resources toward building systems and capacity for action in countries that are lagging, and toward implementation where those capacities are in place and performing.
What’s next for the WaSH Performance Index?
In future versions of the WaSH Performance Index, we plan to explore alternate measures of equity such as wealth quintiles and minority groups. We also plan to address levels of service and other important service characteristics such as water safety and continuity. Finally, we plan to develop Index rankings for hygiene, comparing every country with available data.
Where can I learn more about the Index?
You can read the full report here.
We’ve also posted a recorded webcast presentation about the Index online, including remarks from Ed Cain from the Hilton Foundation and a discussion with Vice Chair of Sanitation and Water for All Catarina de Albuquerque.
We would love to hear your feedback and questions about the Index! Please email Ryan at email@example.com
Editor’s Note: This post comes from the Water Point Data Exchange, an initiative aimed at creating a standard for water point data collection to allow for better information sharing across organizations working in WASH. The Exchange invites those working in the sector to provide feedback on their draft standard, available here. The public comment period will be open through March 15.
No challenge in eradicating water poverty is as pressing as improving sustainability by ensuring continuous high-quality drinking water service provision. High failure rates in the WASH sector squander scarce financial resources. Further, broken water systems can cause harm to millions of people in the Global South who have invested their own time and resources and come to depend on these systems. Despite knowledge of this issue for decades and increased attention over the past few years, there is still a severe lack of understanding about the true nature of this challenge. Limited available data on water points has stymied efforts to better understand and improve sustainability. Facing these challenges, the water sector is at a pivotal point in history where data can play a vital role in the eradication of water poverty.
The amount of water point data is increasing as more organizations and governments monitor their water point functionality over time. Unfortunately this progress has been accompanied by increasingly numerous unique methods through which data is collected, stored and shared. As a result, new data is creating new “data islands.” These emerging monitoring efforts will have limited impact on improving programming and sustainability as long as the data remains inaccessible on organizational servers, in PDF reports, and in proprietary monitoring systems. Unless these different data sources can be harmonized, the learnings will be small and piecemeal, with the true potential of this information remaining untapped.
The Water Point Data Exchange (WPDx) plans to harness this momentum by working with the water sector to create a standard for water point data sharing. To learn more about efforts to develop a framework for sharing this data, view the webinar, hosted by Global Water Challenge, on February 5:
Based on this standard, WPDx will create an effective central hub for the standard-compliant data and will engage the WASH sector to make full use of this dynamic hub, both for sharing and utilizing data. Ultimately, this initiative will transform a growing body of disconnected data into improved water services for millions.
To make this initiative work, we need your input! The WPDx Working Group is inviting water access experts from around the world to provide feedback on the draft standard, available at https://collaborase.com/wpdx. This public comment period will close March 15, so please share your feedback soon. It will only take a moment, and no registration is required. Click here to review the draft standard and share your feedback now.
Editor’s Note: This post is authored by Cor Dietvorst, Programme Officer at IRC. In his piece, Cor discusses the monitoring requirements surrounding India’s Swachh Bharat program, which Prime Minister Narendra Modi launched in October 2014 with the aim of ending open defecation in the country by 2019. He compares India’s sanitation monitoring initiative with other large-scale monitoring efforts with which IRC has been involved in Bangladesh and Indonesia. This post originally appeared here on the IRC blog.
According to some media the Indian government has unleashed “toilet police” or “toilet gestapo” into the country.1 In fact, the central government has instructed local officials to take photographs of new toilets to prove that they have not only been constructed but are also being used. If states don’t upload photos by February 2015, the water and sanitation ministry has threatened to withhold funding from a new national sanitation programme.2
Open defecation free by 2019
Prime Minister Narendra Modi launched the Swachh Bharat (Clean India) Mission on 2 October 2014. His aim is to attain a 100 percent open defecation free India by 2019. Since the launch, over half a million household toilets have been constructed.3
By implementing “real time monitoring” the government hopes it can correct past mistakes caused by ineffective monitoring and wasted investment in sanitation. The 2011 census revealed that 43% of government funded toilets were either “missing” or non-functional.4 Now the government wants to show that its investments in sanitation are delivering lasting results.
The Ministry of Drinking Water and Sanitation is appointing around two dozen additional staff including two joint secretaries and four directors to strengthen the implementation and monitoring of the Swachh Bharat Mission. An Expert Committee for innovative sanitation technologies and a national telephone helpline for rural water supply and sanitation are other new initiatives that will support the Mission.5
Local officials charged with monitoring toilet construction and use need to download an app on a mobile device. The app allows them to upload photos as well as the personal data and geo-coordinates of the beneficiaries to a public website. Progress is slow though: as of 14 January 2015, data of less than half a percent (2,383) of the newly constructed toilets has been recorded. Data collected before 2015 does not include toilet use.
How do other countries carry out large-scale monitoring?
Compared to examples of large-scale sanitation monitoring in Bangladesh and Indonesia, the toilet use indicators collected in India -- is the toilet in use, is it clean and is water available -- are rather limited.
The BRAC WASH programme in Bangladesh uses benchmark indicators developed by IRC for questions like: do all household members use toilets, do they use them at all times, and are there provisions for handwashing and pit emptying.6
In Indonesia IRC has helped design a monitoring system for the SHAW (Sanitation, Hygiene and Water) programme, where every three months 20,000 community volunteers visit more than 300,000 households. For SHAW monitoring is not merely an accountability tool as it is in India, but a way to motivate and encourage people to improve their sanitation facilities and hygiene behaviour.7
India's decision to track toilet use as part of its new monitoring initiative is a major step forward. From its neighbours, India can draw valuable lessons on how to monitor sanitation as a sustainable service that benefits all.
2 Letter to Principal Secretary/Secretaries in charge of Rural Sanitation all States and UTs. Ministry of Drinking Water & Sanitation, 05 Dec 2014
3 Unused rural toilets to face public scrutiny, The Hindu, 01 Jan 2015
4 Tiwari, R. The case of the missing toilets. India Today, 02 Oct 2014. See also: Hueso, A. & Bell, B., 2013. An untold story of policy failure : the Total Sanitation Campaign in India. Water policy ; 15 (6), pp.1001–1017. DOI: 10.2166/wp.2013.032. and Hueso, A., 2014. The untold story of India's sanitation failure, Addendum. Community-Led Total Sanitation.org, 11 Mar 2014
5 Nationwide monitoring of use of toilets will be launched from January, 2015, PIB, 31 Dec 2014
6 IRC - Monitoring at scale in BRAC WASH
7 Baetings, E., 2014. How are you and how is your loo?
Have you searched our map to orient yourself to the WASH funding landscape? Do you regularly read this blog? Are you familiar with our featured case studies or funder profiles? If you’ve visited WASHfunders, we’d love your feedback as part of our evaluation of the site.
Launched more than 2 ½ years ago, WASHfunders has been an experiment for our team at the Foundation Center. We hoped WASHfunders would facilitate knowledge-sharing, foster collaboration, and promote transparency related to foundation funding. But the truth is, we know relatively little about how our audience is using the site. Now is an opportune time for us to step back and take stock -- to understand where we can improve and how the site can be a better resource for the sector.
We’re currently working with an independent evaluation team to assess the impact and effectiveness of WASHfunders and want to hear your thoughts on the site.
If you’re interested in participating, the evaluation team will contact you to conduct a short one-time phone interview. If you would like to contribute to this effort, please contact Seema Shah at firstname.lastname@example.org. Thank you for your help!
Editor’s Note: This guest blog post was authored by Ben Seidl, program director at World Water Relief, an NGO launched in 2008 with the goal of bringing sustainable water purification solutions to people in developing nations. In his post, Ben discusses the push for better monitoring and evaluation (M&E) in the WASH sector and the challenges and opportunities that this trend presents for small NGOs. Ben emphasizes the importance of local engagement as the key to both effective M&E and, ultimately, project sustainability.
As the WASH sector continues to expand and strengthen its role in global health, the sector’s trends and objectives have become more data-oriented and results-focused. Mobile, field-level technology has enabled NGOs to undertake data processing and monitoring of water resources in real-time…a practice that was previously only afforded to large municipal utilities and corporations. While this technological leap has ushered in a new era of transparency and reporting, there are some fundamental building blocks of sustainability that are beyond data.
Human capital is still the true driver behind sustainability and M&E in the WASH sector. Local, dedicated stakeholders are the true source of long-term sustainability and accurate, reliable monitoring and evaluation. Without the involvement of these local community stakeholders, the sustainability of any WASH project will undoubtedly wither over time.
As Program Director for World Water Relief in the Dominican Republic and Haiti, my team and I are tasked with building a responsive and flexible monitoring program to ensure that our projects are creating measurable impact and consistent WASH service delivery. World Water Relief is an NGO with limited manpower and resources. Thus, we are faced with the challenge of producing high-quality WASH projects with a high level of feedback and sustainability on a shoestring budget.
Without the funds for advanced technology and data collection, we are tasked with finding alternative ways to ensure that our WASH projects are meeting these three criteria:
I) Beneficiaries’ needs
II) Industry and international standards
III) Donor expectations
To address each of these criteria in a cost-effective way, we need to craft local, low-technology relationship networks to implement and feed our data and sustainability measures. As an organization of less than ten employees, we depend on the passion, dedication, and involvement of the stakeholders in the communities we work in to be the drivers behind our sustainability and M&E initiatives.
One such program we employ in both Haiti and the Dominican Republic is the Youth Water and Hygiene Club. This type of school-based youth programming has been championed by the WASH sector as an intervention capable of providing youth with leadership training, experiential learning, and an in-depth opportunity to learn and practice water, sanitation, and hygiene solutions firsthand. Our Youth Water and Hygiene Club has been both catalyzing for the participating youth and beneficial to the schools and communities they serve. Students are empowered to be active participants in improving and maintaining the World Water Relief WASH infrastructure in their respective schools and communities. This means helping to clean drinking water stations and hand washing stations, chlorinating potable water holding tanks, initiating trash and recycling collection, teaching WASH principles to student peers, and providing direct monitoring and feedback on WASH service delivery.
The second benefit of a school-integrated program like this is that M&E is conducted on a daily basis at each WASH in Schools site. The Youth Water and Hygiene Clubs provide detailed and dedicated reporting on the status of their schools WASH projects. The World Water Relief program mangers in both the DR and Haiti are in daily communication with the club officers and have frequent regional meetings that feature 82 youth from 16 schools. These meetings provide an excellent opportunity for club leaders to learn from each other and for World Water Relief to continue empowering an inter-connected network of dedicated WASH youth.
The ultimate goal of WASH M&E initiatives is to provide insightful field-level information and analysis that drives accurate and timely project oversight. Ideally, WASH implementers are then able to relay these informative reports to donors and stakeholders in order to prove the efficacy of WASH projects around the world. The rapid progression of technology over the past decades has greatly enhanced the sector’s ability to create and share these important results. However, when we think about sustainability and evaluation, we must remember that data and observation can only take us so far. True sustainability still lies in the hands of the local users and stakeholders.
As the WASH sector moves forward in its pursuit of real-time tracking and evaluation of project efficacy, we mustn’t lose sight of the ability and potential of end-user involvement. Data can inform and guide, but the root of sustainability is still built through long-term relationships, strong personal communication, and direct face-to-face participation.
Editor’s Note: This guest post was authored by Malaika Cheney-Coker, learning and influencing advisor of the Water Team at CARE USA. Malaika reflects on CARE’s recent and first-ever global assessment of its water portfolio, Water+Impact Report: Walking the Talk. The report aims to answer questions such as: What happens after CARE leaves? Is CARE creating the right type of change? What differences has CARE made in the lives of women and girls? It offers a comprehensive meta analysis of 51 different project evaluations from around the world. (The views expressed here are those of the author and do not represent those of CARE.)
Here’s one reason not to do a global impact report: you might not like the results. When the Water Team at CARE headquarters decided to do one, we, were enthusiastic about having a global picture for water performance that hadn’t existed before. That global picture was a mixed bag. Through a fairly subjective review of 51 results-focused documents dating from 2006-2010, we came up with scores: 6 out of 10 for our direct service delivery work, and 4 out of 10 for both our work with policies, institutions, and social norms and for our work in promoting gender-equitable control over water and related resources.
When we presented these findings in-house, our colleagues felt that we had been too self-critical and asked how our work compared with that of others in the sector. Very well indeed, it turned out — at least in 2011 when Philanthropedia ranked CARE International #5 among 116 different nonprofits working in the field of water, sanitation, and hygiene.
So then logical follow-up questions would be: Do we stand by our ranking? (We do.); What does this say of overall performance in the WASH sector? (That there are systemic problems of which the recent sustainability dialogues are one indicator.); And, does a ranking even matter? (Yes and no, I’d contend.)
Let’s take the “yes” answer — that the grade does matter. The grade matters because, quite simply, we can do better. If the effects of our work, or even the infrastructure we set up, were consistently found present and functionial, or even sputtering along, 10 years after project implementation, there would be room to contest the grades. If achieving influence at large scale, rather than in a few hundred communities per country, was routine, the score would be higher. We have decades worth of programming that proves we have a relatively smooth operation of installing infrastructure, organizing committees, and training hygiene promoters. But we want more than this for our work and we know that good development demands more.
What does more look like? Many things is the answer, including the thorny policy and policy implementation, governance, societal norms, and knowledge gaps issues — all of which must be addressed while tackling gender equity and other forms of social injustice (all areas which led to lower scores in our assessment than our direct service delivery work). It is part science, part alchemy; it is part the ingenuity of the humble tippy tap, and part the trick to coax innovations like it into being. Don’t get me wrong, we have work that does all of these things. But if that work was consistent across the board, if the results of the work were permanent, or at least durable, and had the irresistible logic or appeal that makes something go viral, then the score would be higher.
For a look at the “no” answer, the truth is that examples of global impact (of a positive, desirable kind) abound. An impact report can grade itself on such a curve that a satisfying report is assured. But to grade tough with a worthy yardstick is about vigorously seeking change at the cost of self-exposure.
Furthermore, development work is complicated enough and the scale of problems such as depleted natural resources, social oppression, and gender bias is immense enough that even with the most cutting edge programming must strike a balance between ambition and realism. A score implies we know what the ultimate solution to the problem is and can judge our progress relative to it. Development has often come at the expense of the environment, human rights, or the advancement of other nations. We recognize all these things as important but there are wildly varying opinions of the optimum balance of these elements. Even if we were to come to a consensus on such a balance, we all know it would vary by context. And whatever agreements we can cobble together today will not serve us adequately tomorrow as our understanding of development will and must continue to evolve.
In addition, we face limitations in how we design and implement programs. Our funding cycles are not always conducive to investing in long-term results or even allowing for the ability to monitor over the long term. But we must find ways to do so. Measuring impact works muscles of self-inquiry and programming discipline. It illuminates a global picture that exposes the cobwebs in our thinking and programming while also turning up some of the gems in our work. The measurement process itself is a workout, despite the actual score.
In sum, there are reasons why the score matters and at least as many why it doesn’t. So a fair question to ask is, “how did they score themselves?”; a better question to ask is, “how much do they want to change?”
To ensure that WASH projects are successful, it’s critical to apply lessons learned from previous work in the field. Yet it can be challenging to track down relevant evaluations from multiple online and offline sources. In an effort to help our audience make better use of all of the research and knowledge accumulated in the sector, we’ve compiled a list of 20 online evaluation databases that house WASH-related evaluation reports. Some of these databases have a handful of WASH reports, while others have hundreds of reports available.
Take a look and if there’s anything we missed, leave us a note in the comments section below.