Here’s one reason not to do a global impact report: you might not like the results. When the Water Team at CARE headquarters decided to do one, we, were enthusiastic about having a global picture for water performance that hadn’t existed before. That global picture was a mixed bag. Through a fairly subjective review of 51 results-focused documents dating from 2006-2010, we came up with scores: 6 out of 10 for our direct service delivery work, and 4 out of 10 for both our work with policies, institutions, and social norms and for our work in promoting gender-equitable control over water and related resources.
When we presented these findings in-house, our colleagues felt that we had been too self-critical and asked how our work compared with that of others in the sector. Very well indeed, it turned out — at least in 2011 when Philanthropedia ranked CARE International #5 among 116 different nonprofits working in the field of water, sanitation, and hygiene.
So then logical follow-up questions would be: Do we stand by our ranking? (We do.); What does this say of overall performance in the WASH sector? (That there are systemic problems of which the recent sustainability dialogues are one indicator.); And, does a ranking even matter? (Yes and no, I’d contend.)
Let’s take the “yes” answer — that the grade does matter. The grade matters because, quite simply, we can do better. If the effects of our work, or even the infrastructure we set up, were consistently found present and functionial, or even sputtering along, 10 years after project implementation, there would be room to contest the grades. If achieving influence at large scale, rather than in a few hundred communities per country, was routine, the score would be higher. We have decades worth of programming that proves we have a relatively smooth operation of installing infrastructure, organizing committees, and training hygiene promoters. But we want more than this for our work and we know that good development demands more.
What does more look like? Many things is the answer, including the thorny policy and policy implementation, governance, societal norms, and knowledge gaps issues — all of which must be addressed while tackling gender equity and other forms of social injustice (all areas which led to lower scores in our assessment than our direct service delivery work). It is part science, part alchemy; it is part the ingenuity of the humble tippy tap, and part the trick to coax innovations like it into being. Don’t get me wrong, we have work that does all of these things. But if that work was consistent across the board, if the results of the work were permanent, or at least durable, and had the irresistible logic or appeal that makes something go viral, then the score would be higher.
For a look at the “no” answer, the truth is that examples of global impact (of a positive, desirable kind) abound. An impact report can grade itself on such a curve that a satisfying report is assured. But to grade tough with a worthy yardstick is about vigorously seeking change at the cost of self-exposure.
Furthermore, development work is complicated enough and the scale of problems such as depleted natural resources, social oppression, and gender bias is immense enough that even with the most cutting edge programming must strike a balance between ambition and realism. A score implies we know what the ultimate solution to the problem is and can judge our progress relative to it. Development has often come at the expense of the environment, human rights, or the advancement of other nations. We recognize all these things as important but there are wildly varying opinions of the optimum balance of these elements. Even if we were to come to a consensus on such a balance, we all know it would vary by context. And whatever agreements we can cobble together today will not serve us adequately tomorrow as our understanding of development will and must continue to evolve.
In addition, we face limitations in how we design and implement programs. Our funding cycles are not always conducive to investing in long-term results or even allowing for the ability to monitor over the long term. But we must find ways to do so. Measuring impact works muscles of self-inquiry and programming discipline. It illuminates a global picture that exposes the cobwebs in our thinking and programming while also turning up some of the gems in our work. The measurement process itself is a workout, despite the actual score.
In sum, there are reasons why the score matters and at least as many why it doesn’t. So a fair question to ask is, “how did they score themselves?”; a better question to ask is, “how much do they want to change?”