Five Reasons Why Development NGOs Should Invest in the Tools of Modern Impact Evaluation
Why should a development NGO invest in the modern tools of impact evaluation? I’ve met with many Christian development NGOs who are eager to understand the real effects of their program on participants, but other NGOs are hesitant. The reasons for not wanting to take this leap seem to range from anxiety to overconfidence. The anxiety issue is often related to a fear that such an exercise might show negative results (and then what do we do?). The overconfidence excuse is often stated in the form of “Well we know we’re having an impact, so why should we waste resources that could be used in program to simply confirm what we already know to be true?”
But when it comes to self-evaluation, we can never fully trust our own judgment. There are many reasons for this. First, we all suffer from confirmation bias, the desire to have our core beliefs validated by accepting evidence that confirms our beliefs while rejecting evidence that challenges them. This happens constantly and subconsciously with our perception of our own work. Like the rest of us, development practitioners add to this through something called pareidolia, a phenomenon where our brains search for patterns that help us make logical sense of the world. In development practice, these often become patterns in participant outcomes that we conform to tell ourselves what we want to believe. In evaluating our own work, we also are quick to attribute people’s improvement to our own efforts, ignoring basic statistical phenomena such as mean-reversion: The reality is that down times usually get better for people, irrespective of our wonderful interventions. A physician told me in a candid moment that he estimates 80% of his patients would get better without him. Our gut should never substitute for serious program evaluation.
Many NGOs are content with monitoring and evaluation staff making sure programs are happening, that resources aren’t being wasted, and that program participants seem to be doing well. But this kind of activity is not to be confused with impact evaluation. There are new tools that have been developed to estimate real impacts on program participants that NGOs need to learn to embrace–and indeed are often now forced to by major donors. These techniques now used in modern program evaluation were the subject of the 2019 and 2021 Nobel Prizes in Economics–an elegant set of statistical evaluation tools that are not too difficult to learn when guided by a good instructor. Here are five reasons why every NGO should be training staff to use these tools as part of their good work:
1) Running development programs without continual and rigorous impact evaluation is like driving a car with mud-caked windshield, no mirrors, and no speedometer. In this unfortunate situation, you have little idea where you’re going, how quickly, or even how to get where you want to go. Similarly, in moments of quiet honesty, many Christian NGOs today have very little idea of the impact they are realizing on their program participants. Are people growing spiritually? Are they healthier? Less poor? But donors keep donating and participants seem pretty happy overall, so the cycle continues. And the happiest participants are then what the NGO reflects back to donors to illustrate “impact”. Implementing modern tools of program evaluation and even building these tools into programming means that Christian NGOs have access to a full array of genuine causal effects of their programs. And these effects on program participants can be measured across the multiple dimensions of biblical human flourishing: physical, spiritual, social inclusion, psychological, and economic well being.
2) Modern causal impact tools help NGOs understand whom they are helping most and least. Is it women or men in the program who benefit most from microloans, the ultra-poor or the entrepreneurial poor? Is it the most disadvantaged schools that are benefiting from a school-based intervention, the less-disadvantaged, or those in the middle? Among which groups is a particular approach to evangelism most effective at stirring interest in the gospel? Is a Christian discipleship program fostering real, inward change more among rural or urban youth? Understanding heterogeneous program effects allows NGOs to a) identify areas of weakness and strength in programming; and b) focus their resources on populations of greatest impact.
3) Modern impact evaluation tools generate policy results that allow NGOs to alter their program mix and content. Basic training in running small internal experiments is critical for creating a learning organization that is continuing to refine and shape programs for continually improving effectiveness. Even today many NGOs simply monitor outcomes on participants or use gut instincts to inform them about what mix of program components are most and least effective. Contrast this with the kind of serious, rigorous research that occurs in modern medicine to improve pharmaceuticals and health interventions. Yet most development work is just as important to livelihood and lives. Building rigorous study into routine program operations is possible, but it begins with adequate training for monitoring and evaluation staff.
4) Rigorous training in impact evaluation bridges the information divide. A vast canyon exists today between the mountain of good research that has shown what is and isn’t effective at helping people, and the awareness of this body of work among development practitioners. And there is a lack of information flow in the opposite direction too–academic researchers are often in the dark about both promising new innovations being tried by practitioners as well as the myriad challenges practitioners face on the ground. The best way to create a pipeline for two-way learning between development research and development practice is to train practitioners to understand modern program evaluation. This sets the table for fruitful partnerships to emerge between researchers and practitioners. My own involvement with organizations such as Compassion International, Hope Walks, Covenant World Relief, Operation Smile, and the Heifer Project has been edifying for all parties and resulted in third-party evidence of impact and better outcomes among program participants.
5) Donors now demand it. Today it is nearly impossible to secure major donor funding without credible evidence of impact. And today credible evidence of impact is defined as impact that has been assessed through the modern experimental and quasi-experimental methods. In the past, smaller donors have been swayed by carefully curated anecdotes, but appeal solely to the heart and not to the head is waning as donors become increasingly sophisticated. Educated donors today understand that it is no more ethical to donate to a poverty organization that lacks credible evidence of its positive impact than it is to sell food and drugs that lack FDA approval. Donors today are more eager to give than ever–but they rightfully demand the kind of credible evidence that is generated by modern research methods.
The key to all of this is the equipping monitoring and evaluation staff and a general understanding and appreciation of these methods by program directors. The mission of CEIDS (the Collaborative for Econometrics and Integrated Development Studies) based at the University of Notre Dame, is to equip, evaluate, and encourage the relief and development arms of the global church. We have developed an 11-month program in Social Impact Analytics that is designed to bring these modern impact evaluation tools into the hands of Christian NGOs. The purpose is to help Christian NGOs carry out the most effective programming possible. We are enrolling our first cohort in January 2025 through a partnership with the University of San Francisco, and want to involve as many as possible in this effort. Some scholarship money has been made available specifically for Christian NGO practitioners, so please read more about the program on the website and consider enrolling staff before the November 1st deadline.
Bruce Wydick is Professor of Economics at the University of San Francisco, Visiting Professor at the University of California at Davis, and Research Affiliate with the Kellogg Institute at the University of Notre Dame. He can be reached by email at wydick@usfca.edu.