Is My NGO Having a Positive Impact?

I enjoy helping NGOs with designing ways to carry out impact evaluations within the context of their normal operations.  David Evans of the World Bank and I wrote a piece for the World Bank Impact Blog that suggests a few simple ways that NGO practitioners without lots of formal training in statistics can build impact evaluation into their poverty work:

A daunting question faced by many non-government organizations (NGOs) involved in poverty work is—after all the fundraising, logistical work, direct work with the poor, and accounting is all done—one naturally wonders is my NGO having a positive impact? This blog post is intended for development practitioners, specifically those of you without an overload of formal training in economics or statistics, but who want to carry out simple, but valid impact studies on your work.

Most NGOs are good at measuring outcomes on their beneficiaries.  This is a step in the right direction. But to really measure impact, we need something in addition to the outcomes of beneficiaries.  The key to carrying out a valid impact study is the generation of a counterfactual, or what would have happened to your beneficiaries if your program had not existed when they (hopefully) benefited from it.  Valid impact studies are essentially all about generating valid counterfactuals.

Let’s think about two common ways that some NGOs try to estimate program impact. One way is by taking baseline data on these program beneficiaries and then collecting more data on them after the intervention.  The assumption here is that the counterfactual can be represented by the baseline data on the beneficiaries.  But the problem here is that changes created by the program are often confused with changes that would have affected the program beneficiary over time anyway.  For example, the general economic climate might have improved since the baseline study.  Then much of what you would attribute to impact is merely a rising tide lifting all boats.  In this case you would have overstated the impact of the organization’s work.  Conversely, things in general may have gotten a lot worse, in which case you would understate it.  Moreover, sometimes people choose to become involved with an NGO when they face certain opportunities, like when the timing of an economic opportunity drives someone to take a microfinance loan.  Here the borrower would have been at least somewhat better off whether or not the non-profit was there to lend a helping hand.  Indeed recent research finds that about three-quarters of apparent microfinance impact is an optical illusion from before-and-after observations.  So before-and-after studies don’t generate a valid counterfactual.

Other NGOs sometimes measure impact by comparing program beneficiaries to non-beneficiaries.  The assumption here is that a program non-beneficiary represents a valid counterfactual to a beneficiary.  But this doesn’t establish a valid counterfactual either.  Non-beneficiaries may lack the hidden qualities that induce a beneficiary to be part of the program.  These hidden qualities may be positively correlated with self-selection (in which you would overestimate program impact) or negatively correlated—like when someone comes to the NGO in a time of crisis—in which you would underestimate impact.  But either way you are likely to “mis-estimate” it by trying to make these comparisons.

Indeed everything that we observe—before-and-after differences or differences between beneficiaries and non-beneficiaries consists of the true impact and the bias in our observation.  We can see this easily by both subtracting and adding the counterfactual to this difference as seen in the diagram below.

So how do you generate valid counterfactuals?  This is the trick behind all good impact research…  (To read the full article on the World Bank blog site, click here.)

• ← Why Is ISIS So Bad?
• A Letter to My Evangelical Friends about Donald Trump →