What’s Missing in Vox’s Negative TOMS Piece
Last Thursday, Vox ran a story highlighting our randomized trial with TOMS Shoes subtly titled “Buying TOMS Shoes is a Terrible Way to Help People.” In the piece the author, Amanda Taub, accurately relayed some of the major findings from our experiment: no discernable impact on school attendance, overall health, or foot health. Children happy with shoes, but were also somewhat more likely to indicate a dependence on outsiders for aid. 
But in its reach for maximum shock value, the story missed some key points. First, to expect life-changing impacts from an intervention costing only a few dollars is a mistake. Truly life changing interventions (such as GiveDirectly’s large cash transfers, clean water + hygiene programs, or long-term child sponsorship) are usually very costly. And it was a mistake for TOMS to claim or imply that shoe donations had life-changing impacts on average. But it was also a mistake for the media to expect that the impacts of shoe donations should exceed that of a child being reasonably happy with a new pair of inexpensive shoes.
But the greater problem is that the shoe impact results aren’t the end of the story. The Vox piece misses the fact that people and organizations everywhere are learning how to “do development.” Effective development is difficult. Outcomes often violate our expectations. So just as important as measuring the impact of an intervention is a commitment to 1) transparency and 2) learning. In other words, how an organization assesses its impacts, is transparent about the results, learns from its experience, and adjusts to this information is probably more important than a static measure of impact, especially early impact.
In the case of TOMS, here is a situation in which a rapidly growing company genuinely wants to understand the impact of its philanthropic arm. So it funds research that allows outside academics to carry out an impact study on its shoe-giving program. Then it finds that, at least in a certain context, a new pair of shoes doesn’t appear to have life-changing impacts. So it learns. It re-doubles its effort to target shoe donations in places where they are more likely to have a bigger effect. It develops new types of shoes to give away that are better adapted to context. It diversifies into new products that in themselves are more likely to have more substantial impacts: sunglasses (tied to provide vision correction to the poor overseas), coffee (providing clean water), purses (skilled birth attendants), school backpacks (anti-bullying programs).
Not every NGO or social business is willing to undergo this kind of rigorous and transparent self-assessment. For example, in the face of academic studies showing negligible impacts from fair trade coffee and flaws in its system that violate the most basic laws of economics, FairTrade USA keeps chugging along with the same ineffective approach to raising coffee grower incomes. It’s OK to be wrong, because effective poverty intervention is tricky. But it’s not OK these days to keep on being wrong. And to TOMS’ credit, they are not content with being wrong for long.
If we wrote off every major development organization for interventions that have later been proven ineffective, the remaining list would be small. What I and many other development economists are more concerned about today is that those trying to do relief and development among the poor (a) rigorously assess the impact of their work; (b) behave transparently and honestly about their effectiveness; (c) learn from their mistakes; and (d) adjust in the direction of greater impact. TOMS is trying to do that. Hopefully others will join them.