Review of Randomistas: How Radical Researchers Changed Our World
Written for the World Bank Impact Blog, May 21, 2018
It isn’t hard to understand why Andrew Leigh would write a book on randomized controlled trials. A kind of modern renaissance man, Leigh currently serves as a member of the Australian House of Representatives. But in his prior life as an economist (Ph.D. from Harvard’s Kennedy School), Leigh published widely in the fields of public finance, labor, health, and political economy, even winning the Economic Society of Australia’s Young Economist Award–a kind of John Bates Clark medal for Australians. His evolution from economist to politician must constantly evoke the following question: What is the best research approach for informing practical policy? In his new book, Leigh leaves little doubt about his answer. Randomistas: How Radical Researchers Changed Our World (forthcoming, Yale University Press) heralds the widespread incorporation of the randomized controlled trial (RCT) into the mainstream of social science.
With respect to RCTs, Leigh is a true believer. A marathon runner, he wears compression socks after races. Why? Because a randomized controlled trial showed that they speed recovery. The justification for his cup of java each morning? A randomized controlled trial showing its protection against DNA breaks. Leigh even rips off Band-Aids extra fast from the skinned knees of his kids because a randomized trial found that the fast rip was less painful than the slow peel. (Note: Perhaps there are some things we don’t need randomized controlled trials to teach us.)
Randomistas is an ambitious book, in no small part because it is a book exclusively written about a scientific methodology. This could be a tough assignment for a writer to pull off. A 278-page exposition on a scientific methodology does not immediately conjure images of a riveting, flashlight-under-the-covers page-turner. But in the end, Leigh’s pithy and engaging prose proves more than up to the task.
Leigh treats us to the history of RCTs as they emerged as a tool of mainstream science, but he emphasizes the interesting findings of new ones. If you are thirsty for RCT results, you will love this book because reading Randomistas is like drinking from an RCT firehose. The book’s fast-paced discourse tends to gloss over wrinkle and nuance, with Leigh clearly concerned that too much academic detail might cause readers to snooze. The RCT results just keep coming as fast as your eyes can digest them, sometimes at a rate of a half-dozen or more per page.
The book is organized around major areas of RCT application to social science and policy: education, crime, social media, and politics. One of the middle chapters highlights the work of JPAL, IPA and development economists running RCTs to understand the impacts of poverty interventions. If you have been running RCTs in development for a while, you may see your work mentioned in this chapter, as is the work of a good number of the economists who organize and contribute to this blog.
Several books over the last years have been reviewed here that have highlighted the leading role of RCTs in development economics including Karlan and Appel’s More Than Good Intentions and Tim Ogden’s excellent book Experimental Conversations. So what does Randomistas offer a development economist with plenty of exposure to the RCT world?
What grabbed my attention the most were the chapters that included engaging reviews of RCTs outside of development economics. In the chapter on RCTs in education, Leigh reviews a series of randomized trials that produce a sequence of results so counter-intuitive to prevailing wisdom that it might deserve its own TV episode on MythBusters. One example: After-school programs not only don’t seem to improve academic outcomes; they appear to lead to significantly worse school behavior and a higher probability of suspension. In short, after-school programs introduce marginal kids to the “wrong crowd,” probably neglected kids whose parents don’t pay enough attention to them.
Leigh’s chapter on controlling crime reviews the randomized trial on the city of Chicago’s creative Becoming a Man program. Guided by a philosophy that “A boy has problems. A man finds solutions to his problems,” the RCT showed BAM reduced arrests by one-third to one-half. I also liked the review of results from the British “Nudge Unit,” which carries out a myriad of low-budget, behavioral economics experiments in the U.K. These RCTs carried out on nudge treatments induce people to pay their taxes, attend job-training seminars, and donate their organs. They are fun to read about, and for development researchers they may stir up the mental broth for running similar experiments overseas.
I found the review of these non-development RCTs useful. Unless we are hardwired to the world of experiments across disciplines, we often miss experimental studies outside our field because they don’t naturally pop up in development seminars or in our Google scholar alerts. The emphasis of the book on experiments involving behavioral issues will appeal to many blog readers who dabble in behavioral economics but are unfamiliar with experimental work in educational, criminal, and political psychology. As a person who tends to eschew the homo-economicus approach to human beings in favor of the more holistic view, this angle of the book sat well with me.
A few little issues: Randomistas is unabashedly pro-RCT, which is fine. Yet it would have been nice to see a little more space devoted to the qualms with RCTs held by some top researchers such as Angus Deaton, whose thoughtful critique earns literally two sentences in the book. The Hawthorne effect, one of the main detractions of RCTs relative to natural experiments, receives half a paragraph. In this light, Randomistas is better understood as a celebration of the breadth and power of the RCT movement across disciplines rather than an impartial reflection on it.
For many general-public readers interested in contemporary social science, Randomistas will take its place on a bookshelf next to The Undercover Economist, Freakonomics, and other popular non-fiction works of the same genre. Although more rigorous in its science, the book will appeal to a slightly more specialized reader than the Gladwell books. For a living, breathing randomista who is actively running RCTs in development economics, sections of Randomistas will overlap with some familiar territory. But for those seeking a larger view of the RCT forest above the nearby trees, Randomistas is a must-read, an engaging chronicle of a movement that has had an inexorable influence on our profession.