I thought I had written for the last time in a while on the topic of research, microfinance, and randomized controlled trials (RCTs) when I commented on the recent Nobel Prize in Economics announcement and used that as an opportunity to pull together my previous writings and some other resources on this issue. But sometimes things work out in unexpected and surprising ways.
A critique of RCTs that has gnawed at me over the years, and that I have never raised publicly, is one that some public health professionals first brought to my attention. RCTs were widely used to test new pharmaceuticals long before they were extensively used to evaluate things like microfinance and microcredit. It seems to me that the methodology works best when the input being tested and the environment it is being introduced into are homogenous – especially if one wants to draw generalized lessons about how effective it is. Putting 30 milligrams of some compound into a human body seems like a good example of a test that plays to the methodology’s strengths.
But microcredit (and other microfinance products) are much more heterogeneous than medication. They vary in terms of loan size, repayment periods and flexibility, complementary services, the customer service orientation of the loan officer, group liability and support, and much more. Microcredit provided by organizations with different cultures, policies and practices differs a lot, as even a casual observer can discern. I have even seen that inside a single organization, microcredit can look quite different from one branch to another, or even one loan officer to another (despite all the efforts to standardize products).
So, to go into a small number of second- and third-rate microfinance organizations, measure one or two indicators of impact of their version of microcredit over a short period of time, and then to make generalized claims about how well this social innovation works (or, worse, how it compares unfavorably to some mythic standard of transforming people and communities in a matter of months) has always seemed highly questionable to me. (And it is one of the reasons that I always encouraged people to view those results alongside many others distilled by alternative research methodologies, all with their own pros and cons, such as in this meta-analysis written by Professor Kathleen Odell.)
It turns out that microfinance is not the only social movement that has dealt with RCTs and how their conclusions have been overhyped and overgeneralized, leading on occasion to perverse outcomes.
A bit of background is in order. As I explain in my book Changing the World Without Losing Your Mind, for the last 10+ years I have consciously tried to do at least one thing in a fairly serious way that I am a novice or beginner at. One of those has been volunteering for the Court Appointed Special Advocate program in Prince George’s County as a way to serve children in the foster care system. (The CASA program has chapters nationwide and I highly recommend getting involved.)
As a part of going beyond novice status in this volunteer role, I asked experts in this field to suggest books to read. One was Trauma and Recovery: The Aftermath of Violence—From Domestic Abuse to Political Terror by Judith Herman (which the New York Times called “One of the most important psychiatric works to be published since Freud” and which I found accessible to a curious layperson not well versed in psychology).
As I was finishing the 2015 Afterword by the author, I unexpectedly came upon a critique of RCTs. First, she expressed dismay about the extent to which RCTs had treated psychotherapy as a standard product that could be evaluated like medication. The first mention (on page 266) went straight to this point: “The scientific ‘gold standard’ of clinical research is the random controlled trials (RCT) in which an identical dose of a particular treatment is compared with a placebo or with another treatment for the same condition. The RCT design works quite well for drug studies, but it is a poor fit for psychotherapy research because psychotherapy is not a pill.” [Emphasis added.]
She continues: “The RCT design … requires highly standardized outcome measures. This leads to a narrow focus on symptom reduction.” The author then explains how something called cognitive-behavioral therapy (CBT) was tested extensively since it lent itself so well to the RCT methodology. For a time CBT was overhyped as the most “evidence-based” treatment for victims of trauma as a result of RCTs measuring success in its reduction of a single symptom. Yet many psychologists, including the author, rebelled against this vastly oversimplified conclusion and its dangerous public health implications.
She returns to RCTs and their limitations one more time on page 273, and calls for a change in the research agenda that had a familiar ring to me. “As I and many others have argued,” she wrote, “psychotherapy is more craft than science, but it can certainly be studied scientifically. New and different scientific approaches are needed, however. By now it is well established that one of the most important ‘active ingredients’ in psychotherapy is the therapeutic alliance.
“Therefore, rather than seeking to eliminate the individuality of the therapist and patient, as is done in a randomized controlled trial, a good starting point might be to study the common attributes of gifted therapists of different technical schools, the master craftsmen and women of our profession.”
This final recommendation certainly echoes my own calls for using research more to improve microcredit and microfinance than to “prove” whether or not they work, especially since it is obvious that convenient and affordable financial services are helpful to all people, especially the poor – a point made convincingly in David Roodman’s book Due Diligence, and elsewhere.
It also recalls my bewilderment, expressed periodically in writing, about the randomistas never studying microcredit in Bangladesh, where many of the master craftsmen and women of the microfinance field have operated for decades. If the researchers’ goal had been to improve practice and outcomes for the poor, I would have thought that Bangladesh would have been the first place they would have gone to conduct their research.