Archive for: measuring impact

In an August 22 post on DevFinance Dale Adams opined that my CGAP paper on impact “dismissed” expressed demand (also called “revealed preference”) as a way to find out if microcredit helps borrowers:

“In an otherwise excellent note [thanks, Dale!], Rosenberg dismisses expressed-demand as an indicator of the usefulness of microloans: (CGAP, Focus Note No. 59, “Does Microcredit Really Help Poor People?”). This leads him to conclude that more rigorous (and costly) studies are needed to measure the benefits of microloans…. 

My first concern is with the comparison he uses to justify dismissing the votes-people-make-with-their-feet as a measure of the usefulness of loans to borrowers.   To support his claim he says that ‘…repeated use does not by itself prove that a service is benefiting users.  No one would make this argument about repeated use of heroin, for example.’  He goes on to mention that borrowers might be caught in a debt trap and seek loans to keep their heads above water.

Using comparisons, analogies, and parables can be useful (and tricky) in making a point, but equating loans to a habit-forming drug is a stretch.  Does CGAP really want to take the position that microloans are habit forming and therefore dangerous to the health of borrowers?   Even if a tiny percentage of borrowers do fall into debt traps, is it useful to extend the habit-forming analogy to cover all microborrowers? Should we think of the millions of women who take loans from the Grameen Bank as addicts?”

That’s how Dale read it. On the other hand, a May 17 New Yorker article quoted prominent researcher Esther Duflo as saying that our position was “moronic” precisely because it embraced (!) the expressed-demand argument. 

(Were Drs. Adams and Duflos both reading the same paper?) 

I thought the paper expressed a middle position. On the one hand, I used the admittedly extreme example of addictive substances to make the point (a correct one, I think) that you cannot automatically assume that repeated use means something is good for someone.  However, the whole discussion both before and after the sentences that worried Dale was a series of arguments about why we should take the revealed preferences of micro-borrowers (and especially micro-repayers) seriously as evidence that they benefit from their loans. I ended with the point (also correct, I think) that the revealed-preference arguments I was making do not conclusively settle the matter. And that therefore further research would be a good idea.

Rigorous impact studies are fairly expensive–the only on whose price tag I know was high six figures. But I find it a little hard to take this seriously as an argument against more studies.  As a percentage of the mutiple billions that donors and socially-oriented investors have already spent on microfinance, the cost of impact studies is inconsequential. Given how much we still don’t know about what kind of impact all this investment is producing, I think further research makes pretty good sense.

What’s the moral of this little story?  That impact is a really complicated topic? Or maybe just that I should work at writing more clearly?

Richard Rosenberg

How can changes in the lives of microfinance clients best be measured?  How does that differ from tracking impact? Last month I participated in a panel that gathered three MFI practitioners, one social investor, and one researcher from Innovations for Poverty Action (IPA) to find answers to that question.

Anticipating some sparring between the “camps,” I sat between those proposing monitoring tools and non-experimental research as a way to track change, and those supporting impact studies with randomized controlled trials (RCTs) as the way to attribute impact, for example, from a loan or savings product.

Tracking or studying change and measuring impact are fundamentally different things, and there’s a growing schism in microfinance between these two camps. On the one hand, there is a movement composed of social investors, MFIs, management consultants, and researchers that support monitoring and change studies. For this group monitoring indicators, such as those offered by IRIS  or MIX, are sufficient: research sans randomization may give a “good enough” orientation of what microfinance really does for clients. This group also supports monitoring outputs, making diagnostics on how processes affect outputs, and non-invasive academic research that does not need control groups and helps managers to make decisions. They want rigor: but the approach is pragmatic, rather than purist. A list of the various monitoring tools can be found at the Social Impact practice of McKinsey.

The impact supporters, on the other hand, say that impact evaluation with RCTs is all that will actually measure and answer the impact question: there’s simply no other way.  And a significant amount of money and brainpower is going into it. There are currently over 300 RCTs completed or ongoing in the research portfolios of the main RCT researchers: the World Bank , J-Pal,  or IPA. “Tell me what you exactly want to know and we will measure it and give you answers,” the IPA researcher told the practitioners. 

At the heart of this discussion are different interpretations of what impact means and radically different drivers.

The one thing we all agree on is that marketing change or monitoring and evaluation as impact studies is misleading, and simply wrong. The very name “change studies”—coined by an MFI—may reflect a growing awareness and understanding of what monitoring can—and can’t—do, which is good news. But the gulf between the practitioner and the research worlds needs bridging if we’re really going to make progress on such an important discussion.

Meritxell Martinez