Archive for: impact studies

In an August 22 post on DevFinance Dale Adams opined that my CGAP paper on impact “dismissed” expressed demand (also called “revealed preference”) as a way to find out if microcredit helps borrowers:

“In an otherwise excellent note [thanks, Dale!], Rosenberg dismisses expressed-demand as an indicator of the usefulness of microloans: (CGAP, Focus Note No. 59, “Does Microcredit Really Help Poor People?”). This leads him to conclude that more rigorous (and costly) studies are needed to measure the benefits of microloans…. 

My first concern is with the comparison he uses to justify dismissing the votes-people-make-with-their-feet as a measure of the usefulness of loans to borrowers.   To support his claim he says that ‘…repeated use does not by itself prove that a service is benefiting users.  No one would make this argument about repeated use of heroin, for example.’  He goes on to mention that borrowers might be caught in a debt trap and seek loans to keep their heads above water.

Using comparisons, analogies, and parables can be useful (and tricky) in making a point, but equating loans to a habit-forming drug is a stretch.  Does CGAP really want to take the position that microloans are habit forming and therefore dangerous to the health of borrowers?   Even if a tiny percentage of borrowers do fall into debt traps, is it useful to extend the habit-forming analogy to cover all microborrowers? Should we think of the millions of women who take loans from the Grameen Bank as addicts?”

That’s how Dale read it. On the other hand, a May 17 New Yorker article quoted prominent researcher Esther Duflo as saying that our position was “moronic” precisely because it embraced (!) the expressed-demand argument. 

(Were Drs. Adams and Duflos both reading the same paper?) 

I thought the paper expressed a middle position. On the one hand, I used the admittedly extreme example of addictive substances to make the point (a correct one, I think) that you cannot automatically assume that repeated use means something is good for someone.  However, the whole discussion both before and after the sentences that worried Dale was a series of arguments about why we should take the revealed preferences of micro-borrowers (and especially micro-repayers) seriously as evidence that they benefit from their loans. I ended with the point (also correct, I think) that the revealed-preference arguments I was making do not conclusively settle the matter. And that therefore further research would be a good idea.

Rigorous impact studies are fairly expensive–the only on whose price tag I know was high six figures. But I find it a little hard to take this seriously as an argument against more studies.  As a percentage of the mutiple billions that donors and socially-oriented investors have already spent on microfinance, the cost of impact studies is inconsequential. Given how much we still don’t know about what kind of impact all this investment is producing, I think further research makes pretty good sense.

What’s the moral of this little story?  That impact is a really complicated topic? Or maybe just that I should work at writing more clearly?

Richard Rosenberg

How can changes in the lives of microfinance clients best be measured?  How does that differ from tracking impact? Last month I participated in a panel that gathered three MFI practitioners, one social investor, and one researcher from Innovations for Poverty Action (IPA) to find answers to that question.

Anticipating some sparring between the “camps,” I sat between those proposing monitoring tools and non-experimental research as a way to track change, and those supporting impact studies with randomized controlled trials (RCTs) as the way to attribute impact, for example, from a loan or savings product.

Tracking or studying change and measuring impact are fundamentally different things, and there’s a growing schism in microfinance between these two camps. On the one hand, there is a movement composed of social investors, MFIs, management consultants, and researchers that support monitoring and change studies. For this group monitoring indicators, such as those offered by IRIS  or MIX, are sufficient: research sans randomization may give a “good enough” orientation of what microfinance really does for clients. This group also supports monitoring outputs, making diagnostics on how processes affect outputs, and non-invasive academic research that does not need control groups and helps managers to make decisions. They want rigor: but the approach is pragmatic, rather than purist. A list of the various monitoring tools can be found at the Social Impact practice of McKinsey.

The impact supporters, on the other hand, say that impact evaluation with RCTs is all that will actually measure and answer the impact question: there’s simply no other way.  And a significant amount of money and brainpower is going into it. There are currently over 300 RCTs completed or ongoing in the research portfolios of the main RCT researchers: the World Bank , J-Pal,  or IPA. “Tell me what you exactly want to know and we will measure it and give you answers,” the IPA researcher told the practitioners. 

At the heart of this discussion are different interpretations of what impact means and radically different drivers.

The one thing we all agree on is that marketing change or monitoring and evaluation as impact studies is misleading, and simply wrong. The very name “change studies”—coined by an MFI—may reflect a growing awareness and understanding of what monitoring can—and can’t—do, which is good news. But the gulf between the practitioner and the research worlds needs bridging if we’re really going to make progress on such an important discussion.

Meritxell Martinez

(Plus afterthoughts on mission drift and the quality of Bangladeshi microcredit products.)

I recommend reading David Roodman’s blog from last Wednesday about framing a “bottom line” verdict on microcredit.  I was particularly interested in his concluding observations:

…[W]hile high-quality impact studies are valuable, they can never give us the whole story, for each is a static snapshot. (Often, it should be said, of impacts at low doses, because randomized trials are often performed as MFIs roll out services to new customers, which in microcredit means making those small first loans.) Much of the story of the impact of credit lies in the dynamics of the market, how it evolves over time, as we have just seen here in the United States. You don’t understand those through traditional impact studies.

It also means, by the way, that impact studies ought to report doses and impacts with equal prominence.

Some would argue that USAID’s AIMS impact studies did not successfully screen out selection bias.  But my recollection is that the impact that they found (whether truly caused by the microcredit or not) tended to be associated with a series of loans, not just one, let alone the initial MFI loan that is often far below what the client wants and can handle.  That doesn’t mean that we should ignore the results of randomized trials that cover 15 or 18 months of entering clients’ experience.  But it does suggest that longer-term results, if obtainable, should be a lot more significant.

While we’re on the subject of small initial loans…  People tend to see loan size as a rough proxy of client poverty, which appears to be more or less true as long as you say the word “rough” very emphatically.  But the first few loans that an MFI makes to a client typically reflect, not the client’s ability to use and repay the amount, but rather the MFI’s risk management policy (i.e., we’ll give the client more serious money only after she’s established a good track record in repaying little—i.e., low risk—loans.)  This is one of several reasons why it is a serious mistake to view increase of average loan size as ipso facto evidence of mission drift in an MFI.

When Compartamos started out, no client could get an initial loan bigger than $50.  After some years of great loan collection performance, management decided that they could loosen the reins a bit, and give clients the choice of a range of initial loan sizes, from $50 up to several hundred.  Once the new policy was implemented, almost no one chose a $50 loan, and most new clients took the maximum loan size.  This produced a big increase in average loan size, which had nothing to do with the poverty level of the incoming clients.

Microloans per 1000 households (or per 1000 poor households) are a lot higher in Bangladesh than anywhere else.  Why is the country such an outlier, with some other countries appearing to approach market saturation at much lower levels?  And why haven’t we seen more signs of price competition and pressure on profit levels in Bangladesh, given that large percentages of potential clients have access to multiple providers.  My speculation is that both of these things may reflect MFI loan size policies that are not well matched with client needs and repayment capacity.  Anecdotally, one hears that there are very high levels of multiple indebtedness in Bangladesh.  But I’ve seen a couple of studies reporting that in Bangladesh, unlike most other countries, multiple indebtedness has not been strongly correlated with repayment problems.  Maybe the MFIs are handing out loans that are too small, forcing clients into the hassle of going to multiple MFIs to borrow an amount that fits their needs and repayment capacities.  If I want loans from all three MFIs in town, the fact that one of them charges a little more interest than the others is unlikely to deter me.  In an environment like that, one wouldn’t expect to see much price competition.

If any readers have any data bearing on this speculation, I’d be very interested to hear about it.