Archive for: Impact

In an August 22 post on DevFinance Dale Adams opined that my CGAP paper on impact “dismissed” expressed demand (also called “revealed preference”) as a way to find out if microcredit helps borrowers:

“In an otherwise excellent note [thanks, Dale!], Rosenberg dismisses expressed-demand as an indicator of the usefulness of microloans: (CGAP, Focus Note No. 59, “Does Microcredit Really Help Poor People?”). This leads him to conclude that more rigorous (and costly) studies are needed to measure the benefits of microloans…. 

My first concern is with the comparison he uses to justify dismissing the votes-people-make-with-their-feet as a measure of the usefulness of loans to borrowers.   To support his claim he says that ‘…repeated use does not by itself prove that a service is benefiting users.  No one would make this argument about repeated use of heroin, for example.’  He goes on to mention that borrowers might be caught in a debt trap and seek loans to keep their heads above water.

Using comparisons, analogies, and parables can be useful (and tricky) in making a point, but equating loans to a habit-forming drug is a stretch.  Does CGAP really want to take the position that microloans are habit forming and therefore dangerous to the health of borrowers?   Even if a tiny percentage of borrowers do fall into debt traps, is it useful to extend the habit-forming analogy to cover all microborrowers? Should we think of the millions of women who take loans from the Grameen Bank as addicts?”

That’s how Dale read it. On the other hand, a May 17 New Yorker article quoted prominent researcher Esther Duflo as saying that our position was “moronic” precisely because it embraced (!) the expressed-demand argument. 

(Were Drs. Adams and Duflos both reading the same paper?) 

I thought the paper expressed a middle position. On the one hand, I used the admittedly extreme example of addictive substances to make the point (a correct one, I think) that you cannot automatically assume that repeated use means something is good for someone.  However, the whole discussion both before and after the sentences that worried Dale was a series of arguments about why we should take the revealed preferences of micro-borrowers (and especially micro-repayers) seriously as evidence that they benefit from their loans. I ended with the point (also correct, I think) that the revealed-preference arguments I was making do not conclusively settle the matter. And that therefore further research would be a good idea.

Rigorous impact studies are fairly expensive–the only on whose price tag I know was high six figures. But I find it a little hard to take this seriously as an argument against more studies.  As a percentage of the mutiple billions that donors and socially-oriented investors have already spent on microfinance, the cost of impact studies is inconsequential. Given how much we still don’t know about what kind of impact all this investment is producing, I think further research makes pretty good sense.

What’s the moral of this little story?  That impact is a really complicated topic? Or maybe just that I should work at writing more clearly?

Richard Rosenberg

How can changes in the lives of microfinance clients best be measured?  How does that differ from tracking impact? Last month I participated in a panel that gathered three MFI practitioners, one social investor, and one researcher from Innovations for Poverty Action (IPA) to find answers to that question.

Anticipating some sparring between the “camps,” I sat between those proposing monitoring tools and non-experimental research as a way to track change, and those supporting impact studies with randomized controlled trials (RCTs) as the way to attribute impact, for example, from a loan or savings product.

Tracking or studying change and measuring impact are fundamentally different things, and there’s a growing schism in microfinance between these two camps. On the one hand, there is a movement composed of social investors, MFIs, management consultants, and researchers that support monitoring and change studies. For this group monitoring indicators, such as those offered by IRIS  or MIX, are sufficient: research sans randomization may give a “good enough” orientation of what microfinance really does for clients. This group also supports monitoring outputs, making diagnostics on how processes affect outputs, and non-invasive academic research that does not need control groups and helps managers to make decisions. They want rigor: but the approach is pragmatic, rather than purist. A list of the various monitoring tools can be found at the Social Impact practice of McKinsey.

The impact supporters, on the other hand, say that impact evaluation with RCTs is all that will actually measure and answer the impact question: there’s simply no other way.  And a significant amount of money and brainpower is going into it. There are currently over 300 RCTs completed or ongoing in the research portfolios of the main RCT researchers: the World Bank , J-Pal,  or IPA. “Tell me what you exactly want to know and we will measure it and give you answers,” the IPA researcher told the practitioners. 

At the heart of this discussion are different interpretations of what impact means and radically different drivers.

The one thing we all agree on is that marketing change or monitoring and evaluation as impact studies is misleading, and simply wrong. The very name “change studies”—coined by an MFI—may reflect a growing awareness and understanding of what monitoring can—and can’t—do, which is good news. But the gulf between the practitioner and the research worlds needs bridging if we’re really going to make progress on such an important discussion.

Meritxell Martinez

(Plus afterthoughts on mission drift and the quality of Bangladeshi microcredit products.)

I recommend reading David Roodman’s blog from last Wednesday about framing a “bottom line” verdict on microcredit.  I was particularly interested in his concluding observations:

…[W]hile high-quality impact studies are valuable, they can never give us the whole story, for each is a static snapshot. (Often, it should be said, of impacts at low doses, because randomized trials are often performed as MFIs roll out services to new customers, which in microcredit means making those small first loans.) Much of the story of the impact of credit lies in the dynamics of the market, how it evolves over time, as we have just seen here in the United States. You don’t understand those through traditional impact studies.

It also means, by the way, that impact studies ought to report doses and impacts with equal prominence.

Some would argue that USAID’s AIMS impact studies did not successfully screen out selection bias.  But my recollection is that the impact that they found (whether truly caused by the microcredit or not) tended to be associated with a series of loans, not just one, let alone the initial MFI loan that is often far below what the client wants and can handle.  That doesn’t mean that we should ignore the results of randomized trials that cover 15 or 18 months of entering clients’ experience.  But it does suggest that longer-term results, if obtainable, should be a lot more significant.

While we’re on the subject of small initial loans…  People tend to see loan size as a rough proxy of client poverty, which appears to be more or less true as long as you say the word “rough” very emphatically.  But the first few loans that an MFI makes to a client typically reflect, not the client’s ability to use and repay the amount, but rather the MFI’s risk management policy (i.e., we’ll give the client more serious money only after she’s established a good track record in repaying little—i.e., low risk—loans.)  This is one of several reasons why it is a serious mistake to view increase of average loan size as ipso facto evidence of mission drift in an MFI.

When Compartamos started out, no client could get an initial loan bigger than $50.  After some years of great loan collection performance, management decided that they could loosen the reins a bit, and give clients the choice of a range of initial loan sizes, from $50 up to several hundred.  Once the new policy was implemented, almost no one chose a $50 loan, and most new clients took the maximum loan size.  This produced a big increase in average loan size, which had nothing to do with the poverty level of the incoming clients.

Microloans per 1000 households (or per 1000 poor households) are a lot higher in Bangladesh than anywhere else.  Why is the country such an outlier, with some other countries appearing to approach market saturation at much lower levels?  And why haven’t we seen more signs of price competition and pressure on profit levels in Bangladesh, given that large percentages of potential clients have access to multiple providers.  My speculation is that both of these things may reflect MFI loan size policies that are not well matched with client needs and repayment capacity.  Anecdotally, one hears that there are very high levels of multiple indebtedness in Bangladesh.  But I’ve seen a couple of studies reporting that in Bangladesh, unlike most other countries, multiple indebtedness has not been strongly correlated with repayment problems.  Maybe the MFIs are handing out loans that are too small, forcing clients into the hassle of going to multiple MFIs to borrow an amount that fits their needs and repayment capacities.  If I want loans from all three MFIs in town, the fact that one of them charges a little more interest than the others is unlikely to deter me.  In an environment like that, one wouldn’t expect to see much price competition.

If any readers have any data bearing on this speculation, I’d be very interested to hear about it.

To the Editor of the New Yorker

Dear Sir,

As I read the excellent May 17 article  on Esther Duflo and her randomized evaluations of social policy tools, I was surprised to find CGAP (a World-Bank affiliated research and policy center for microfinance) lumped together with the microfinance networks that have been discounting the results of recent studies by her and other randomistas. On the CGAP blog, I have been criticizing—tactfully, I hope, but clearly—the defensive reaction of these networks. Far from discounting randomized trials, CGAP has been supporting them to the tune of $1.2 million of our own money and funds we raised from our members.

Prof Duflo dismissed a line I had written (Does Microfinance Improve Their Lives? The Poor Say Yes) as “the moronic revealed-preference argument,” that is, the argument that if people keep using something it must be good for them. The offending line occurred in a position paper I wrote  for CGAP. It was the title of a section that included the statement: “Of course, repeated use does not by itself prove that a service is benefitting users. No one would make this argument about repeated use of heroin, for instance. People do not always borrow wisely.” 

The CGAP paper clearly acknowledges that randomized studies by Prof. Duflo and others have raised fundamental questions about the claim that microcredit raises incomes and consumption, lifting poor people out of poverty—so much so that funding solely on the strength of that claim would not be justified at present. The paper then argued that there were reasonable (not conclusive) grounds to think that microfinance at least helps poor people to cope with poverty, for instance by smoothing irregular consumption and cushioning against shocks.  Consumption gaps and uncontrollable calamities are serious problems for poor people. They place a high value on financial tools that help them cope with these problems. And well-executed microfinance can create such tools at very low cost in terms of public subsidy. Finally, the paper recognized that these arguments need to be tested by further research.

Richard Rosenberg
Senior Adviser, CGAP

After reading Neil MacFarquhar’s article “Banks Making Big Profits from Tiny Loans”, I was shocked to hear about how exorbitant are the interest rates that some microfinance institutions (MFIs) around the world are charging their borrowers and saddened to hear that the focus of these institutions has clearly drifted from helping the poor.  However, in my experience, founding Wokai, a China-focused microfinance organization (www.wokai.org), I have to point out that high interest rates is not a universal problem.  In fact, China is facing the opposite issue: interest rates charged by MFIs are too low.

Read the rest of this page »

With more and more ready-to-use assessment tools out there, such as poverty scorecards, and an increasing interest in randomized control trials to measure the impact of financial services on the poor, it’s easy for MFIs to get lost in the process of choosing the right tools. In recent weeks, I’ve seen a number of instances of MFIs misinterpreting the data generated by some of the tools, ranging from interpreting poverty data as demand information, to using the change in the probability of being poor of a client as a metric of impact on the client, to distilling product development strategies out of indicators that can’t tell much about clients’ financial products needs.

MFIs may want more information about their clients’ needs and behavior, but using the right tool and making it part of its processes requires human and financial resources. Hiring internal researchers is often seen as an unnecessary expense, while having external people come in also imposes demands on already-stretched staff time .

It’s all well and good for donors to require their investees and grant recipients to measure results. After all, donors need to make sure that their investees are accountable, and that the research on what works and what doesn’t work in financial access expands. But from what I’m observing in the field, MFIs will need a lot more guidance and training on how to make important monitoring and evaluation choices and how to use the tools and the information that pops out if they are to prove useful, and not just a big waste of time and money.

A recent blog by Chris Dunford and colleagues at Freedom from Hunger–Different Levels of “Knowing” the Impact of Microfinance–struck me as balanced and insightful.  It recognized, with a refreshing lack of defensiveness, that evidence from randomized controlled trials is essential (not just useful) for testing claims about the effects of microfinance.  At the same time it insisted—correctly, I think—that practitioner experience during years of work with poor clients constitutes real knowledge.

Maybe the question is “knowledge for whom?” For instance, my own 27 years of unsystematic experience with a variety of development funding agencies have left me with some conclusions regarding them that I feel very confident about—confident enough to base career decisions on those conclusions. But it’s quite another matter to communicate that knowledge publicly in a way that justifies others who haven’t shared my experience to be equally confident about those conclusions, or to accept my conclusions rather than differing conclusions that someone else might be offering. That’s why we need social sciences.

On another subject, Chris recognizes that in the real world individual stories raise more funding than statistics do. So none of us should complain if fund-raising brochures are long on anecdotes and short on regression analyses.  At the same time, I think Chris would agree that stories which may be appropriate in a fund-raising brochure are much more problematic when they are trotted out as evidence in a serious discussion about how we can know whether microfinance is effective or not.

Six practitioner networks published a statement last week in an effort to counteract negative publicity arising from recent randomized trials of the impact of microcredit.  These studies failed to find evidence that microcredit was producing gains in household income or consumption, at least over the short term.

Much of the networks’ statement was reasonable, but I thought it failed to hold water on a few key points, which have been ably analyzed in posts by David Roodman of CGD  and Sushmita Meka of IFMR.

Read the rest of this page »

A journalist from Jeune Afrique sits in the front of the room and derides the audience made up of donors, practitioners, and communications specialists.  He says that when he receives press releases from development agencies, they only announce a high level conference or the agency’s  head attending a ribbon cutting ceremony.  He finds most of the language in the press releases incomprehensible and throws them in the bin.  A journalist from Radio France Internationale follows and recounts her preparation for this event.  Searching last year’s press releases, she found only one that announced any development results.

We are all gathered in Tunis to think about communicating development results. The discussion inevitably takes us to the question, “What results?…do we have any to communicate?”

Read the rest of this page »

The Center for Global Development’s David Roodman and NYU economist Jonathan Morduch have just published a working paper  that revisits impact data based on household surveys  from Bangladesh in the 1990s. And it makes pretty sobering reading.

Read the rest of this page »