Archive for: Innovations for Poverty Action

David Freedman in the November issue of The Atlantic  profiles John Ioannidis, one of the world’s most respected and sought-after experts on the credibility of medical research.

Ioannidis argues that much of what biomedical researchers conclude in published studies is misleading, exaggerated, and often flat-out wrong. “Gold-standard” randomized controlled trials (RCTs) are not immune: 25 percent of published RCT results are subsequently convincingly refuted. When Ioannidis narrowed the focus to the very pinnacle of the research pyramid—49 of the most highly regarded medical research findings of the past 13 years—he found the same problems.

The causes are varied and complex, including researcher bias (and sometimes even outright fraud), publication bias that highlights certain types of findings and buries others, inadequate statistical procedures, and of course financial conflicts of interest when pharmaceuticals are tested.

“Medical research is not especially plagued with wrongness,” says Freeman. “Other meta-research experts have confirmed that similar issues distort research in all fields of science, from physics to economics (where the highly regarded economists J. Bradford DeLong and Kevin Lang once showed how a remarkably consistent paucity of strong evidence in published economics studies made it unlikely that any of them were right).”

Does any of this bear on the credibility of microfinance RCTs? I could speculate about some differences.  For instance, I wouldn’t expect financial conflicts of interest to very much of a problem in current microfinance RCTs. And publication bias may be lesser; after all, a major reason that Dean Karlan and his buddies started IPA was to carry out research that academics would have a hard time getting published.

But I’m a statistical illiterate with no business opining on these issues.  Watch this space tomorrow for Jonathan Morduch’s much more informed reflections on the matter.

There are encouraging signs from new randomized impact assessments of pilot projects in the CGAP-Ford Foundation Graduation Program presented at the Microfinance Impact & Innovation conference recently held in New York. The initial results coming out the Bandhan study show that households who participated in the program have experienced a 25% average monthly increase in consumption—with important increases in nutritious food such as fruit, nuts, diary, eggs, and meat. The early results from SKS’s Ultra-Poor Program are less clear-cut, but show that participants are saving more and are less likely to borrow from money lenders than the control group.

The buzz around early results from these two studies is growing in the blogosphere: on the Financial Access Initiative blog Dean Karlan, President of Innovations for Poverty Action and Assistant Professor of economics at Yale, writes that that the results are proof “you can have an impact on the poorest of the poor.”  Jonathan Morduch of New York University and  Managing Director of the Financial Access Initiative (FAI) also referrers to the “staggeringly large” results from the Bandhan pilot, and suggests the next step will be to examine costs and sustainability (see FAI’s web site). In another recent post, Timothy Ogden, the editor in chief of Philanthropy Action, an online journal for high net worth donors, also put up a more detailed review of the results from Bandhan and SKS revealed at the New York Conference.

These recent posts suggest that the results coming out of the Graduation Pilots impact assessments could get a lot of attention in the microfinance community and the development sector more broadly. The full results of the two India studies are due soon… stay tuned!

Many of us have thought for a long time that microcredit clients are not very sensitive to interest rates.   But is that actually true?  Does the interest rate impact clients’ decision to take the loan, or are most borrowers so eager for reliable credit (or in need of funds right now), that they do not weigh the interest rate of the loan that heavily in their decision to borrow?

A recent CGAP-commissioned study by Innovations for Poverty Action measured borrowers’ sensitivity to interest rates at 132 branches of Mexican MFI Compartamos. A randomized trial offered some potential borrowers a lower interest rate relative to another group of potential borrowers, offering a range of interest rates from 3 to 4.5% a month. The results were interesting:

1. Loan uptake was 22.3% in low-interest rate clusters, versus 15% in high interest-rate clusters. Evidently, interest rates mattered to a substantial numbers of clients. 

2. Over an 18-month period, lower interest rate groups took more loans and bigger loans. Take-up rates for the lower interest rate groups were 22.3%, versus 15% amongst the higher interest rate groups. Similarly, average loan size disbursed grew by 21% in the lower interest rate groups post-treatment, resulting in an average loan size that was 4% larger than in the higher interest rate groups post-treatment.

3. Existing clients were particularly influenced by the change in interest rates, with a 17% increase in loans disbursed, versus an 11% increase amongst new clients (defined as their first loan disbursed within the current month). Maybe the existing clients were more sensitive to rate differentials because they had a baseline (the rate on their prior loans) to compare the new rates with?

4. The business case will depend on the individual MFI. In some cases, MFIs might improve profits by lowering their rates, as the revenue gained from doing more lending might exceed the revenue lost due to the lower interest rate.  In Compartamos’ case, however, the revenue gains from additional lending did not fully offset the revenue losses from the lower rates. But the balance might tip the other way if other factors are considered—for instance, gains from other services that the new customers eventually buy, or reputational value.

Now that we have some strong evidence that interest rates do matter to many of Compartamos’ borrowers, I think we could benefit from further field trials to answer two additional questions on this topic:

1. Were the new borrowers attracted by the lower interest rates people who would not have borrowed otherwise? In other words, do lower rates produce an increase in aggregate credit use, or do they just move borrowers back and forth among competing MFIs?

2. Beyond the immediate gains and losses from reduced interest rates, can we quantify some of the other elements in the equation? To do so we will need studies of total client profitability that go beyond revenue gains in the first year or two of the client’s relationship with the MFI. Glenn Westley’s recent paper on the business case for small savers looked at marginal rather than average costs, and the profitability of other products eventually sold to the client, and came away with some promising results on long-term client profitability, even if short-term profits weren’t present.  Also, does the larger average loan size produced by lower rates bring with it lower operating cost ratios?  A $400 loan produces twice the revenue of a $200 loan, but the administrative cost of the two loans might be pretty much the same.

Stay tuned for more work in this area, as CGAP is currently funding two other Innovations for Poverty Action studies of interest rate sensitivity, and we look forward to sharing their results in the future.

–Rafe Mazer and Rich Rosenberg

How can changes in the lives of microfinance clients best be measured?  How does that differ from tracking impact? Last month I participated in a panel that gathered three MFI practitioners, one social investor, and one researcher from Innovations for Poverty Action (IPA) to find answers to that question.

Anticipating some sparring between the “camps,” I sat between those proposing monitoring tools and non-experimental research as a way to track change, and those supporting impact studies with randomized controlled trials (RCTs) as the way to attribute impact, for example, from a loan or savings product.

Tracking or studying change and measuring impact are fundamentally different things, and there’s a growing schism in microfinance between these two camps. On the one hand, there is a movement composed of social investors, MFIs, management consultants, and researchers that support monitoring and change studies. For this group monitoring indicators, such as those offered by IRIS  or MIX, are sufficient: research sans randomization may give a “good enough” orientation of what microfinance really does for clients. This group also supports monitoring outputs, making diagnostics on how processes affect outputs, and non-invasive academic research that does not need control groups and helps managers to make decisions. They want rigor: but the approach is pragmatic, rather than purist. A list of the various monitoring tools can be found at the Social Impact practice of McKinsey.

The impact supporters, on the other hand, say that impact evaluation with RCTs is all that will actually measure and answer the impact question: there’s simply no other way.  And a significant amount of money and brainpower is going into it. There are currently over 300 RCTs completed or ongoing in the research portfolios of the main RCT researchers: the World Bank , J-Pal,  or IPA. “Tell me what you exactly want to know and we will measure it and give you answers,” the IPA researcher told the practitioners. 

At the heart of this discussion are different interpretations of what impact means and radically different drivers.

The one thing we all agree on is that marketing change or monitoring and evaluation as impact studies is misleading, and simply wrong. The very name “change studies”—coined by an MFI—may reflect a growing awareness and understanding of what monitoring can—and can’t—do, which is good news. But the gulf between the practitioner and the research worlds needs bridging if we’re really going to make progress on such an important discussion.

Meritxell Martinez

The Ethiopia Graduation Pilot builds on the government of Ethiopia’s food for work program.

A few months ago my colleague Aude de Montesquiou wrote on the launch of the Ethiopia Graduation Pilot. We were both in Mekelle in the North of Ethiopia with the Relief Society of Tigray (REST) last week, where the pilot has just got underway. Tigray has a population of over 4.3 million people of which 80% reside in rural areas. The region is characterized by endemic food insecurity and an estimated 90% of the population earns less than two dollars a day.

The 500 participants in this graduation pilot have been selected in 10 communities that are particularly poor, ecologically diverse (both dry and dry-wet zones) and relatively accessible for REST staff. Participants in the program have been selected based on their food insecurity, their vulnerability to shocks, their household’s irregular income, landlessness and the fact that they own no large livestock such as oxen.

Building on the recommendations from the market analysis  conducted last year combined with REST’s experience, participants have been encouraged to choose among the following livelihood options: sheep and goats, cattle fattening, honey production, petty trade, and an “open” option. All have already been trained in these activities and the asset transfers have just started.

The government public works stipend (15kg of grain or 150 birr per household member) will provide five months’ of in-kind support and one month of cash support paid out over the seasonal lean period—this is when participants particularly need the extra “breathing space” provided by the support.  Savings are mandatory, and 80% of participants have already opened individual savings accounts at Debit Credit & Savings Institute. 

On the research side, Innovations for Poverty Action has already completed the baseline survey with the participants and 500 randomly-selected control households. BRAC Development Institute has started its first round of in-depth interviews to understand the process of change in participant’s lives. 

Keep looking out for updates here: www.cgap.org/graudation/ethiopia

Syed Hashemi

The Yemen Graduation Pilot is located in the governorates of Aden, Lahij, and Taiz.

Yemen is the poorest country in the Middle East with an estimated 43% of its population below the poverty line. Currently about 1.5 to 2 million households receive support from a government safety net program. The government is interested in trying to link its existing cash support program with livelihoods training and savings services in order to help the poorest become self-sufficient over two years.

The government’s Social Fund for Development (SFD) and Social Welfare Fund (SWF) are launching the Yemen Graduation Pilot, with guidance from CGAP and the Ford Foundation. This pilot will reach 750 households in the governorates of Aden, Lahij, and Taiz.

The Social Funds have already identified very poor localities. Last week, CGAP met with the implementing team in Yemen to guide the selection of the poorest households in these communities.

Like other pilot sites, the Yemen Graduation Pilot has a sophisticated targeting process: the team first selects the poorest communities based local knowledge and gets a list of the government safety net recipients in the area. The Yemen Progress out of Poverty Index is then used to rank the poorest recipients of the safety net within these poor communities. Through household visits, field workers further narrow the list of potential beneficiaries to make sure only the poorest are included. Finally, supervisors visit all the households on the shortlist in order to minimize targeting errors. A consultant from BRAC has also cross-checked a sub-sample of targeted households.

The project partners are discussing a randomized impact assessment with Innovations for Poverty Action. The BRAC Development Institute will conduct qualitative research.