The first email that caught my eye this morning as I opened my inbox was Markus Goldstein’s most recent World Bank blog post, “Do impact evaluations tell us anything about reducing poverty?” Having worked in this field for four years, I too have been thinking that we were in the business of fighting poverty, and like him, I expected that impact evaluations, especially impact evaluations of anti-poverty programmes, would tell us whether we are reaching the poor and helping them to a better life. Like Markus, I have been finding otherwise.

In May this year, 3ie commenced work on a USAID-supported evidence gap map on productive safety net programmes.  The striking finding from the impact evaluations we’ve identified so far is that they echo Markus’ findings. While about half of the studies at some point mentioned that the population of interest was the poor, the majority of the studies do not report what that means.

The 3ie-USAID Evidence Gap Map is focusing on important poverty alleviation outcomes along the causal chain.  For this exercise, we defined productive social safety nets as programmes that include livelihood or income-generating components to expand market opportunities and help stabilise consumption, increase and diversify incomes, build and protect assets or improve food security of extreme poor individuals, households and communities.

Our mapping is still ongoing, but so far, we have identified 216 impact evaluations of productive safety net programmes that use rigorous experimental and quasi-experiment study designs. This includes both published and unpublished studies, and both completed and ongoing impact evaluations identified through a systematic search of the literature.  For more information about the evidence gap map methods, click here.

Our gap map covers six broad intervention categories:  social protection, financial services, land reform, microenterprise support services, collective action facilitation and multicomponent safety net interventions. As these interventions are frequently used to help people escape poverty, we would expect that the majority of the impact evaluations would target the poor, report details of the target population and measure whether people indeed escaped poverty.

However, this information too often is not reported. And where studies report targeting the poor, they rarely report whether the definition of being  poor was based on the US$1.25 cut-off, a national poverty line, their daily calorific intake, asset ownership or some other non-income deprivation measure. Only nine of the 216 included studies (4 per cent) report enough information that allow us to determine whether the population or sample was on average living on less than US$1.25.  Getting even those details often required detective work. Only 16 of the included studies (7 per cent) report whether the population or sample was on average living below a national poverty line. Only 32 studies that report targeting the poor (25 per cent) conduct some heterogeneity analysis by income or poverty status to see if the effect of the intervention varies for different target populations.

The evaluation community needs to pay more attention to this crucial issue. If we do not clearly report information about the target population and how impacts vary across people’s incomes or wealth status, our impact evaluation findings are nigh on useless to programme implementers who need to know whom to target to achieve the best results.

*****

The evidence gap map for USAID should be completed by the end of October 2014.  We will post it on the 3ie website. It will present a visual overview of existing and ongoing systematic reviews and impact evaluations of productive safety net programmes, schematically representing the types of interventions evaluated and outcomes reported. It will include links to summaries of the included evidence on the 3ie database and enable policymakers and practitioners to explore the findings and quality of the existing evidence and facilitate informed judgement and evidence-informed decision-making in international development policy and programming. The gap map will also identify key gaps where little or no evidence from impact evaluations and systematic reviews is available, and where future research should be focused.

If you want to sign up to get our announcement when the gap map is released, please email announcements@3ieimpact.org mentioning ‘gap map’ in the subject line.