At its annual colloquium being held in Hyderabad, India, the Cochrane Collaboration is focusing on evidence on the global burden of disease of mostly treatable illnesses that are concentrated among populations living in low- and middle-income countries (L&MICs).  We already have a lot of systematic review evidence about what works to prevent and treat them.  Yet they remain prevalent due to the lack of resources, implementation capacity and population attitudes. What we lack is implementation evidence about what works to overcome those known barriers. At 3ie, we face similar ones in producing high-quality and policy-relevant systematic reviews in international development.

Like 3ie, Cochrane provides guidance on how to conduct high-quality systematic reviews, builds capacity among authors from L&MICs to undertake them, and supports their use by decision makers. The Cochrane model, which originated in the clinical trials community, provides for a rigorous, transparent and trustworthy review product. As one leading development evaluator commented recently, “If a review of intervention effects is not published by Cochrane, or its sibling organisation, the Campbell Collaboration, I simply don’t trust the findings.”

What 3ie has found is that simple application of the traditional review method to development interventions is problematic (see here). A traditional review would not be able to answer many of the most relevant development questions for policymakers and implementers or provide sufficiently nuanced evidence to apply to complex interventions and contexts. Furthermore, findings from international development reviews are usually communicated badly, in formats which are impenetrable to decision makers. And they take ages to do, frequently 24 months (sometimes longer). Even studies that are able to communicate answers to relevant questions may arrive in the hands of implementers after decisions have been taken.

3ie has done much work to show that systematic reviews can answer relevant questions on complex development programmes.  Our reviews draw on a theory (or theories) of change to understand the intervention, the implementation processes and the evolution in outcomes for beneficiaries. These reviews can answer a very broad range of policy questions, not just on what works, where and for whom but why and at what costs, as well as questions around scaling-up. For example, a new review which examines the effectiveness of targeting agricultural programmes is available here.

Better evidence from systematic reviews can make development more effective and improve people’s lives. So review findings need to be written in easily accessible language and available to a wide audience of policymakers, international development professionals, and other users of evidence. Cochrane provides short, user-friendly summaries for informing clinical practice, while the summary of findings tables in these reviews can effectively communicate technical information about the quality of the evidence. These methods should be used, but they are not sufficient to communicate the nuanced findings typical of reviews on development topics.

3ie recently launched a summary report series that aims to present review evidence in a way which does justice to the development interventions and synthesised evidence in a more useful format for non-research specialists. These summaries are usually 25 pages and we require that they are written in accessible, jargon-free language that is easy to comprehend by non-research audiences. Our first summary report using this approach is on the effectiveness of farmer field schools.  We will be launching it in October in London.  We are now turning our efforts to launching improved policy briefs on systematic reviews that will meet our audience’s needs for succinct presentations of main points.

For all of our efforts to produce easily comprehensible summaries of full reviews, what matters is that decision makers have the evidence they need when they have to make decisions.  For some users, this means sacrificing rigour of analysis for the sake of meeting immediate evidence demands. It is my firm belief that this will undermine the key value of full systematic reviews – their reliability. More careful question setting by those demanding reviews is needed to ensure they are answerable. But the key constraint to review timelines is on the supply side. Some obvious things which can be done include better study management by author teams, reforming currently slow and inflexible review processes and reducing external peer review timelines to no more than one month.

Another constraint on the supply side is that many more resources are needed to develop authors’ skills in undertaking rigorous reviews and to train peer reviewers to support these reviews. 3ie is partnering with Cochrane’s Global Evidence Synthesis Initiative to build capacity to undertake and quality assure systematic reviews in L&MICs. The Campbell International Development Coordinating Group is also seeking expressions of interest from experienced review organisations in L&MICs for a satellite to support peer review and capacity building.

To help address the growing need to fund relevant systematic reviews in international development, 3ie is also announcing its seventh call for proposals from research teams on 30 September to answer six systematic review questions. These questions were developed in consultation with major implementing agencies. They include effectiveness of hygiene and non-food aid in humanitarian situations, programmes to empower women in the workplace, and the effects of workfare programmes, including the Government of India’s National Rural Employment Guarantee Scheme.  Weighting will be given to teams either from or partnering with organisations in L&MICs. Successful teams will also need to demonstrate that products from their rigorous reviews will be policy-relevant and written in accessible language according to 3ie’s report guidelines. We look forward to reviewing applications and hope that the reviews we fund can help answer questions for complex development programmes.