In early 2016, 193 governments across the world put together a to-do list that would intimidate even the most workaholic overachiever: wipe out poverty, fight inequality and tackle climate change over the next 15 years. The United Nations led in articulating these into 17 Sustainable Development Goals (SDGs) which were then translated into 169 target indicators that will be monitored – a remarkable feat given the disparate views of the various stakeholders.
But what needs to be done to get these indicators to move? What actions will be most effective, especially given the constrained resources globally? The Copenhagen Consensus, for example, claims that the goals and indicators need to be prioritised for their cost-effectiveness. Since countries the world over have been addressing most of the SDGs for some time, one would assume there must be a body of lessons of what works, where and when. That’s where the problem lies.
There are indeed many lessons. But they need to be curated. Anecdotes, correlations and rigorous counterfactual evaluations need to be sorted out, the conditions for which lessons in one setting can be applied in another need to be identified, and how an intervention’s effects can be traced over time and over different scales have to be tracked.
Research syntheses do this much-needed curation. Systematic reviews (SRs) use transparent methods to find, assess and synthesise the best available results on a research question, such as what works best in achieving outcomes like some of the SDG targets. Evidence Gap Maps (EGMs) provide a visualisation of the density of the evidence available on a particular topic. These kinds of research syntheses are sorely needed for informing decision-making related to the SDGs.
How systematic reviews offer a sound basis for decision-making
An impact evaluation of a programme may show different results based on the context. Let’s take the example of supplementary feeding programmes that aim to reduce undernutrition in young children. A recent 3ie-supported systematic review shows that these interventions are effective overall, but there is great deal of variability in the results depending on the context. Food supplements are effective at critical times, especially in infants and children younger than two years. They are also effective when children are poor or malnourished and when supplementary feeding is supervised. Finally, the quality and quantity of the food also matters.
SRs, unlike single impact evaluations, thus show the variety of results that a programme can achieve under different circumstances. Programme designers and policymakers need to consider these contextual factors, if they want their programmes to succeed. Evidence from systematic reviews can be used for making decisions that are better tailored to specific context. See this 3ie blog for some more examples of SRs that can be used for informing approaches to achieving the SDGs.
There are however challenges to using these reviews. No matter how good they are, SRs cannot be expected to provide a policy ‘magic bullet’ or define a specific set of interventions that nations should implement to move an SDG needle. It is therefore important to manage expectations while dealing with the challenge of distilling lessons from research syntheses to move on the SDG indicators.
How systematic reviews can address complexity
Another challenge is that by their nature, SDGs and their targets are breathtakingly broad. 3ie’s in-house systematic review on education effectiveness, difficult enough as it was to carry out, was able to study only three or maybe four of SDG 4’s ten targets. SDG 4 seeks to ensure inclusive and equitable quality education and promote lifelong learning opportunities for all. The targets excluded from the systematic review included those that relate to ensuring equal access to affordable vocational and technical education, increasing the number of youth and adults who have relevant skills for employment, promoting culture and peace, and providing access to persons with disability.
The inclusion of these targets would have made the study too unwieldy and impossible to finish. The problem is compounded because all of the SDGs are related to one another. Moving the needle on one target has implications for the others. Mapping these interlinkages was such a complex exercise that the UN sponsored a competition on how to visualise them interactively (See winner’s graph).
Research syntheses can address this complexity if they provide a theory of change which maps out the most important linkages. For example, a recently published 3ie-supported SR investigates the impact of self-help groups (SHGs) on women’s empowerment. SDG5 seeks to empower all women and girls, and SHGs are promoted in South Asia precisely to do that. But do they actually empower women? And if yes, why? A 3ie-supported review examined not only the impact of SHGs on various kinds of empowerment but also the intermediate outcomes of SHG participation. It does so by building a solid theory of change that informs the search and review process. The authors found that SHGs have a positive effect on the economic, social and political empowerment of women. They build women’s independence by improving the ability of women to access, own and control resources. They also increased women’s bargaining power within the household and also fostered solidarity amongst SHG members.
Source: United Nations
How EGMs explore the causal chain
EGMs are a first step in understanding the complexity of the causal linkages between interventions and their final outcomes. EGMs do not tell us whether interventions are successful but they show the available evidence on the effectiveness of a particular intervention. Again, it is not only the final outcome that is interesting but also the several steps leading to it.
For example, a 3ie EGM on education interventions points to the evidence on teacher performance, student attendance and test scores, one outcome leading to the other. Similarly, another 3ie EGM on youth and transferable skills directs us to the evidence in the following causal order: students’ learning, market behaviours, employment and wages. A 3ie blog provides an example of how an EGM on productive safety net programmes can quickly tell us what we know and don’t know for designing solutions to eradicate poverty and meet SDG1.
In sum, evaluation is critical if we are to know what will move the SDG indicators – a point already made forcefully by the UN Evaluation Group. An important component of this effort is to synthesise the results of rigorous evidence gathering. Applying the lessons from these syntheses must account for local context and be based on a robust theory of change.
3ie is organising its second London Evidence Week from 11-15 April at venues in the London School of Hygiene and Tropical Medicine (LSHTM) and Birkbeck, University of London. It is a series of events that will bring together evaluators, researchers, policymakers and programme managers The week-long discussions will try to explore the challenges and opportunities in using high-quality evidence to inform decision-making, especially around key sectors identified by the UN SDGs.
