“The woods are lovely, dark and deep”, wrote Robert Frost.

Then why have we lost 129 million hectares of forest cover (almost the size of South Africa) in the past decade?

The conference of parties (COP21) meeting in Paris last week arrived at a historic and new global consensus to stop climate change by committing to 1.5 Celsius increase in average temperature target.

Promoting greenhouse gas (GHG) sinks or areas that absorb GHGs are an important way to mitigate climate change (see more in the IPCC report). Forests, along with oceans are the most important sinks in the world. On the face of it, there have been some initiatives that have invested in forestry programmes, e.g. the Global Environment Fund estimates that to date, over US$10 billion has been invested in forest-related programmes. Similarly, the Norwegian aid agency, NORAD have spent $2.5 billion on forest conservation initiatives. (In Paris, they extended a promise to continue to support more such activities till 2030.)

Despite these initiatives, very little is still known about strategies that can help preserve forests even while they continue to provide livelihood benefits. So, why are there so few impact evaluations in this area? Equally important is the question, are impact evaluations even useful for this work?

Mind the gap

3ie’s soon to be published Evidence Gap Maps on forest conservation, and forest and land use change show that there are very few impact evaluations that examine forest conservation and overall land use change. The few impact evaluations that exist provide important pointers on what helps to reduce deforestation and GHG emissions.

An article that one of the blog authors recently published shows that most studies in this area use quasi-experimental methods rather than experimental methods to measure change, which can be causally attributed to forestry and land use programmes. In our review, we found only two impact evaluations that used random assignment (one of them is an evaluation to assess the impact of decentralised forest management in Brazil and the other one assesses payment for ecosystem services in Rwanda). This is surprising because the burden of proof on researchers is much higher with quasi-experimental methods than randomised controlled trials. It is simply easier to explain a randomised evaluation than a quasi-experimental evaluation. However, the main reason for the predominant use of quasi-experimental methods appears to be that evaluators and implementers of forestry programmes don’t usually plan to measure causal change at the design stage of their study. Early planning is critical if a randomised controlled trial is going to be used.

Impact evaluations that use experimental or quasi-experimental methods are also not usually planned at an early-enough stage because they are not popularly acknowledged as tools for measuring changes in final outcomes (such as impact on GHGs and livelihoods). In forestry programmes, output-focused evaluations that just measure the number of trees that stay protected at the end of the programme are far more popular than evaluations that go further along the causal chain.

How can impact evaluations make a difference?

Impact evaluations not only measure attributable change but they can also be very useful in several other ways.

Firstly, they help in dealing with problems related to bias. Biases may arise because forestry programmes are placed in areas where they are likely to be most successful. ‘As they should be’, many would argue. The problem here is that this also means that effects caused by programmes are either overstated or understated. For example, a study in Thailand showed that protected areas (such as national parks or wildlife sanctuaries) are likely to be located in areas that have low agricultural productivity and profitability. This meant that even if there was no designated ‘protected area’, the likelihood that the forest would be cleared, other things held constant, was low. In this case, arguing that reduction in deforestation occurred entirely because of a national park is an overestimation of the effectiveness of national parks.

Secondly, impact evaluations can help to understand if forestry programmes have targeted the right target areas or groups. For instance, with a budget of more than US$5 million, a payment of ecosystem services programme in Mexico, was successful in targeting households that were eligible for the programme and were more likely to clear forests. (See video for highlights from this study). A similar payment for ecosystems programme in Costa Rica, the PSA (Pago por Servicios Ambientales) programme, did not however achieve the same result. This occurred due to wrong targeting, i.e. people who were targeted would not have cleared forests even if they had not received the payment. So, the change caused by the payment system was minimal because the programme did not target people who were most likely to clear forests.

Impact evaluations can also compare the effectiveness of different sorts of programmes in addressing deforestation. So, for instance, an impact evaluation can help examine if programmes that employ government bodies to manage forests are more successful than those that depend on communities to manage forests. Impact evaluations in India and Nepal have found that programmes, in which governments co-manage forests with communities, have a greater impact on reducing deforestation compared to programmes that are solely managed by either the community or the state. This is because communities have incentives to save forests that benefit them, but in many cases do not have the jurisdictional authority to enforce rules. Impact evaluations have also shown that wildlife sanctuaries are more effective in protecting forests than protected areas as more resources are devoted to them.

For researchers, it has now become easier and cheaper to plan impact evaluations for forestry programmes (see here a recently published article that explores this). With advancement in technology and data collecting mechanisms, researchers can now combine the use of satellite imagery, aerial photography, and geographical information systems with social and household data. This data can be used for estimating forest cover, deforestation and land use while also understanding livelihood and welfare effects.

To sum up, ecosystems have non-linear dynamics at various spatial and temporal scales. It is often not clear when and if programmes impact forest cover as well as livelihoods. However, impact evaluations that combine rapid and easy-to-collect, aggregated and geospatial data with household data can build a compelling evidence base that answers crucial questions. As the reality of climate change looms large, it is about time we know whether forestry programmes work, and if so, for whom and why.