Leaders and practitioners across the international development space have emphasized, re-emphasized, and re-re-emphasized the importance of integrating rigorous costing in evaluation research. Despite these calls to action, rigorous cost analyses that go beyond back-of-the-envelope calculations are rarely included in impact evaluation, due in part to insufficient guidance in fields other than health and education (Brown & Tanner, 2019; Glandon et. al, 2023). To meet this need, the Cost Evidence Group at 3ie has developed a handbook for Measuring Cost Effectiveness in Impact Evaluation.
Collecting thorough and accurate data on program costs can help organizations and policymakers compare costs and effects to support policy decisions. Detailed cost information enables policymakers to consider trade-offs among programs that target similar objectives, and inform budgeting or resource planning. Considerations of cost are critical for the decision maker in the scale-up or replication of a program to new regions. The addition of costs to effectiveness findings can help policymakers make informed decisions based on robust evidence from multiple sources. For this reason, the collection of transparent and granular data are critical steps in providing the right ingredients for creating the most useful possible systematic reviews of the existing evidence.
The handbook provides a comprehensive ‘how-to’ guide for implementing cost-effectiveness analysis (CEA) in impact evaluation. It adds value to prior CEA resources by providing 1) practical guidance on integrating CEA into evaluation, including during the evaluation design, implementation, and reporting phases; 2) applied examples for empirical application; and 3) sector-agnostic guidance to ensure generalizability for practitioners and policymakers working across diverse sectors.
A key aspect emphasized in our handbook is the importance of granular, prospective cost capture, in which cost data collection is treated as primary research. Often, cost analyses rely on budget data, which may not reflect how resources are utilized in practice. The use of high-level program expenditures, collected when the implementation of a program is over, may not allow for the flexibility of investigating cost drivers at the input, activity, or regional level. By collecting actual expenditures throughout program implementation, prospective cost capture ensures costs are accurate, and maximizes flexibility in how costs are identified, aggregated, and reported.
It is ethically imperative to plan prospective cost capture in a way that supports implementing partners and is not extractive. Our approach aims to build partners' capacity to gather and use cost evidence for their own decision-making, including during implementation, and in coordination with other monitoring and learning processes.
The overarching method for costing laid out in our new guide can be adapted for different programs by researchers, implementation teams, monitoring and evaluation (M&E) officers, and funders, such as government and donors. It includes stepwise practical guidance on how to:
- Define the intervention and study scope
- Design a cost collection template
- Collect and adjust costs
- Report costs and impact
We are actively piloting the handbook with our partners and will continue to refine and iterate these guidelines based on user feedback. To that end, we are currently expanding our scope to include other common CEA applications, such as scale-up decision-support, using incremental cost-effectiveness ratio (ICER) thresholds, and budget impact analysis.
We welcome all inquiries, critiques, questions, and suggestions. If you are interested in learning more about 3ie’s costing approach and training resources, or partnering on cost-evidence projects, reach out to info@3ieimpact.org.
