This is the first in a series of blogs on ethics in social science research. Read the introductory post here.
In order to conduct an impact evaluation, researchers and implementation partners sometimes justify withholding an intervention from some eligible people to form a control group – for example to conduct a randomized controlled trial (RCT) – on the grounds that resources for an intervention are scarce. The argument is that since there are insufficient resources (e.g. money or bureaucratic capacity) to offer an intervention to all eligible people, it is fair to allocate access to the treatment by means of a lottery.
This is a promising line of argument for justifying random assignment in cases where the treatment is clearly superior to its absence. After all, lotteries are sometimes a fair way to allocate access to a scarce benefit.
Implicit in this line of argument, however, is the claim that randomization amid scarcity is permissible because it is consistent with the requirements of distributive fairness. In this post, I outline these requirements more fully, exploring when it is fair to randomly assign participants to a superior but scarce intervention.
Consider an example. In 2010, the Zambian Ministry of Community Development, Mother and Child Health (MCDMCH) began implementing the Child Grant Program in three of Zambia’s poorest districts, providing a flat unconditional cash transfer to families with a child under five. Researchers partnered with the MCDMCH to study its impacts, randomly assigning communities to either the cash grant or a delayed-entry control group. The researchers acknowledged that unconditional cash transfers had been shown to improve consumption, food security, and school enrollment, but argued that randomization was “ethically feasible” because the MCDMCH did not have sufficient resources or capacity to provide the intervention to all eligible households.
Was random assignment in this study fair? The principal requirement of distributive fairness is that scarce goods should be allocated according to the strength of people’s claims to them. This requirement is acknowledged in the design of many social protection programs, where the use of means-testing to determine eligibility for programs such as public health insurance, cash grants, or housing assistance, expresses the claim that low-income people have stronger claims to public resources than middle- or high-income people. Similarly, in the roll-out of COVID-19 vaccines, the U.S. Centers for Disease Control and Prevention recommended that healthcare personnel, essential workers, people 65 and older, and adults with high-risk medical conditions receive priority during the initial rollout. The former two groups have stronger claims to vaccines than the broader population given their role in maintaining societal critical infrastructure, and the latter two groups have stronger claims given their increased risk for severe illness.
Lotteries are consistent with distributive fairness when it is necessary to allocate a scarce good among a group of people who (1) have equally strong claims to the good; and (2) have stronger claims to the good than everyone else. For example, if there was an insufficient supply of vaccines to offer a dose to all people within a certain risk category, such as adults with high-risk medical conditions, it would be fair to allocate the limited doses using a lottery.
Turning back to policy experiments, it is fair to randomly assign people to a superior but scarce intervention when three conditions are satisfied:
1. The people subject to randomization all have equally strong claims to the intervention.
2. The people subject to randomization have stronger claims to the intervention than those not subject to randomization.
3. No person’s claim is left unmet longer than is necessary due to legitimate scarcity.
In the case of the Zambian RCT, condition 1 would have been satisfied if the total eligible population for the intervention – all families with children under 5 living in the poorest districts – all had an equally strong claim to the cash transfer.
Condition 2 ensures that those subject to randomization have the strongest claims to the intervention. If others have a stronger claim to the intervention, for example because they are worse off, distributive fairness requires that they should receive it. Relatedly, anyone whose claim is equally strong as those subject to random assignment should be eligible for random assignment. In the case of the Zambian RCT, condition 2 thus requires comparing the claims of the eligible population to the population that is ineligible for the intervention – all households living in the poorest districts with no children under 5 as well as households living in other districts.
Condition 3 requires that scarcity not be manufactured or prolonged to carry out the RCT. For example, the intervention must not be withheld from participants in the control arm, either at the start of the study, or at any period during the study, because doing so is necessary to produce meaningful data or to complete the study once it has begun. In short, randomization justified by scarcity is only fair when it is consistent with rolling out the intervention to those who need it as quickly as is feasible. In the case of Zambian RCT, this condition would have been satisfied if the delayed rollout of the intervention to all eligible households was dictated by a lack of bureaucratic capacity as opposed to the imperatives of study design.
So in this case, were the researchers right that random assignment was ethically feasible in the MCDMCH RCT? It’s difficult to say, but they can make a strong case. First, while it is likely that some eligible households had stronger claims to the child grant than others, for example, because they were poorer or had more children, it was likely not feasible to identify and target these households. The inability to more effectively target the intervention meant that all people subject to randomization, to the extent that could be discerned by researchers and implementation partners, had equally strong claims to the intervention. Indeed, 95% of eligible households fell below the Zambian national extreme poverty line, and the researchers highlighted the administrative simplicity of the program as a benefit for resource-poor governments. Second, the study was conducted in three of Zambia’s poorest districts which also featured the highest rates of child mortality, making it likely that those subject to randomization had stronger claims to the intervention than those not subject.
Finally, there is no reason to doubt the researcher’s claims regarding the MCDMCH’s lack of resources. Of course, if it became possible to roll out the cash grant to eligible households faster, the MCDMCH should have done so, regardless of whether it would compromise the findings of the study. As the researchers recognize, randomization in this case is permissible because it is consistent with the requirements of distributive fairness. But if distributive fairness prohibits the schedule of randomization favored by researchers, so much the worse for randomization.
After reading this blog from Dr. MacKay, the 3ie team offers these reflections:
When scarcity is the main ethical argument for the study design, balance the costs and risks of conducting the research against the expected social value from the use of research findings. Lotteries may be an ethical way to allocate a superior but scarce treatment across an eligible population, but this logic doesn’t necessarily justify the additional burden and risk introduced by implementing a research study across the participant population. As researchers producing evidence for decision-makers, we need to not just ask ‘can we study this intervention using these methods’ but also ‘should we?’.
Research teams should elevate discussions with implementation partners on how to prioritize allocating the intervention to the control group if scarcity can be resolved. Unlike the treatment group, the control group bears the burden of research without the potential benefits. When feasible, the research team should work with implementation partners to determine the extent to which the control group can be prioritized to receive the treatment if scarcity of resources is resolved, particularly if the study demonstrates the treatment results in superior outcomes.
Match research methods to intervention design and implementation rules. Understanding the participants’ claim to the intervention should be built into intervention design. Research teams should understand intervention eligibility rules to assess scarcity. If a program targets all households with adults 70 and older and there is scarcity, the research team may be able to randomize treatment across all eligible households with adults 70 and older. If there is no scarcity or scarcity is resolved, then generating rigorous evidence may still be possible via other research methods, such as regression discontinuity (example) based on the intervention eligibility rules.
When the ethical argument for withholding treatment is scarcity, have an exit strategy. Research teams and implementation partners should consider their exit strategies for any point in time when scarcity is resolved and the treatment should no longer be withheld, even if the study is incomplete.
When you have an exit strategy, think about how to exit in a way that respects staff and minimizes harm. For many studies, field staff commit several months to data collection. Some have postponed or declined other work and have set expectations for income that is defined through their contract with the data collection firm. How do research teams build in plans to cancel or scale back a study if scarcity is resolved while also accounting for fair treatment of field staff? This question should be posed early in budgeting and contract planning.
