Measuring the effectiveness of disaster relief with population-representative samples: Challenges and opportunities

Assessing the effectiveness of post-disaster relief and reconstruction efforts is notoriously difficult. Researchers and development professional face three specific and related challenges: First, the “treatment area” for post-disaster aid is not known until after a disaster occurs. Second, an appropriate “control group” is hard to identify. Third, several time points are relevant for impact evaluation, including pre-disaster, immediately post-disaster, and after reconstruction. Fourth, disaster-related displacement introduces considerable selection bias when surveying affected populations. While qualitative case studies and analyses of convenience samples of affected areas can provide some insight, these studies usually cannot be used to draw causal inferences about either the specific detrimental effects of the disaster or the benefits of relief and reconstruction aid. In this paper I review the use of population-representative surveys as baseline samples for impact evaluations of post-disaster relief. Three case studies are discussed in detail: The 1998 floods in Bangladesh, the 2004 Indian Ocean tsunami, and the 2005 Pakistan earthquake. For each case study, I highlight the difficulties in identifying a baseline sample, review data collection efforts and impact evaluation studies that were carried out post-disaster, and identify lessons learned that can be applied in future disaster settings.