Statistical techniques for assessing and increasing external validity

There has recently been increased emphasis on the identification of evidence-based services in a number of fields, including the National Registry of Evidence-based Programs and Practices in mental health, the Cochrane Collaboration in health care, and the Campbell Collaboration in the social, education, and behavioral fields. Efficacy and effectiveness trials are critical steps in establishing that an intervention merits use and dissemination in these registries. A next step is to consider the generalizability of the effectiveness results when the intervention is disseminated more widely. To this point, emphasis has been on internal validity–obtaining unbiased effect estimates for the subjects in a trial—rather than on external validity—addressing whether those subjects are representative of the population and whether the effects generalize. This talk will discuss design and analysis options for assessing and increasing external validity. Design options include the enrollment of representative samples in clinical trials, as exemplified by the recent push towards “practical clinical trials.” Analysis techniques include meta-analysis and research synthesis approaches that combine information from multiple studies, as well as newer methods that combine population-level data with data from randomized trials. The talk will discuss methodological and practical challenges in this area and highlight recent developments and applications. Motivating examples will primarily come from mental health, including both prevention and treatment. The methods have the potential to help policy makers, service providers, advocates, and consumers use available data to determine which programs and practices will be most beneficial in terms of improving public health.