Evaluating fidelity to the intervention model and evaluating outcomes by dose

Background: Because fidelity to the intervention model may vary across schools, we documented delivery and coverage of the intervention.

Methods: The Environmental Strategy Implementation Survey (ESIS) collected data from the “point-people” at each of the 10 schools. The ESIS documented inputs and processes implementing environmental strategies through community organizing, and generated a site-level “dose” measure in the secondary outcome analysis. Data were masked and 5 expert Reviewers rated each school based on how closely they adhered to the community-organizing model. The Shrout-Fleiss inter-rater reliability was 0.97 for the 5 raters. The resulting measures of “Inputs” into and “Process” of community organizing were used as site-level “dose” measures in secondary analyses of outcome data. These measures were tested for 3-way interactions with group and time for each of the main outcomes.

Results: The intervention schools were consistently stronger than the 5 comparison schools in both Inputs and Process, illustrating varying but increased fidelity to the community-organizing model. Higher levels of fidelity to the model were associated with: decreased number of days drunk in past week (p=0.01) and number of days students drank in past 30 days (p=0.04); reduced dorm-level consequences of drinking (p<0.02); reduced dorm-level high-risk drinking environment (p<0.03); and reduced dorm-level high-risk drinking activities (p<0.01).

Conclusions: Our findings indicate that higher levels of fidelity to the model were associated with reductions in high-risk drinking and its consequences. The ESIS may serve as a guide for researchers and practitioners to develop more precise ways to measure program implementation and dose response.