A comparison of methods for measuring fidelity of implementation 0utcomes

Implementation fidelity is an important variable to measure, and measure accurately, when evaluating evidence-based programs (EBPs), particularly when implementation fidelity is the outcome of a study, such as when evaluating training programs for educators implementing EBPs. The simplest and most commonly used data sources for measuring implementation fidelity are educator self-report implementation logs. These typically ask educators to indicate which activities in a lesson they implemented and what type of changes they made. However, these self-report measures are fraught with error and often overestimate implementation fidelity. Other measures such as interviews and in-person observations are sometimes used, but are more burdensome and costly. Through a NIH-funded SBIR, we evaluated an online training program designed to improve the implementation fidelity of teachers implementing the Reducing the Risk program. Two-hundred nineteen educators across the United States were randomly assigned to either the intervention (training) or control (standard preparation using only the teaching guide) condition. Our primary outcome was increased implementation fidelity. We used several methods to measure implementation validity: online implementation logs for all 16 lessons, interviews with 24 educators within 2 days of implementing one of 4 key lessons, observations of 25 educators implementing one of 4 key lessons, and coding of audio-tapes of each of the 4 key lessons. We present a comparison of results from each of these methods, including similarities and differences in fidelity scores using each method, qualitative differences in results, resources required of each method, and contributions of each to an overall picture of fidelity.