Recommendations for conducting evaluations of scaled-up program delivery: Outcomes of scaling-up an evidence-based walking program across Oregon

Scaling-up evidence-based programs is needed to have population-level impacts, but evidence on effectiveness of such efforts is lacking due to costs of large-scale evaluation. Evaluation methods that reduce costs while providing useful feedback are needed. This presentation describes the process and outcomes of evaluating a multi-site, state-wide implementation of an 18-session arthritis self-management program.

The Re-AIM model was used to guide the evaluation approach. Process and outcome evaluation tools were developed from existing program resources and data collection was integrated into implementation protocols. Process data were collected by researchers through interviews with leaders (n=39) and class observations (n=33). Participant (n=598) pre/post health surveys were collected by volunteer leaders that delivered the program.

Sixty-six percent of sites successfully completed data collection. Best-practices identified in the process evaluation included training leaders for program delivery and data collection in-person vs. online, and developing partnerships across multiple sectors to share resources and increase reach. Fidelity scores varied across sites. Outcomes showed that most participants fit the programs’ target audience: 71% had arthritis; 38% did not meet physical activity guidelines pre-program. Mean class attendance was 8.5(SD=2.4). Self-reported pain and fatigue were significantly reduced post-intervention (p<.01; p<.05, respectively)

Using multiple methods for evaluation yielded important insights into factors facilitating evaluation, scale-up and delivery. Outcome evaluation, however, was challenged by incomplete data collection and poor participant retention. We describe our approach, methods, challenges and findings in conducting this large-scale evaluation and share key tools we used. We also offer practical methodological recommendations for future efforts.