Online Program

Recommendations for conducting evaluations of scaled-up program delivery: Outcomes of scaling-up an evidence-based walking program across Oregon

Wednesday, November 4, 2015

Kathleen Conte, PhD, School of Social and Behavioral Health Sciences, Oregon State University, Corvallis, OR
S. Marie Harvey, DrPH, MPH, College of Public Health and Human Sciences, Oregon State University, Corvallis, OR
Michelle Odden, PhD, College of Public Health and Human Sciences, School of Biological and Population Health Sciences, Oregon State University, Corvallis, OR
Scaling-up evidence-based programs is needed to have population-level impacts, but evidence on effectiveness of such efforts is lacking due to costs of large-scale evaluation. Evaluation methods that reduce costs while providing useful feedback are needed. This presentation describes the process and outcomes of evaluating a multi-site, state-wide implementation of an 18-session arthritis self-management program.

The Re-AIM model was used to guide the evaluation approach. Process and outcome evaluation tools were developed from existing program resources and data collection was integrated into implementation protocols. Process data were collected by researchers through interviews with leaders (n=39) and class observations (n=33). Participant (n=598) pre/post health surveys were collected by volunteer leaders that delivered the program.

Sixty-six percent of sites successfully completed data collection. Best-practices identified in the process evaluation included training leaders for program delivery and data collection in-person vs. online, and developing partnerships across multiple sectors to share resources and increase reach. Fidelity scores varied across sites. Outcomes showed that most participants fit the programs’ target audience: 71% had arthritis; 38% did not meet physical activity guidelines pre-program. Mean class attendance was 8.5(SD=2.4). Self-reported pain and fatigue were significantly reduced post-intervention (p<.01; p<.05, respectively)

Using multiple methods for evaluation yielded important insights into factors facilitating evaluation, scale-up and delivery. Outcome evaluation, however, was challenged by incomplete data collection and poor participant retention. We describe our approach, methods, challenges and findings in conducting this large-scale evaluation and share key tools we used. We also offer practical methodological recommendations for future efforts.

Learning Areas:

Conduct evaluation related to programs, research, and other areas of practice
Implementation of health education strategies, interventions and programs
Program planning
Public health or related research
Systems thinking models (conceptual and theoretical models), applications related to public health

Learning Objectives:
Define scaling up and explain importance of evaluation when scaling up programs Discuss approaches to reducing costs while maximizing impact of large-scale evaluations

Keyword(s): Evaluation, Outcomes Research

Presenting author's disclosure statement:

Qualified on the content I am responsible for because: I am designed, implemented, and evaluated the scale up of the Walk With Ease program via the OSU Extension Service. I am a PhD student in Public Health, with experience in program implementation and evaluation of evidence-based programs
Any relevant financial relationships? No

I agree to comply with the American Public Health Association Conflict of Interest and Commercial Support Guidelines, and to disclose to the participants any off-label or experimental uses of a commercial product or service discussed in my presentation.