In order to remain innovative and competitive in the field, Boston University School of Public Health (BUSPH) developed and implemented strategies for evaluating the entire educational experience. Alumni offer unique insights into the curriculum and bring perspectives based on their application of knowledge and skills in the workforce. As such, alumni are integral stakeholders for education evaluation efforts conducted at BUSPH. The alumni evaluation, which is ongoing, utilizes a mixed methods approach with quantitative and qualitative data collection strategies. Qualitative research methods, specifically focus groups, allow for the exploration of experiences, perceptions, and attitudes of stakeholders. Alumni focus groups are used to gather data pertaining to curricular design of BUSPH degree programs, application of knowledge and skills gained from degree programs in the workforce, and alumni resources. Brief demographic surveys are used to inform focus group discussions. Specifically, between Fall 2017 and Spring 2018, semi-structured interview guides were used to facilitate four focus groups in US cities with high concentrations of BUSPH alumni. Detailed focus group notes were de-identified and NVivo was used to conduct qualitative thematic analysis to identify and investigate emerging consistencies and discrepancies across focus groups. Themes included 1) an appreciation for the breadth of the MPH program and the importance of leadership and management skills; 2) a desire for more practice-based learning opportunities; 3) an interest in online continuing education opportunities; and 4) the need to raise awareness of alumni resources. The qualitative data has enriched the School’s evaluation; results have been used to inform curricular adjustments, resource allocation decisions, and recommendations for alumni engagement.

Hospital community benefit initiatives are programs tailored to improve health in communities. St. Louis Children’s Hospital, an urban pediatric hospital, provides many community benefit programs to the surrounding community.

The Child Health Advocacy and Outreach (CHAO) Department provides health-related resources and education to families to keep them safe and healthy. All of CHAO's programs qualify as community benefit and make contributions to improving quality health in the community.

One CHAO program, Head to Toe, had previously been deemed as meeting the criteria as an evidence-based practice in exercise/nutrition/weight, wellness/lifestyles, and children’s health. The program consists of 17 90-minute sessions that are led by a registered dietitian, exercise specialist, and a licensed social worker. Topics include aerobic exercise, flexibility and strength, eating away from home, making healthy snacks, reading food labels, portion size(s), family communication, body image, and stress and coping.

From 2011 to 2016, a sample of 485 child participants exhibited statistically significant improvements in knowledge of nutrition, physical activity, and emotional health. Additional statistically significant results related to mental health included the increase in goals management (p=0.001), improvements in positive ways to cope with stress (p=0.003), and limiting screen time to less than 2 hours a day (p=0.002). Through these program evaluation results, an opportunity was discovered to expand the evidence-based categories to include mental health and mental disorders.

By thoroughly examining all aspects of the curriculum and outcomes associated with a community benefit program, the St. Louis Children's Hospital is fully able to report the accurate contributions programs are making to improve population health.

Background: Since 2002, the Pennsylvania Department of Health, Division of Tobacco Prevention and Control (DTPC) has administered the PA Free Quitline (Quitline), a free telephone counseling service designed to help individuals end their tobacco use. This presentation will explore five years of evaluation, outcomes, and impact.

Methods: Data was collected between fiscal year (FY) 2013 and FY2017. Quitline’s vendor provides intake and service use data. A third party conducts follow-up among all callers who complete intake to assess quit success. Budget data are from DTPC. Evaluators update an evaluation plan every three years to reflect program priorities.

Results: 111,666 callers contacted the Quitline during the evaluation period, 91,744 of whom were tobacco users; 72.2% of individuals who completed intake completed at least one counseling call. Over five years, call volume and provision of services increased. Each year, more than 80% of callers made a quit attempt. Overall quit rate has slightly declined in the past five years, but clients receiving NRT and those completing two or more counseling calls were consistently more likely to quit than those receiving lower levels of service. Based on overall annual quit rates, an estimated 21,408 Quitline clients have quit tobacco since January 2013. Estimated savings totaled $16.9 million a year, a return of about $6 for every dollar invested in Quitline services.

Conclusion: Using five years of data allowed evaluators to assess services used by tobacco users, the overall quit rate, and annual cost savings from investing in cessation, connecting outcomes to initial evaluation priorities and program improvement.

Obtaining an accurate measure of self-efficacy is a challenge in youth program evaluation. Traditional pre-posttests are effective for knowledge and behavioral change, but may not accurately measure intrinsic psychosocial change. Retrospective pretests are an increasingly accepted method for assessing this change, especially among youth, in scenarios where the intervention changes the internalized measurement standard. For example, a child may be confident in her ability to run a 5K before participating in Girls on the Run (GOTR), but after the 10-week program, realize she didn’t initially understand the commitment involved. A traditional pre-posttest would measure the child confident at pre and post, but using a retrospective pretest allows the child to estimate their earlier confidence level from their current perspective. We use retrospective pretests with a modified-Bandura self-efficacy scale in three youth physical activity evaluations for Bicycle Skills Clinics (n=1,168), GOTR (n=17), and Run Hard (n=52). Skill self-efficacy is specific to the program’s curriculum, but each survey asks about the ability to finish a hard task. Participants rate their ability to complete a hard task on a scale of 0 to 10 for both pre and post program and their responses are analyzed using Wilcoxon Signed-Rank tests. Average change from pre and post self-efficacy estimates for completing a hard task is 1.36 (z=21.6, p<0.00) for Bike Skills Clinic, 3.0 for GOTR, and 1.92 for Run Hard (z=4.99, p<0.00). The use of retrospective pretests in youth program evaluation offer the potential for improved self-efficacy measurement and an alternative to measurement at multiple timepoints.

Background: Indigenous communities face many health inequities compared to Whites including higher rates of chronic illness (CI). Existing CI management interventions are not as effective due to the lack of cultural specificity. A partnership between Messengers for Health and Montana State University used a community-based participatory research approach to develop Báa nnilah, a CI self-management intervention for the Apsáalooke (Crow) Nation. Ten facilitators (Aakbaabaaniilea), identified by the community as good managers of their CI, led twenty participants in seven intervention gatherings. To maximize intervention effectiveness, it is important to address treatment fidelity strategies outlined by the National Institutes of Health Behavioral Change Consortium. Despite its significance, there is an underreporting of treatment fidelity strategies in health interventions, particularly among Indigenous health interventions. Methods: Centering on Apsáalooke cultural strengths, we developed a treatment fidelity monitoring plan to improve two of the five fidelity domains: 1) provider (Aakbaabaaniilea) training and 2) treatment delivery. We assessed the effectiveness of the monitoring plan through semi-structured interviews with Aakbaabaaniilea, field notes, intervention checklists, and a 10% random sample evaluation of intervention gatherings. Interviews were audio-recorded, transcribed verbatim, and compared for quality control. Seven Indigenous and non-Indigenous researchers reviewed and analyzed the transcripts using thematic analysis. Results: This presentation will share emerging themes and a conceptual framework for understanding two treatment fidelity domains in the Apsáalooke community. Conclusion: Implementing a culturally-tailored treatment fidelity monitoring plan enhances the reliability and validity of Báa nnilah, which can be adapted to meet the specific cultures of other Indigenous communities.

Healthcare Georgia Foundation developed The Two Georgias Initiative, a multiyear investment in rural Georgia community partnerships to increase community capacity for change, improve health outcomes, and ultimately reduce health disparities in Georgia’s rural communities. The Foundation provides grant funding, technical assistance, and coaching to 11 community partnerships. In Year 1, grantees formed diverse, multisectoral partnerships, conducted local community health needs assessments, and developed a three-year community health improvement plan to be implemented. Local health needs assessments and prioritization sessions informed partnerships’ selection of a variety of social determinants of health and related health issues to address, including poverty, education, literacy, economic development, and health care access. Desired health equity outcomes include chronic disease prevention as well as improved health care access and services provided.

The Emory Prevention Research Center supports initiative grantees in developing and implementing their local evaluation plans and is conducting a cross-site evaluation. Baseline evaluation data collected to date includes partnership surveys (n=236), community surveys (in progress, goal of 2,750), and key informant interviews (n=46). Process measures include partnership composition/functioning, strategies to address health equity, strategy implementation, barriers and facilitators, support provided to partnerships, and steps taken to ensure sustainability. Outcome measures focus on changes in community and organizational capacity to address health equity, changes in health disparities and health outcomes, and sustainability beyond the initiative. This presentation describes how partnerships conceptualize health equity, how they are addressing inequities in their communities, and how the evaluation will assess health equity throughout the project.

Introduction: Childhood obesity is a complex public health issue with multiple contributing factors, including health behaviors and the built environment. To address this epidemic, multi-sector collaboratives are needed to align efforts to common goals and implement changes larger than a single organization could achieve. The San Diego County Childhood Obesity Initiative (COI), funded by the County of San Diego (COSD), is a collective impact initiative promoting healthy eating and active living through policy, systems, and environmental (PSE) changes. From 2017 to 2019, COSD partnered with Harder+Company Community Research to evaluate the COI’s collective impact model to reduce and prevent childhood obesity.

Approach: COSD assessed how the COI applies the core constructs of collective impact (common agenda, shared measurement, mutually reinforcing activities, continuous communication, and backbone organization) to advance PSE changes. Surveys were administered to COI members to assess the COI’s organizational structure and use of shared measurement.

Results: Of members surveyed on organizational structure (n=30), most agreed that the COI works toward outcomes that collectively reduce childhood obesity (97%) and its backbone organization has capacity to advance PSE efforts (84%). Of members surveyed on shared measurement (n=93), most agreed that the COI uses data to inform decisions and has ways to measure progress (80%); however, 23% reported not understanding how progress is measured.

Discussion: While the COI’s engagement of diverse stakeholders around a common agenda and capacity to advance PSE efforts emerged as strengths, increasing member involvement and communication around tracking progress are recommended to address gaps in shared measurement.

Background: Although the leading health agencies and organizations have been advocating for a decade regarding the needs of integrating family health history (FHH) into the practice among public health professionals, public health professionals still face challenges. One of the major barriers that prevent them from practicing is the lack of genomic competences, especially for school health educators at the K-12 setting. To increase this particular group’s genomic competencies, we delivered a theory-based, full-day FHH training workshop for 97 school health educators at one Independent School District in Texas. This study reports the evaluated findings.

Methods: Pre-workshop, immediate post-workshop, and three-month follow-up survey data were collected and analyzed using the Linear Mixed Modeling.

Results: Most school health educators in our sample were males (62.8%), Latino (84.0%), with an average age of 43.6 years and 15.4 years of work experience. Participants’ knowledge, attitudes, self-efficacy, and intention in adopting FHH into their classes significantly improved immediately after the workshop compared to the pre-workshop data (ps<0.001). At three-month post-workshop, there was a significant increase of the number of participations in the adoption of genomics into FHH into their classes.

Discussion: Our workshop successfully increased participating school health educators’ knowledge, attitudes, self-efficacy, intention, and behavior in integrating FHH into their classes. More workshops are needed to reach to a larger number of school teachers in Texas and other states.

Background: Childhood obesity is a major global health concern. Weight-management camps involving delivery of a program of physical activity, health education, and healthy eating are an effective treatment, although post-intervention weight-management is less well understood.

Methods: Participants (n = 300) were Qatari schoolchildren who were overweight or obese, ages 8 – 14 years, recruited over a 3-year period across 14 randomly selected schools in the Doha area. They attended a 2-week weight management camp, then a 10-week program of weekly lifestyle education and activity sessions. Effectiveness of the weight-management camp followed by a community intervention in supporting weight-management was investigated.

Results: Participants saw a significant body mass index standard deviation score (BMI SDS) reduction as a result of the entire intervention (camp + education and activity sessions) both at the individual (p<0.0001) and cluster/school (p=0.0002) levels, and weight loss occurred during each intervention stage separately for the camp (p<0.0001 for both the individual and cluster/school levels) and the lifestyle education and activity phase (p<0.0001 and p=0.0220 at the individual and cluster/school levels, respectively).

Conclusions: Weekly lifestyle education and activity sessions may be useful in promoting continued weight management in the period following intensive, immersive childhood obesity interventions.

Background: A contemporary need for physicians trained in the public health sciences and LCME accreditation requirements mandating integration of public health objectives into medical school curricula focused an evaluation of Wayne State University School of Medicine’s (WSU-SOM) public health content delivery in the first-year (M1) course, Population, Patient and Physician (P3).

Methods: Stake’s countenance framework guided and organized this program evaluation. Mixed methods analysis, including quantitative outcomes review, key-informant student interviews, and thematic analysis, were used to determine whether the course effectively implements public health curricular elements.

Results: Four themes emerged: 1) The current structure of the P3 course lacks incentives to engage students with the material, thus students are not motivated to take the public health content seriously, 2) Students regard the public health standards and content as important and suggest an organizing framework to strengthen and reinforce the ideas throughout the M1 curriculum, including P3 and basic medical sciences courses, 3) Volunteer and service-learning components of the course are valuable for application, but formalized follow-up is necessary to have students draw connections between course content and experiential learning, and 4) As an instructional method, Small Group sessions work well for public health topics that require discussion, open dialogue, and reflection; however, not all small group leaders have similar skill set or teaching style.

Conclusions: These themes and supportive literature focus recommendations for improving delivery of WSU-SOM’s public health curriculum. Curricular evaluations are strengthened by models that incorporate student voices.