Step 5: Evaluation using a strategic logic model

This 40 minute presentation, the fifth in the full-day Learning Institute, “Social norms marketing interventions: Context, evidence, planning, implementation & evaluation," will guide the audience through the creation of a strategic logic model for social norms marketing interventions, and is based largely on the published work for Keller and Bauerle. The presentation will begin with a brief consideration of the gold standard for evidence of effectiveness, randomized control trials, and a brief guided discussion of what might constitute evidence in the absence of a randomized control trial. This will lead participants to the necessity of strengthening the “inferential line” connecting an action to an outcome.

Following this, the presentation will focus briefly on the differentiation of strategic planning from tactical planning. In general, logic models focus more on the tactical level rather than the strategic level of program planning and evaluation. That is, they specify in some detail the tactics that will be used to accomplish the project, rather than the strategic vision that informs the project. In applying a logic model to social norms interventions, we embedded the tactical within the strategic.

This approach offers three levels of integration. It integrates evaluation into program planning and promotes thinking of evaluation as integral task of program planning and implementation rather than as a separate necessity. It also integrates the strategic with the tactical, ensuring that the implementation of a program corresponds to the strategic vision for the program. And it integrates sources of data into program components, ensuring that adequate information exists to relate the process of the program to the outcome. The second advantage, related to the integration aspect, is simplicity and clarity that can be easily communicated to stakeholders, funders and other interested parties. It condenses the essential elements of an intervention and its evaluation into a single figure that can be demonstrated and explained in a straight-forward manner. Similarly, such a model can also be used to ensure that all program staff are oriented to the overall goals, strategies, tactics and evaluation of a program. Thirdly, as programs are held to higher standards of accountability, this form of logic model represents a reasonable compromise between relying exclusively on evidence of effectiveness from the literature and expert testimony, on the one hand, and facing the cost-prohibitive challenge of a formal research study, on the other hand.

Participants will then be guided through a discussion, with generous use of diagrams and illustrations, of the five components of the strategy logic model. The first component, identification of the underlying problem, is crucial: the effect of a social norms intervention is achieved solely through correction of a misperception, not by directly challenging unhealthy behaviors. If there is no misperception and/or if there is no correlation between misperception and personal behavioral choices, then theoretically a social norms approach could not have a beneficial effect. The second component, intervention strategy, specifies the type of intervention that will be used. In this part of the presentation, participants will be introduced to a variety of forms of interventions incorporating social norms at different levels of prevention (universal, indicated, targeted), including small groups, individualized feedback and Brief Motivational Interviewing. The third component specifies the immediate goal of the intervention and is crucial to anticipating the benefits of the intervention. The immediate goal is simply to reduce the misperception. The fourth component of this logic model specifies the outcome that is expected if the misperceptions are corrected. In guided discussion, participants will relate both component three and component four back to the identification of underlying problems in component one. Components three and four delineate the correction of the underlying problems. Component five then specifies the impact that correcting the underlying problems will have in epidemiological terms: reduced morbidity and mortality. The strategic logic model builds from individual effects to population effects. The presentation will conclude with a discussion of how to incorporate the tactical into such a strategic logic model; that is, what measurements and data are necessary for each component. In particular, the importance of gathering valid and reliable information relevant to the second component, the intervention strategy will be emphasized. In the absence of a research design with intervention and control groups, only by demonstrating the fidelity of the intervention to an evidence-based strategy can the remediation of the underlying problems be logically associated with the intervention. In the best sense, fidelity means that the essential components of a model are understood and then adapted, as necessary and while remaining “faithful” to original model, to the particular context and population

Following this presentation, participants will have the opportunity to work in small groups to identify the components of an applied logic model. The small group work will then be reviewed and critiqued with the participants as a whole.