Evaluating a complex system: A proposed methodology to assess the impacts of a comprehensive tobacco control strategy on smoking cessation objectives

Background: Reducing the burden of tobacco disease requires a comprehensive strategy, including a multifaceted cessation system with multi-level interventions (policies and programs) that address environmental, institutional, and social systems. Evaluating such a complex system presents several challenges including, but not limited to: 1) planning for synergistic effects, feedback loops and non-linearity, 2) selecting appropriate multi-level indicators, 3) determining attribution for an outcome, 4) identifying paths, through which different services work to achieve smoking cessation.

Objectives: The objective of this study is to develop and implement an approach that will assess the impacts of multiple interventions on cessation outcomes.

Methods: We are using a multi-method approach to evaluate the smoking cessation system; path logic models are being used to represent the complex system. After identifying the paths through which interventions work to advance overall strategy objectives (through literature review, and administrative data), we are now determining t the expected contribution of each intervention (through literature review, and program plans) and determining the actual contribution of each intervention in helping achieve the strategy's goals (performance measurements and program evaluation).

Results The evaluation, which is still in progress, already shows the paths each intervention is affecting to help people quit smoking, as well as the paths that are being underutilized by current interventions.

Conclusions The insights that will be gained from this evaluation, including underutilized paths, and the inconsistencies between expected contributions and actual contributions, will be helpful for future planning of a comprehensive tobacco control strategy.