Reliability of self-trained data collection for tobacco retail surveillance: STARS Dissemination & Implementation Research

Background: It is essential to monitor tobacco industry activity at the point of sale (POS) to inform evidence-based policy making.  With this in mind, the State and Community Tobacco Control Research Initiative and other partners developed the Standardized Tobacco Assessment for Retail Settings (STARS) survey to measure tobacco product availability, prices, and retail marketing in local areas.  Support materials include a self-administered training presentation with practice opportunities and a reference guide for use in the field.  To test the reliability of STARS for self-trained data collectors, we conducted a concordance study comparing tobacco POS marketing measures collected in the same stores by professional, highly trained data collectors and by self-trained volunteers.

Methods: Forty-five items from the STARS survey were incorporated into the 2014 New York Retailer Advertising Tobacco Survey (RATS), conducted by highly-trained, professional data collectors as part of the state tobacco control program evaluation.  Self-trained data collectors used STARS to complete observations in 147 stores within an average of 41 days following RATS data collection (min = 0 days, max = 73 days).  Item-specific STARS and RATS data were analyzed for concordance between self-trained and professional data collectors using percentage agreement.

Results: For individual questions, concordance ranged from 95.8% (Were cigarettes sold here?) to 40.4% (exact match for single pack price of cheapest cigarettes).  Percentage concordance between professionally trained and self-trained collectors averaged 72.3% across all common survey items.    

Conclusions:  STARS appears to be an acceptable surveillance tool for self-trained data collectors for some measures and certain purposes.  STARS can be deployed rapidly and at low cost by community groups, but without more support for training, low reliability for some measures may limit the utility of volunteer-implemented STARS for rigorous evaluation of tobacco control programs.