132 Annual Meeting Logo - Go to APHA Meeting Page  
APHA Logo - Go to APHA Home Page

Observer Training Strategies and Interrater and Test-Retest Reliability: The Neighborhood Observational Checklist

Shannon N. Zenk, PhD, Cancer Center, University of Illinois at Chicago, Health Research and Policy Centers, 850 W. Jackson Blvd. M/C 275, Chicago, IL 60607, 312-922-8426, szenk@umich.edu, Amy Schulz, PhD, Health Behavior and Health Education, University of Michigan, School of Public Health, 5134 SPH II, 1420 Washington Heights, Ann Arbor, MI 48109, Graciela Mentz, PhD, Health Behavior Health Education, University of Michigan, School of Public Health, 2029 SPH II, 1420 Washington Heights, Ann Arbor, MI 48109, James S. House, PhD, Department of Sociology, University of Michigan, Institute for Social Research, Survey Research Center, Ann Arbor, MI 48109, Patricia Miranda, MPH, Health Behavior and Health Education, University of Michigan, SPH II, 1420 Washington Heights, Ann Arbor, MI 48109, Clarence Gravlee, PhD, Anthropology, Florida State University, 1847 West Tennessee Street, Tallahassee, FL 32304, Pat Miller, Southwest Counseling and Development Services, 1700 Waterman, Detroit, MI 48209, and Srimathi Kannan, PhD, Department of Environmental Health Sciences, University of Michigan School of Public Health Human Nutrition Program, 1420 Washington Heights, School of Public Health, Ann Arbor, MI 48109.

Neighborhood observation, whereby trained observers systematically document well-defined aspects of neighborhood environments, is an increasingly popular method to identify mechanisms by which living in impoverished neighborhoods negatively impacts health. In this presentation, we provide an overview of the content of the Neighborhood Observational Checklist (NOC), a tool we used to assess aspects of the physical, built, and social environments of neighborhoods. It was developed using a community-based participatory research process by the Healthy Environments Partnership. We describe strategies employed to train residents to collect data with the NOC. The observer training strategies included the following: instruction on operational definitions including use of photographs to illustrate items and rating scale categories, instruction on handheld computers for data collection, group and individual practice sessions, group discussions around NOC content and community sensitivity during data collection, feedback of interrater reliability (IRR) results based on performance on practice blocks to observers including individualized feedback, and certification of observers based on individual IRR results. We then share reliability results from data collected on 550 blocks in Detroit. We evaluated IRR in two ways: for blocks across twelve observers, including the gold standard, using a Kappa statistic proposed by Gwet (2002) and for street segments rated by two different observers using Cohen’s Kappa statistic. Test-retest reliability was evaluated for street segments rated by the same observer when observing adjacent blocks at different time points using Cohen’s Kappa statistic. Finally, we offer suggestions for future research and discuss implications for community health planning efforts using neighborhood observation.

Learning Objectives:

Presenting author's disclosure statement:
I do not have any significant financial interest/arrangement or affiliation with any organization/institution whose products or services are being discussed in this session.

Methodological Techniques and Tools Utilized in Health Care Planning, Policy Development and Evaluation-II

The 132nd Annual Meeting (November 6-10, 2004) of APHA