Stanford Healthy Neighborhood Discovery Tool: Reliability testing of a computerized tool used by older adults to audit their neighborhood environment

Recognition that the built environment (BE) can impact opportunities for active living has resulted in the proliferation of environmental assessment tools. These are traditionally designed for use by researchers as opposed to community residents. The Stanford Healthy Neighborhood Discovery Tool (SHNDT) is a hand-held computerized audit tool recently developed to allow community residents to record audio narratives and collect photographs of individuals' BE. This study assessed the inter-rater reliability (IRR) between coders (n=8) who rated audio narratives (n=153) and photographs (n=164) recorded by 28 older adults in San Mateo County, California. BE features were identified a priori from existing validated audit tools and interviews with stakeholders. Both users and coders required minimal training. Evaluation of IRR was conducted for 14 audio and 16 photographic items, issue valence (positive, negative, neutral) and user-identified solutions to recorded issues. Results indicated good reliability. Mean observed agreement between coders was: audio narratives=93.0%, photographs=94.2%, and 91.3% and 78.8% for rated issue valence of audio narratives and photographs, respectively. The mean prevalence-adjusted and bias-adjusted Kappa (PABAK) statistic between coders was 0.86 for audio narratives and photographs, 0.83 for issue valence for audio narratives, and 0.58 for photographic issue valence. Observed agreement for identifying solutions was 89.4%, with a PABAK of 0.79. Results indicate that technology-naïve individuals can readily use the SHNDT to gather meaningful data that can be reliably categorized by coders. Further research is being conducted to develop automated channels for data categorization, including crowd-sourced and user-generated solutions.