Testing for equivalence: Establishing benchmarks to validate the use of primary care electronic health records as a chronic disease surveillance tool

Background:  Increased uptake of Electronic Health Record (EHR) systems provides great potential for chronic disease surveillance. Studies validating EHR based prevalence estimates primarily use traditional assessments of difference methods (i.e., T-test) to compare prevalence estimates. Contrary to traditional hypothesis testing, equivalence testing can ask if two estimates are largely the same; that is, their difference falls within a pre-established acceptable margin of difference or “equivalence margin.”  We demonstrate the utility of equivalence testing in validating population-based prevalence estimates, and investigate the most appropriate benchmarking methods to validate the NYC Macroscope, a primary care EHR chronic disease surveillance tool developed by the New York City Department of Health and Mental Hygiene, in collaboration with City University of New York School of Public Health.

Methods: Weighted population-based prevalence estimates for diabetes, hypertension, high-cholesterol, obesity, smoking, depression, and uptake of flu vaccination from the 2013 NYC Community Health Survey (CHS) and the 2013-2014 NYC Health and Nutrition Examination Survey (NYC HANES) were compared to each other with the two one-sided test of equivalence (TOST), and traditional t-test. The study population was restricted to individuals 20 years and older who have seen a healthcare professional within the last 12 months. 

Results:  Population-based prevalence estimates for each health indicator for CHS and NYC HANES, respectively, were: diabetes (12.46%, 12.57%), hypertension (31.55%, 32.36%), high-cholesterol  (47.86%, 47.35%), overweight/obese (57.32%, 65.89%), obesity (24.74%, 31.29%), extreme obesity (3.50, 5.14), smoking (14.91%, 17.73%), depression (16.44%, 15.24%), and uptake of the flu vaccination (47.26%, 47.59%). Equivalence and difference hypothesis test results were concordant for all indicators except smoking and extreme obesity. An equivalence margin of +/- 5 was appropriate for most indicators. Smaller equivalence margins maybe needed for prevalence estimates less than 10%. 

Conclusion: Equivalence testing allows for unique comparisons of prevalence estimates from two sources, and agreement from multiple benchmarking methods provides greater confidence in results. For this reason, we will continue to use multiple benchmarking methods, including TOST, to validate the use of EHR-based indicators for population health surveillance.