Online Program

Testing for equivalence: Establishing benchmarks to validate the use of primary care electronic health records as a chronic disease surveillance tool

Wednesday, November 4, 2015 : 11:42 a.m. - 12:00 p.m.

Kathleen Tatem, Division of Epidemiology, New York City Department of Health and Mental Hygiene, Queens, NY
Katharine McVeigh, PhD, MPH, Division of Family and Child Health, New York City Department of Health and Mental Hygiene, Long Island City, NY
Pui Ying Chan, MPH, Bureau of Epidemiology Services, NYC Department of Health and Mental Hygiene, Queens, NY
Elizabeth Lurie, MPH, Bureau of Epidemiology, New York City Department of Health and Mental Hygiene, Division of Epidemiology, Queens, NY
Lorna Thorpe, PhD, Epidemiology and Biostatistics Program, CUNY School of Public Health at Hunter College, New York, NY
Sharon Perlman, MPH, Division of Epidemiology, New York City Health Department, Queens, NY
Background:  Increased uptake of Electronic Health Record (EHR) systems provides great potential for chronic disease surveillance. Studies validating EHR based prevalence estimates primarily use traditional assessments of difference methods (i.e., T-test) to compare prevalence estimates. Contrary to traditional hypothesis testing, equivalence testing can ask if two estimates are largely the same; that is, their difference falls within a pre-established acceptable margin of difference or “equivalence margin.”  We demonstrate the utility of equivalence testing in validating population-based prevalence estimates, and investigate the most appropriate benchmarking methods to validate the NYC Macroscope, a primary care EHR chronic disease surveillance tool developed by the New York City Department of Health and Mental Hygiene, in collaboration with City University of New York School of Public Health.

Methods: Weighted population-based prevalence estimates for diabetes, hypertension, high-cholesterol, obesity, smoking, depression, and uptake of flu vaccination from the 2013 NYC Community Health Survey (CHS) and the 2013-2014 NYC Health and Nutrition Examination Survey (NYC HANES) were compared to each other with the two one-sided test of equivalence (TOST), and traditional t-test. The study population was restricted to individuals 20 years and older who have seen a healthcare professional within the last 12 months. 

Results:  Population-based prevalence estimates for each health indicator for CHS and NYC HANES, respectively, were: diabetes (12.46%, 12.57%), hypertension (31.55%, 32.36%), high-cholesterol  (47.86%, 47.35%), overweight/obese (57.32%, 65.89%), obesity (24.74%, 31.29%), extreme obesity (3.50, 5.14), smoking (14.91%, 17.73%), depression (16.44%, 15.24%), and uptake of the flu vaccination (47.26%, 47.59%). Equivalence and difference hypothesis test results were concordant for all indicators except smoking and extreme obesity. An equivalence margin of +/- 5 was appropriate for most indicators. Smaller equivalence margins maybe needed for prevalence estimates less than 10%. 

Conclusion: Equivalence testing allows for unique comparisons of prevalence estimates from two sources, and agreement from multiple benchmarking methods provides greater confidence in results. For this reason, we will continue to use multiple benchmarking methods, including TOST, to validate the use of EHR-based indicators for population health surveillance.

Learning Areas:

Biostatistics, economics
Chronic disease management and prevention
Public health or related research

Learning Objectives:
Describe the two one-sided t-test, how it differs from a traditional t-test, and why it is used. Assess the equivalence of and difference between surveillance estimates from two surveys.

Keyword(s): Methodology, Epidemiology

Presenting author's disclosure statement:

Qualified on the content I am responsible for because: I am a current MPH in epidemiology student obtaining a certificate in applied biostatistics, and working as a data analyst on this project.
Any relevant financial relationships? No

I agree to comply with the American Public Health Association Conflict of Interest and Commercial Support Guidelines, and to disclose to the participants any off-label or experimental uses of a commercial product or service discussed in my presentation.