227021 Advanced psychometric methods for developing and evaluating cut-point-based indicators in complex public health surveys

Monday, November 8, 2010 : 2:30 PM - 2:50 PM

Adam C. Carle, MA, PhD , Health Policy and Clinical Effectiveness, Cincinnati Children's Hospital and Medical Center, Cincinnati, OH
In this study, I offer advice on the use of empirical techniques to create categorical indicators from responses to survey questions that ostensibly compose scales. I discuss the use of psychometric models [confirmatory factor analysis for ordered categorical measures and item response theory (IRT)], to expand previously published criteria. I present an application of this advice by developing categorical cut-points for the two subscales of the U.S. National Survey of Children's Health (NSCH) Social Competence Scale.

Methods: I used data from the 2007 NSCH (n = 63,364), a large representative sample of US children. Parents responded to 4 questions about their children's problematic behaviors and 4 about their children's positive social skills.

Results: IRT indicated that model-based scores of +1.8 and -0.7 provided suitable high and low cut-points for the problematic behaviors subscale. These cut-points had acceptable marginal reliability and corresponded to raw scores of 13 and 8. IRT indicated that a model-based score of -1.8 provided a suitable cut-point for identifying children with low positive social skill levels. This cut-point had acceptable marginal reliability and corresponded to a raw score of 13.

Conclusions: These findings indicate that analysts can use raw scores and cut-points of 13 and 8 to identify children with high, moderate, and low problematic behavior levels. They show that, for the positive social skills subscale, the subscale's psychometric properties only support a single cut-point (13) that identifies children with low positive social skill levels. The study highlights the iterative, empirically-based process that cut-point development should follow.

Learning Areas:
Biostatistics, economics

Learning Objectives:
At the end of the session, participants will be able to: 1. Identify the problems that exist when trying to create categorical indicators using data from large scale, complex survey data. 2. Explain the importance of empirically evaluating cut-points and scales. 3. Describe the cut-points developed in the National Survey of Children’s Health.

Presenting author's disclosure statement:

Qualified on the content I am responsible for because: In the past 5 years, I have published over 20 peer-reviewed articles and delivered over 30 national and international research presentations. Several of these have addressed latent variable modeling, psychometrics, and complex survey methodology. For the research presented here, I worked individually, conducted the literature searches and summaries of previous related work, undertook the statistical analyses, and developed the presentation.
Any relevant financial relationships? No

I agree to comply with the American Public Health Association Conflict of Interest and Commercial Support Guidelines, and to disclose to the participants any off-label or experimental uses of a commercial product or service discussed in my presentation.