Advanced psychometric methods for developing and evaluating cut-point-based indicators in complex public health surveys

In this study, I offer advice on the use of empirical techniques to create categorical indicators from responses to survey questions that ostensibly compose scales. I discuss the use of psychometric models [confirmatory factor analysis for ordered categorical measures and item response theory (IRT)], to expand previously published criteria. I present an application of this advice by developing categorical cut-points for the two subscales of the U.S. National Survey of Children's Health (NSCH) Social Competence Scale.

Methods: I used data from the 2007 NSCH (n = 63,364), a large representative sample of US children. Parents responded to 4 questions about their children's problematic behaviors and 4 about their children's positive social skills.

Results: IRT indicated that model-based scores of +1.8 and -0.7 provided suitable high and low cut-points for the problematic behaviors subscale. These cut-points had acceptable marginal reliability and corresponded to raw scores of 13 and 8. IRT indicated that a model-based score of -1.8 provided a suitable cut-point for identifying children with low positive social skill levels. This cut-point had acceptable marginal reliability and corresponded to a raw score of 13.

Conclusions: These findings indicate that analysts can use raw scores and cut-points of 13 and 8 to identify children with high, moderate, and low problematic behavior levels. They show that, for the positive social skills subscale, the subscale's psychometric properties only support a single cut-point (13) that identifies children with low positive social skill levels. The study highlights the iterative, empirically-based process that cut-point development should follow.