Constructed response scoring application based on signal detection theory to improve precision of rater effect: Analysis of Life Story project for victims of Hurricane Katrina

Unlike multiple choice (MC) items which can be objectively scored as right or wrong, constructed response (CR) items are evaluated on a range of ordinal values (scores) by multiple raters. Individual rater's leniency and strictness influence CR scoring. Therefore, it is of interest to understand how these assessments behave—whether they vary by individual raters or how well these ratings conform to the test-taker's true ability.

This proposal is based on the data from Life Story Project examining the effect of the Testimony Therapy and oral history in the form of story-telling, to improve recovery of the Hurricane Katrina and Rita-afflicted Gulf Coast residents. 100 subjects from the Mississippi Gulf Coast, New Orleans, and trailer communities in Louisiana are randomly assigned to a case or control group. Subjects randomized to case group begin their Life Stories session, describing their personal experience of Katrina/Rita. Two raters (from a pool of ten) are assigned to evaluate each subject's taped interview on three attributes: Affect/Attitude, Focus, and Attention.

We will analyze the rating assessment data of video-taped interview sessions to estimate individual raters' discrimination and criteria usage of scoring categories. This will not only provide information on rater characteristics, but can also be used as a weighting variable to improve precision of primary outcome variables. The latent class Signal Detection Theory (SDT) model (DeCarlo, 2002, 2005) will be used to estimate rater effects.