209748 Constructed response scoring application based on signal detection theory to improve precision of rater effect: Analysis of Life Story project for victims of Hurricane Katrina

Monday, November 9, 2009: 8:50 AM

Yoon Soo Park, MS , National Center for Disaster Preparedness, Mailman School of Public Health, Columbia University, New York, NY
Akilah Banister, MPH , National Center for Disaster Preparedness, Mailman School of Public Health, Columbia University, New York, NY
David M. Abramson, PhD MPH , National Center for Disaster Preparedness, Columbia University, New York, NY
Unlike multiple choice (MC) items which can be objectively scored as right or wrong, constructed response (CR) items are evaluated on a range of ordinal values (scores) by multiple raters. Individual rater's leniency and strictness influence CR scoring. Therefore, it is of interest to understand how these assessments behave—whether they vary by individual raters or how well these ratings conform to the test-taker's true ability.

This proposal is based on the data from Life Story Project examining the effect of the Testimony Therapy and oral history in the form of story-telling, to improve recovery of the Hurricane Katrina and Rita-afflicted Gulf Coast residents. 100 subjects from the Mississippi Gulf Coast, New Orleans, and trailer communities in Louisiana are randomly assigned to a case or control group. Subjects randomized to case group begin their Life Stories session, describing their personal experience of Katrina/Rita. Two raters (from a pool of ten) are assigned to evaluate each subject's taped interview on three attributes: Affect/Attitude, Focus, and Attention.

We will analyze the rating assessment data of video-taped interview sessions to estimate individual raters' discrimination and criteria usage of scoring categories. This will not only provide information on rater characteristics, but can also be used as a weighting variable to improve precision of primary outcome variables. The latent class Signal Detection Theory (SDT) model (DeCarlo, 2002, 2005) will be used to estimate rater effects.

Learning Objectives:
Describe how to statistically test rater differences. Discuss how to use the results to refine interpretation of the outcome variable.

Presenting author's disclosure statement:

Qualified on the content I am responsible for because: I am a Ph.D. Candidate in the Measurement, Evaluation, and Statistics program at Columbia University. My dissertation analyzes rater characteristics in constructed response scoring. I am also the Data Manager/Analyst at the National Center for Disaster Preparedness at Columbia University where the data for the current study was collected and analyzed.
Any relevant financial relationships? No

I agree to comply with the American Public Health Association Conflict of Interest and Commercial Support Guidelines, and to disclose to the participants any off-label or experimental uses of a commercial product or service discussed in my presentation.