Cognitive Diagnostic Modeling Approach to Assess the Mastery and the Prevalence of Specific Competencies in a Basic Emergency Preparedness Course

Evaluators of public health courses are often interested in identifying whether their participants mastered specific competencies or skills following an assessment. Although test developers base their item construction to diagnose specific skills, direct statistical inference of such information can be biased. If estimating the mastery of specific skills is the goal rather than a holistic measurement of a latent ability, traditional methods of item analysis can often lead to inaccuracies if a given item simultaneously diagnoses multiple competencies. This study shows the advantages of employing a Cognitive Diagnostic Model (CDM) to address fine-grained competencies, which contrasts from other models that can only examine item-level information. Using the CDM-based Deterministic, Inputs, Noisy, “and” Gate (DINA; Junker & Sijtsma, 2001) model, one of the most parsimonious and interpretable CDMs developed, we estimate competency mastery by combining information from participants' response as well as a Boolean algebra-based incidence matrix (i.e., Q-Matrix) that represents the relationships between items and competencies. An empirical data set (n=5,559) containing the item-level responses to the performance of a basic emergency preparedness course was analyzed to demonstrate this application. DINA parameters were estimated using Markov chain Monte Carlo, and both conjunctive and stochastic elements of the model were examined. Analyses using other latent trait models such as Item Response Theory (IRT) models are also conducted for comparison. Results showed there was rich information obtained from using the CDM method that can directly aid policy makers and curriculum developers to enhance the effectiveness of the course.