Beyond Polemics: The Merits and Challenges of Assessing Intercoder Agreement in a Qualitative Research Study

Most researchers agree that assessing intercoder agreement is a key element of qualitative data analysis. However, there is little agreement about how best to assess it. The use of quantitative measures to assess intercoder reliability has been criticized on a variety of grounds. Yet, few alternatives exist for ensuring consistent and reliable coding of complex qualitative data. This paper describes the merits and challenges of assessing intercoder agreement using quantitative means (e.g. kappa scores) for the complex qualitative data from the PHRESH.comm project and offers additional practical tools for conducting such assessments. We describe the processes of developing the codebook, applying codes, assessing reliability, and reconciling coding differences, including the project's development of a typology of coding disagreements during the process of reconciliation. We also present strategies to address coder disagreements, such as limiting the number and nature of codes, reading and discussing text related to disagreements in order to refine code definitions and develop a common understanding of them, and discussing the nature of segmentation and the relative importance of including context versus potentially “missing” text. Finally, we outline our experience with consensus coding and adjunct coding review as alternative means to maintain coding rigor with less than optimal kappa scores from complex data.