Kappa coefficients for dichotomous-nominal classifications

Matthijs J. Warrens*

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review

Abstract

Two types of nominal classifications are distinguished, namely regular nominal classifications and dichotomous-nominal classifications. The first type does not include an 'absence' category (for example, no disorder), whereas the second type does include an 'absence' category. Cohen's unweighted kappa can be used to quantify agreement between two regular nominal classifications with the same categories, but there are no coefficients for assessing agreement between two dichotomous-nominal classifications. Kappa coefficients for dichotomous-nominal classifications with identical categories are defined. All coefficients proposed belong to a one-parameter family. It is studied how the coefficients for dichotomous-nominal classifications are related and if the values of the coefficients depend on the number of categories. It turns out that the values of the new kappa coefficients can be strictly ordered in precisely two ways. The orderings suggest that the new coefficients are measuring the same thing, but to a different extent. If one accepts the use of magnitude guidelines, it is recommended to use stricter criteria for the new coefficients that tend to produce higher values.

Original languageEnglish
Pages (from-to)193–208
Number of pages16
JournalAdvances in Data Analysis and Classification
Volume15
Early online date7-Apr-2020
DOIs
Publication statusPublished - Mar-2021

Keywords

  • Cohen's unweighted kappa
  • Weighted kappa
  • Unordered classifications
  • Agreement studies
  • Inter-rater agreement
  • Inter-rater reliability
  • WEIGHTED KAPPA
  • AGREEMENT
  • RELIABILITY

Cite this