AI Summary & Key Takeaways
AI InsightModeling agreement can provide more information than kappa.
📌 3 Key Takeaways
Abstract
Agreement between 2 or more independent raters evaluating the same items and same scale can be measured by kappa coefficient. In recent years, modeling agreement among raters rather than summarizing indices has been preferred. In this study, the disadvantages of kappa are reviewed. Agreement models are introduced and these models are applied to a real data set. Materials and methods: Three pathologists classified each of 118 slides in terms of carcinoma in situ of the uterine cervix, based on the most involved lesions. Using log-linear agreement models, agreement between 3 pathologists according to their evaluations was investigated. Results: Coefficient of kappa was found to be 0.48 among the 3 pathologists, which indicates a moderate agreement. Models were applied to the data. The agreement parameter was estimated for the best model among models. The probability of giving the same decision by the 3 pathologists was 2.5 times higher than that of giving a different decision. Conclusion: Log-linear models can be used to measure the agreement among more than 2 raters. Modeling agreement can provide more information than kappa.
Özet
Agreement between 2 or more independent raters evaluating the same items and same scale can be measured by kappa coefficient. In recent years, modeling agreement among raters rather than summarizing indices has been preferred. In this study, the disadvantages of kappa are reviewed. Agreement models are introduced and these models are applied to a real data set. Materials and methods: Three pathologists classified each of 118 slides in terms of carcinoma in situ of the uterine cervix, based on the most involved lesions. Using log-linear agreement models, agreement between 3 pathologists according to their evaluations was investigated. Results: Coefficient of kappa was found to be 0.48 among the 3 pathologists, which indicates a moderate agreement. Models were applied to the data. The agreement parameter was estimated for the best model among models. The probability of giving the same decision by the 3 pathologists was 2.5 times higher than that of giving a different decision. Conclusion: Log-linear models can be used to measure the agreement among more than 2 raters. Modeling agreement can provide more information than kappa.