SavedAdd your journal

Journals / Kastamonu Üniversitesi Kastamonu Eğitim Dergisi / 2013 / Cilt: 21 - Sayı: 2

Comparison of interrater agreement calculated with generalizability theory and logistic regression

Genellenebilirlik kuramı ve lojistik regresyona dayalı hesaplanan puanlayıcılar arası tutarlığın karşılaştırılması

Derya ESER ÇAKICI
Selahattin GELBAL
Pages
421–438
Publication Date
2022-10-18
DOI
—
🏛️Bibliographic metadata and full text access for this article are provided via DergiPark.
DergiPark Record ↗
✨

AI Summary & Key Takeaways

AI Insight
⚡ TL;DR(Too Long; Didn't Read)

According to the results; generalizability theory and logisticregression analysis both give parallel results; but logistic regression analysis do not generate as sensitive outputs as G theory and is a superficial statistics in referenceto G theory.

📌 3 Key Takeaways

🎯
Objective & Hypothesis:The purpose of this study is to determine the rater agreement and interrater agreement of a performance rating using G theory and LR analysis.
📊
Key Findings & Data:In this research 106 students are asked to answer a 15 item scale and the answers have been rated by three raters with an analytic rubric.
💡
Conclusion & Impact:According to the results; generalizability theory and logisticregression analysis both give parallel results; but logistic regression analysis do not generate as sensitive outputs as G theory and is a superficial statistics in referenceto G theory.
🔬Design:Research Article
📍Scope:Comprehensive Sample
⚙️Variables:Domain Parameters

Abstract

The purpose of this study is to determine the rater agreement and interrater agreement of a performance rating using G theory and LR analysis. In this research 106 students are asked to answer a 15 item scale and the answers have been rated by three raters with an analytic rubric. Data set has beenbeen analyzed with G theory and LR analysis based on the items of the test and the whole test. According to analyses; rater variance component and total variance explaining percentage of G Study and classification percentage of LR analysis have been interpreted. According to the results; generalizability theory and logisticregression analysis both give parallel results; but logistic regression analysis do not generate as sensitive outputs as G theory and is a superficial statistics in referenceto G theory.

Özet

Araştırmanın amacı G Kuramı ve LR analizinden yararlanarak gerçekleştirilen bir performans puanlamada ortaya çıkan puanlayıcı tutarlığını belirlemek ve karşılaştırmaktır. Araştırmada 106 öğrenciye 15 maddelik bir ölçme aracı verilmiş, öğrencilerin verdikleri cevaplar üç puanlayıcı tarafından dereceli puanlama anahtarı kullanılarak puanlanmıştır. Puanlama ile elde edilen veri seti, G kuramı ve lojistik regresyon analizi ile madde bazında ve testin tamamına dayalı olarak analiz edilmiştir. Analiz sonucunda G kuramı ile elde edilen puanlayıcıvaryans bileşenleri ve toplam varyansı açıklama yüzdeleri ile LR analizi ile elde edilen sınıflama yüzdeleri yorumlanmıştır. Elde edilen bulgulara dayalı olarak G Kuramı ve LR analizinin puanlayıcılar arası tutarlığı belirlemede paralel sonuçlar ürettiği, ancak lojistik regresyon analizinin G kuramı kadar hassas çıktılar vermediği ve G kuramına göre daha yüzeysel bir istatistik olduğu sonucuna varılmıştır.

🔖

Cite This Article

ÇAKICI, D. E., GELBAL, S. (2022). Comparison of interrater agreement calculated with generalizability theory and logistic regression. Kastamonu Üniversitesi Kastamonu Eğitim Dergisi, Cilt: 21 - Sayı: 2, 421–438. https://ajindex.com/articles/572227

📄 Full Text PDF Viewer

📄

PDF görüntüleyici hazırlanıyor...

References (11)

  1. AHMANN, J. S. and GLOCK, M. D. (1971). Measuring And Evaluating Educational Achievement, Boston: Allyn and Bacon BRENNAN, R. L. (2001). Generalizability Theory. New York: Springer- Verlog. CROCKER, L. ve ALGİNA, J. (1986). Introduction To Classical And Modern Test Theory, New York: Wadsworth. ÇOKLUK, Ö., ŞEKERCİOĞLU, G., BÜYÜKÖZTÜRK, Ş. (2010). Sosyal Bilimler İçin Çok Değişkenli İstatistiksel Analiz: SPSS ve LISREL Uygulamaları. Ankara:
  2. PegemA Yayınları.
  3. GLESER, G.C., CRONBACH, L. J., RAJARATHAM N. (1965). Generalizability Of Scores Influenced By Multıple Sources Of Varıance. Psychometrika, 30(4),395-418.
  4. HOSMER, D. W., LEMESHOW, S. (2000).
  5. Applied Logistic Regression, Second Edition. USA: A Wiley Inter Science Publication. KAYRİ, M. Ve ÇOKLUK, Ö. (2010). Using Multinominal Logistic Regression Analysis In Artificial Neural Networt: An Application. Ozean Journal Of Applied Sciences, 3(2), 259-268.
  6. KING, J.E. (2003). Running A Best-Subsets Logistic Regression: An Alternative To Stepwise Methods, Educaional And Psychological Measurement, 63, 392-403.
  7. LANE, S. and STONE, C. A. (2006).
  8. Educational Measurment, Fourth Editon ( Ed: Robert Brennan ) USA:American Council On Education Praeger. MEB (2010). PISA 2006 Nihai Raporu Milli Eğitim Bakanlığı Eğitimi Araştırma ve Geliştirme Dairesi Başkanlığı. MENARD, S. W. (2002).
  9. Applied Logistic Regression Analysis, Thous and Oaks.: Sage. ÖZTÜRKCAN; M. (2009). Regresyon Analizi. İstanbul: Maltepe Üniversitesi Yayınları, 40. PALM, T. (2008). Performance Assessment and Authentic Assessment: A Conceptual Analysis of the Literature. Practical Assessment, Research & Evaluation, 13(4), 1-11.
  10. PEDHAZUR, E. J. (1997). Multiple Regression In Behavioral Research Explanation And Prediction (3rd. Ed.).Australia: Wadsworth, Thomson Learning. RENTZ, J. O. (1987). Generalizability Theory: A Comprehensive Method For Assessing And Improving The Dependability of Marketing Measures. Journal Of Marketing Research, 24(1), 19-28. SHAVELSON,R.J.,WEBB,N.www.stanford.edu/dept/SUSE/SEAL/Reports-Papers/Generalizability% 20Theory-ESM-Final.doc. adresinden 19.11.2010 tarihinde alınmıştır. SMITH, V. E., KULIKOWICH, J. M. (2004). An Application Of Generalizability Theory And Many Facet Rasch Measurement Using A Complex Problem Solving Skills Assesment. Educational and Psychological Measurement, 64(4), 617-639.
  11. TABACHNICK, B. G., FIDELL, L. S. (2007). Using Multivariate Statistics (5th Ed.).USA:Education Pearson Education Inc. TOBAR, D. A., STEGNER