SavedAdd your journal

Journals / Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi / 2019 / Cilt: 10 - Sayı: 3

Examination of the Reliability of the Measurements Regarding the Written Expression Skills According to Different Test Theories

Şeref TAN
Merve YILDIRIM SEHERYELİ
Pages
327–347
Publication Date
2022-10-12
DOI
—
🏛️Bibliographic metadata and full text access for this article are provided via DergiPark.
DergiPark Record ↗
✨

AI Summary & Key Takeaways

AI Insight
⚡ TL;DR(Too Long; Didn't Read)

CTT results showed that Eta coefficient revealed that there was no difference between the scoring of the raters (η = .926); Cronbach Alpha coefficients were over .88.

📌 3 Key Takeaways

🎯
Objective & Hypothesis:The aim of the study is to examine the reliability estimations of written expression skills analytical rubric based on the Classical Test Theory (CTT), Generalizability Theory (GT) and Item Response Theory (IRT) which differ in their field of study.
📊
Key Findings & Data:In this descriptive study, the stories of the 523 students in the study group were scored by seven raters.
💡
Conclusion & Impact:CTT results showed that Eta coefficient revealed that there was no difference between the scoring of the raters (η = .926); Cronbach Alpha coefficients were over .88.
🔬Design:Modeling & Simulation
📍Scope:Comprehensive Sample
⚙️Variables:Domain Parameters

Abstract

The aim of the study is to examine the reliability estimations of written expression skills analytical rubric based on the Classical Test Theory (CTT), Generalizability Theory (GT) and Item Response Theory (IRT) which differ in their field of study. In this descriptive study, the stories of the 523 students in the study group were scored by seven raters. CTT results showed that Eta coefficient revealed that there was no difference between the scoring of the raters (η = .926); Cronbach Alpha coefficients were over .88. GT results showed that G and Phi coefficients were over .97. The students’ expected differentiation emerged, the difficulty levels of the criteria did not change from one student to another, and the consistency between the scores among raters was excellent. In the Item Response Theory, parameters were estimated according to Samejima’s (1969) Graded Response Model and item discrimination differed according to the different raters. According to b parameters, for all the raters; individuals are expected to be at least -2.35, -0.80, 0.41 ability level in order to be scored higher than 0, 1 or 2 categories respectively with .50 probability. Marginal reliability coefficients were quite high (around .93). The Fisher Z’ statistic was calculated for the significance of the difference between all reliability estimates. GT revealed more detailed information than CTT in the explanation of error variance sources and determination of reliability; while IRT provided more detailed information than CTT in determining the item-level error estimations and the ability level. There was a significant difference between the estimated parameters of CTT and GT in interrater reliability (p .05).

🔖

Cite This Article

TAN, Ş., SEHERYELİ, M. Y. (2022). Examination of the Reliability of the Measurements Regarding the Written Expression Skills According to Different Test Theories. Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi, Cilt: 10 - Sayı: 3, 327–347. https://ajindex.com/articles/497199

📄 Full Text PDF Viewer

📄

PDF görüntüleyici hazırlanıyor...

References (13)

  1. Arsan, N. (2012). Buz pateninde hakem değerlendirmelerinin genellenebilirlik kuramı ve Rasch modeli ile incelenmesi (Yayımlanmamış doktora tezi). Hacettepe Üniversitesi Sosyal Bilimler Enstitüsü, Ankara.
  2. Atılgan, H. (2005). Genellenebilirlik kuramı ve puanlayıcılar arası güvenirlik için örnek bir uygulama. Eğitim Bilimleri ve Uygulama, 4(7), 95-108.
  3. Ayala, R. J. (2009). The theory and practice of item response theory. USA: The Guildford Press.
  4. Bağcı, V. (2015). Matematiksel muhakeme becerisinin ölçülmesinde klasik test kuramı ile genellenebilirlik kuramındaki farklı desenlerin karşılaştırılması (Yayımlanmamış yüksek lisans tezi). Gazi Üniversitesi Eğitim Bilimleri Enstitüsü, Ankara.
  5. Baker, F. B. (2016). Madde tepki kuramının temelleri (Çev. N. Güler ve M. İlhan). Ankara:
  6. Pegem Akademi. Baykul, Y. (2010). Eğitimde ve psikolojide ölçme: Klasik test teorisi ve uygulaması. Ankara:
  7. Pegem Akademi. Brennan, R. L. (2011). Generalizability theory and classical test theory. Applied Measurement And Education, 24, 1-21. doi: 10.1080/08957347.2011.532417 Brennan, R.L. (2000). Performance assessments from the perspective of generalizability theory. Applied Psychological Measurement, 24(4), 339–353.
  8. Büyükkıdık, S. (2012). Problem çözme becerisinin değerlendirilmesinde puanlayıcılar arası güvenirliğin klasik test kuramı ve genellenebilirlik kuramına göre karşılaştırılması (Yayımlanmamış yüksek lisans tezi). Hacettepe Üniversitesi Sosyal Bilimler Enstitüsü, Ankara.
  9. Cardinet, J., Johnson, S., & Pini, G. (2010). Applying generalizability theory using EduG. USA: Routledge-Taylor & Francis Group. Çelen, Ü., & Aybek, E. C. (2013). Öğrenci başarısının öğretmen yapımı bir testle klasik test kuramı ve madde tepki kuramı yöntemleriyle elde edilen puanlara göre karşılaştırılması. Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi, 4(2), 64-75.
  10. Çokluk, Ö., Şekercioğlu, G., & Büyüköztürk, Ş. (2014). Sosyal bilimler için çok değişkenli istatistik spss ve lisrel uyglamaları. Ankara:
  11. Pegem Akademi. Crocker, L., & Algina, J. (2006). Introduction to classical and modern test theory. USA: Cengage Learning. Deliceoğlu, G. (2009). Futbol yetilerine ilişkin dereceleme ölçeğinin genellenebilirlik ve klasik test kuramına dayalı güvenirliklerinin karşılaştırılması (Yayımlanmış doktora tezi). Ankara Üniversitesi Eğitim Bilimleri Enstitüsü, Ankara.
  12. DeMars, C. (2016). Madde tepki kuramı (Çev. E. H. Özberk ve H. Kelecioğlu). Ankara:
  13. Nobel Akademi. Doğan, N., & Tezbaşaran, A. A. (2003). Klasik tes