Journals / International Journal of Assessment Tools in Education / 2022 / Cilt: 9 - Sayı: 2

Reliability of Ratings of Multidimensional Fluency Scale with Many-Facet Rasch Model

Pages
470–491
Publication Date
2022-10-18
DOI
—
🏛️Bibliographic metadata and full text access for this article are provided via DergiPark.
DergiPark Record ↗
✨

AI Summary & Key Takeaways

AI Insight
⚡ TL;DR(Too Long; Didn't Read)

It was found that raters were ranked reliably in terms of severity/leniency, and that their levels of severity/leniency differed from each other.

📌 3 Key Takeaways

🎯
Objective & Hypothesis:The aim of this study was to determine whether the reliability of raters was provided by assessing reading prosody using the Multidimensional Fluency Scale (MDFS).
📊
Key Findings & Data:The study was completed with a cross-sectional design, and in line with this, the prosodic reading skills of 41 fifth-grade students were rated by elementary school classroom teachers and Turkish language arts teachers using the MDFS.
💡
Conclusion & Impact:It was found that raters were ranked reliably in terms of severity/leniency, and that their levels of severity/leniency differed from each other.
🔬Design:Modeling & Simulation
📍Scope:Comprehensive Sample
⚙️Variables:Rater bias · Rubric · Reading fluency

Abstract

The aim of this study was to determine whether the reliability of raters was provided by assessing reading prosody using the Multidimensional Fluency Scale (MDFS). The study was completed with a cross-sectional design, and in line with this, the prosodic reading skills of 41 fifth-grade students were rated by elementary school classroom teachers and Turkish language arts teachers using the MDFS. Data obtained from the ratings were analyzed with the many-facet Rasch model (MFRM). When the findings are investigated, the reading prosody rubric used in the research served the purposes of the reading prosody criteria, the sub-dimensions of the rubric could be reliably differentiated, the determined criteria were reliable, and the criteria categories appear to be adequate. Additionally, the severity and leniency of raters were found to differ, and Turkish language arts teachers were found to perform more severe ratings than classroom teachers. It was found that raters were ranked reliably in terms of severity/leniency, and that their levels of severity/leniency differed from each other. Another result obtained is that the prosody criterion that students completed with the most difficulty was phrasing. Therefore, it was concluded that the MDFS is a reliable rubric and that researchers and teachers can reliably use it to assess prosodic reading skills.

Özet

The aim of this study was to determine whether the reliability of raters was provided by assessing reading prosody using the Multidimensional Fluency Scale (MDFS). The study was completed with a cross-sectional design, and in line with this, the prosodic reading skills of 41 fifth-grade students were rated by elementary school classroom teachers and Turkish language arts teachers using the MDFS. Data obtained from the ratings were analyzed with the many-facet Rasch model (MFRM). When the findings are investigated, the reading prosody rubric used in the research served the purposes of the reading prosody criteria, the sub-dimensions of the rubric could be reliably differentiated, the determined criteria were reliable, and the criteria categories appear to be adequate. Additionally, the severity and leniency of raters were found to differ, and Turkish language arts teachers were found to perform more severe ratings than classroom teachers. It was found that raters were ranked reliably in terms of severity/leniency, and that their levels of severity/leniency differed from each other. Another result obtained is that the prosody criterion that students completed with the most difficulty was phrasing. Therefore, it was concluded that the MDFS is a reliable rubric and that researchers and teachers can reliably use it to assess prosodic reading skills.

🔖

Cite This Article

ARIKAN, Ç. A., UYSAL, P. K., BİLGE, H., YILDIRIM, K. (2022). Reliability of Ratings of Multidimensional Fluency Scale with Many-Facet Rasch Model. International Journal of Assessment Tools in Education, Cilt: 9 - Sayı: 2, 470–491. https://ajindex.com/articles/540023

📄 Full Text PDF Viewer

📄

PDF görüntüleyici hazırlanıyor...

References (7)

  1. Acar, M., & Anıl, D. (2009). Sınıf öğretmenlerinin performans değerlendirme sürecindeki değerlendirme yöntemlerini kullanabilme yeterlilikleri, karşılaştıkları sorunlar ve çözüm önerileri [Classroom teacher evaluation methods to use in the performance assessment process qualification of able, they comparison problems and solution proposals]. Journal of TUBAV Science, 2(3), 354-363.
  2. American Psychological Associaton. (2015). Halo effect. In APA dictionary of psychology (2nd ed., p. 667). Allington, R.L. (1983). Fluency: the neglected reading goal. The Reading Teacher, 36(6), 556-561.
  3. Armut, M., & Türkyılmaz, M. (2017). Ortaokul öğrencilerinin okuma becerileri üzerine bir inceleme [An investigation on reading skills of middle school students]. Erzincan University Journal of Education Faculty, 20(1), 217 236. https://doi.org/10.17556/erziefd.330587
  4. Aşıkcan, M. (2019). Üçüncü sınıf öğrencilerinin akıcı okuma becerilerinin geliştirilmesine yönelik bir eylem araştırması [An action research on ımproving fluent reading skills of third-grade primary school students] [Unpublished doctoral dissertation]. Necmettin Erbakan University. Baird, J.A., Hayes, M., Johnson, R., Johnson, S., & Lamprianou, I. (2013). Marker effects and examination reliability. A Comparative exploration from the perspectives of generalisability theory, Rash model and multilevel modelling. Oxford: University of Oxford for Educational Assessment. Retrieved from http://dera.ioe.ac.uk/17683/1/2013-01-21-marker-effects-and-examinationreliability.pdf
  5. Barkaoui, K. (2008). Effects of scoring method and rater experience on ESL essay rating processes and outcomes [Doctoral Dissertation, Available from ProOuest Dissertations and Theses database]. UMI No: 304360302. Barrett, S., (2001). The impact of training on rater variability. International Education Journal, 2(1), 49-58.
  6. Baştuğ, M., (2021). Akıcı okumayı geliştirme: kavramlar, uygulamalar, değerlendirmeler [Developing reading fluency: concepts, practices, assessments]. Pegem Akademi Yayıncılık Baştuğ, M., & Keskin, H.K. (2012). Akıcı okuma becerileri ile anlama düzeyleri (basit ve çıkarımsal) arasındaki ilişki [The relationship between fluent reading skills and comprehension level (literal and ınferential)]. Ahi Evran University Journal of Kırşehir Education Faculty, 13(3), 227-244.
  7. Baştürk, R., & Işıkoglu, N. (2008). Analyzing process quality of early childhood education with many facet rash measurement model. Educational Sciences:Theory and Practice