SavedAdd your journal

Journals / Bayburt Eğitim Fakültesi Dergisi / 2026 / Cilt: 21 Sayı: 51

Artificial Intelligence Versus Human Raters in English Writing Assessment: The Multifaceted Rasch Model

İngilizce Yazmanın Değerlendirilmesinde Yapay Zekâ ve İnsan Puanlayıcıların Karşılaştırılması: Çok Yüzeyli Rasch Modeli

Hatice Gürdil(MEB, Atam İlkokulu)0000-0002-0079-3202
Hatice Özlem Anadol(GAZI UNIVERSITY, ACADEMY OF FOREIGN LANGUAGES, DEPARTMENT OF FOREIGN LANGUAGE)0000-0002-2881-9806
Salih Salihoglu(Department of Industrial and Systems Engineering, University of Miami)0000-0002-0992-6577
İsmail Hakkı Karaman(MIDDLE EAST TECHNICAL UNIVERSITY, FACULTY OF ENGINEERING, DEPARTMENT OF INDUSTRIAL ENGINEERING)0000-0003-3529-4857
Şakir Çınkır(ANKARA UNIVERSITY, FACULTY OF EDUCATIONAL SCIENCES, DEPARTMENT OF EDUCATION MANAGEMENT AND POLICY)
Pages
1187–1210
Publication Date
2026-09-26
🏛️Bibliographic metadata and full text access for this article are provided via DergiPark.
DergiPark Record ↗
✨

AI Summary & Key Takeaways

AI Insight
⚡ TL;DR(Too Long; Didn't Read)

Findings indicate that ChatGPT 3.5 outperforms Gemini and YouChat regarding accuracy, precision, and recall, with an overall high IRR among all raters.

📌 3 Key Takeaways

🎯
Objective & Hypothesis:This study examines the inter-rater reliability (IRR) of three AI tools (ChatGPT 3.5, Gemini, YouChat) and four human raters in evaluating English writing skills using the multifaceted Rasch model.
📊
Key Findings & Data:The sample comprised 206 students whose responses were evaluated using a holistic scoring rubric based on the Common European Framework of Reference for Languages (CEFR) writing competencies.
💡
Conclusion & Impact:Findings indicate that ChatGPT 3.5 outperforms Gemini and YouChat regarding accuracy, precision, and recall, with an overall high IRR among all raters.
🔬Design:Modeling & Simulation
📍Scope:Comprehensive Sample
⚙️Variables:puanlayıcılar arası güvenirlik · yapay zekâ (YZ) · Çok Yüzeyli Rasch Modeli

Abstract

This study examines the inter-rater reliability (IRR) of three AI tools (ChatGPT 3.5, Gemini, YouChat) and four human raters in evaluating English writing skills using the multifaceted Rasch model. The evaluation focuses on the consistency and dependability of scores assigned by the AI tools compared to human judgment. The sample comprised 206 students whose responses were evaluated using a holistic scoring rubric based on the Common European Framework of Reference for Languages (CEFR) writing competencies. Findings indicate that ChatGPT 3.5 outperforms Gemini and YouChat regarding accuracy, precision, and recall, with an overall high IRR among all raters. The results suggest that AI tools can effectively supplement human raters, providing consistent and reliable assessments, which has significant implications for educational evaluations and the potential integration of AI in various scoring scenarios. The study contributes to understanding the role of AI in educational assessment and emphasizes the potential of AI tools to improve the scalability and efficiency of scoring processes. The study also discusses the importance of AI tools in providing prompt feedback, analyzing large datasets, and ensuring scoring consistency across different contexts. It also highlights the need for hybrid models that combine AI's consistency with human assessors' contextual understanding.

Özet

Bu çalışma, İngilizce yazma becerilerinin değerlendirilmesinde üç yapay zekâ aracı (ChatGPT 3.5, Gemini, YouChat) ile insan puanlayıcılar arasındaki puanlayıcılar arası güvenirliği Çok Yüzeyli Rasch Modeli ile incelemektedir. 206 öğrenciden toplanan yazma yanıtları, Avrupa Dilleri Ortak Başvuru Çerçevesi temelli bütüncül rubrikle puanlanmış; yapay zekâ araçlarının doğruluk, kesinlik ve duyarlılık ölçütleri hesaplanarak insan yargısıyla tutarlılığı test edilmiştir. Bulgular, tüm puanlayıcılar arasında yüksek düzeyde puanlayıcılar arası güvenirlik elde edildiğini ve ChatGPT 3.5’in Gemini ve YouChat’e kıyasla daha iyi performans gösterdiğini ortaya koymaktadır. Sonuçlar, yapay zekânın hızlı geri bildirim, büyük veriyi işleme ve farklı bağlamlarda puanlama tutarlılığını artırma potansiyeline işaret etmektedir. Bununla birlikte, bağlamı doğru yorumlama ve ayrıntılı, amaca uygun geri bildirim üretme aşamalarında insan uzmanlığı kritik olduğundan, yapay zekânın insan puanlayıcıları tamamladığı hibrit yaklaşımlar önerilmektedir. Standartlaştırılmış yönergeler ve ortak istemler, insan ve yapay zekâ puanlayıcıların aynı koşullarda çalışmasını ve yöntemsel karşılaştırılabilirliği sağlamıştır.

🔖

Cite This Article

Gürdil, H., Anadol, H. Ö., Salihoglu, S., Karaman, İ. H., Çınkır, Ş. (2026). Artificial Intelligence Versus Human Raters in English Writing Assessment: The Multifaceted Rasch Model. Bayburt Eğitim Fakültesi Dergisi, Cilt: 21 Sayı: 51, 1187–1210. https://doi.org/10.35675/befdergi.1788625

📄 Full Text PDF Viewer

📄

PDF görüntüleyici hazırlanıyor...

References (44)

  1. Anastasi, A. (1979). Fields of applied psychology (2nd ed.). McGraw-Hill.
  2. Brennan, R. L. (2001). Generalizability theory. Springer-Verlag
  3. Brookhart, S.M. (2010). How to assess higher-order thinking skills in your classroom. ASCD.
  4. Brookhart, S.M. (2015). Making the most of multiple choice. educational leadership, 73(1), 36 39. https://eric.ed.gov/?id=EJ1075062
  5. Brooks, V. (1980) Improving the reliability of essay marking: a survey of the literature with particular reference to the English language composition. (CSE Research Project Report 5). Leicester University.
  6. Burton, S. J., Sudweeks, R. R., Merrill, P. F., & Wood, B. (1991). How to prepare better multiple-choice test items: Guidelines for university faculty. Bringham Young University Testing Services and the Department of Instructional Science. https://testing.byu.edu/info/handbooks/betteritems.pdf.
  7. Chen, J. (2021). Refining the teacher emotion model: Evidence from a review of literature published between 1985 and 2019. Cambridge Journal of Education, 51(3), 327–357.
  8. Creswell, J. W., & Creswell, J. D. (2017). Research design: Qualitative, quantitative, and mixed methods approaches. SAGE publications.
  9. Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. CBS College Publishing
  10. DeMars, C. E. (2010). Item response theory. Oxford University Press.
  11. Doğan, N. (2013). Yazılı Yoklamalar. H. Atılgan (Ed.), Eğitimde ölçme ve değerlendirme içinde (145-168). Anı Yayıncılık.
  12. Eckes, T. (2023). Introduction to many-facet Rasch measurement: Analyzing and evaluating rater-mediated assessments (2nd rev. ed.). Peter Lang.
  13. Educational Testing Service (1987). ETS standards for quality and fairness.
  14. Gerard, L. F., & Linn, M. C. (2016). Using automated scores of student essays to support teacher guidance in classroom inquiry. Journal of Science Teacher Education, 27(1), 111-129.
  15. Goodwin, L.D. (2001). Interrater agreement and reliability. Measurement in Physical Education and Exercises Science, 5 (1), 13-34. http://dx.doi.org/10.1207/S15327841MPEE0501_2.
  16. Gürdil, H., Soğuksu, Y. B., Salihoğlu, S., & Coşkun, F. (2025). Eğitimde ölçmede yapay zekânın entegrasyonu: Madde tepki kuramı kapsamında veri üretiminde ChatGPT’nin etkililiği. Trakya Eğitim Dergisi, 15(2), 887–918. https://doi.org/10.24315/tred.1509299
  17. Gürdil, H., Anadol, H. Ö., & Soğuksu, Y. B. (2026). The use of artificial intelligence tools in assessing content validity: A comparative study with human experts. Ankara University Journal of Faculty of Educational Sciences, 59(2), 549–590. https://doi.org/10.30964/auebfd.1807153
  18. Harlen, W. (2007). Assessment of Learning. SAGE Publications.
  19. Harunasari, S. Y. (2023). Examining the effectiveness of AI-integrated approach in EFL writing: A case of ChatGPT. International Journal of Progressive Sciences and Technology(IJPSAT),39(2),357–368. https://ijpsat.org/index.php/ijpsat/article/download/5516/3447
  20. Imran, M., & Almusharraf, N. (2023). Analyzing the role of ChatGPT as a writing assistant at higher education level: A systematic review of the literature. Contemporary Educational Technology, 15(4), ep464. https://doi.org/10.30935/cedtech/13605
  21. Jackaria, P. M., Hajan, B. H., & Mastul, A. H. (2024). A comparative analysis of the rating of college students' essays by ChatGPT versus human raters. International Journal of Learning, Teaching and Educational Research, 23(2), 478-492. https://doi.org/10.26803/ijlter.23.2.23
  22. Johnson, R. L., Penny, J. A., & Gordon, B. (2008). Assessing performance: Designing, scoring, and validating performance tasks. Guilford Press.
  23. Jonsson A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2(2), 130-144. https://doi.org/ 10.1016/j.edurev.2007.05.002.
  24. Karakaya, İ. (2018). Uygulamalı ölçme ve değerlendirme. Pegem Akademi. https://doi.org/10.14527/9786052412367
  25. Kim, H., Baghestani, S., Yin, S., Karatay, Y., Kurt, S., Beck, J., & Karatay, L. (2024). ChatGPT for writing evaluation: Examining the accuracy and reliability of AI-generated scores compared to human raters. In C. A. Chapelle, G. H. Beckett, & J. Ranalli (Eds.), Exploring artificial intelligence in applied linguistics (pp. 73–95). Iowa State University Digital Press. https://doi.org/10.31274/isudp.2024.154.06
  26. Kollias, C. (2025). AVANT STAMP CEFR English proficiency test: Reading and listening cut score study—Technical report. Polytomous Limited / Avant Assessment. https://cdnprodwpv2.avantassessment.com/wp-content/uploads/AVANT-STAMP-for-CEFR-Cut-Scores-Technical-Report_final.pdf
  27. Krippendorff, K. (2019). Content analysis: An introduction to its methodology. SAGE. https://doi.org/10.4135/9781071878781
  28. Kutlu, Ö., Doğan, C. D., & Karakaya, İ. (2014). Ölçme ve değerlendirme: Performansa ve portfolyoya dayalı durum belirleme. Pegem Akademi.
  29. Lamprianou, J. (2004) Marking quality assurance procedures: identifying good practice internationally. [Report prepared for the National Assessment Agency].
  30. Linacre, J. M. (2002). Number of person or item strata. Rasch Measurement Transactions, 16, 888.
  31. Linacre, J. M. (2004). Rasch model estimation: Further topics. Journal of Applied Measurement, 5(1), 95-110.
  32. McMurtrie B (2022, December 13). AI and the future of undergraduate writing. The Chronicle of Higher Education. https://www.chronicle.com/article/ai-and-the-future-of-undergraduate-writing
  33. Moskal, B. M. (2000). Scoring rubrics: What, when and how? Practical Assessment, Research & Evaluation, 7(3), 1–7. https://doi.org/10.7275/q7rm-gg74
  34. O'Connell, B., De Lange, P., Freeman, M., Hancock, P., Abraham, A., Howieson, B., & Watty, K. (2016). Does calibration reduce variability in the assessment of accounting learning outcomes? Assessment & Evaluation in Higher Education, 41(3), 331–349. https://doi.org/10.1080/02602938.2015.1008398
  35. Popham, W.J. (2000). Modern educational measurement: practical guidelines for educational leaders (3rd ed.). Allyn & Bacon.
  36. Provost, F., & Fawcett, T. (2001). Robust classification for imprecise environments. Machine learning, 42, 203-231.
  37. Sabharwal, N., & Agrawal, A. (2021). Hands-on question answering systems with BERT: Applications in neural networks and natural language processing. Apress. https://doi.org/10.1007/978-1-4842-6664-9
  38. Sokolova, M., & Lapalme, G. (2009). A systematic analysis of performance measures for classification tasks. Information Processing & Management, 45(4), 427-437.
  39. Tabachnick, B.G., & Fidell, L.S. (2012). Using multivariate statistics (6th ed.). Allyn & Bacon.
  40. Weng, X., Xia, Q., Gu, M., Rajaram, K., & Chiu, T. K. F. (2024). Assessment and learning outcomes for generative AI in higher education: A scoping review on current research status and trends. Australasian Journal of Educational Technology, 40(6), 37–55. https://doi.org/10.14742/ajet.9540
  41. Woehr, D.J., & Huffuct, A.I. (1994). Rater training for performance appraisal. A quantitative review. Journal of Occupational and Organizational Psychology, 67(3), 189-205.
  42. Xia, Q., Weng, X., Ouyang, F., Lin, T. J., & Chiu, T. K. F. (2024). A scoping review on how generative artificial intelligence transforms assessment in higher education. International Journal of Educational Technology in Higher Education, 21, Article 40. https://doi.org/10.1186/s41239-024-00468-z
  43. Yen, W. M. (1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8(2), 125–145. https://doi.org/10.1177/014662168400800201
  44. Zacharis, G., & Papadakis, S. (2025). Can AI grade like a human? Validity, reliability, and fairness in university coursework assessment. Educational Process: International Journal, 19, e2025591. https://doi.org/10.22521/edupij.2025.19.591