KaydedilenlerDerginizi ekleyin

Dergiler / Sakarya University Journal of Computer and Information Sciences / 2026 / Volume: 9 Issue: 3

Evaluating Large Language Models in Turkish Short Answer Scoring: Validity, Reliability, and Fairness Perspectives

Abdulkadir Kara(BAYBURT UNIVERSITY, DISTANT EDUCATION APPLICATION AND RESEARCH CENTER)0000-0003-3255-1408
Serkan Yıldırım(ATATURK UNIVERSITY)0000-0002-8277-5963
Sayfa
—
Yayın Tarihi
2026-06-30
🏛️Bu makalenin bibliyografik verileri ve tam metin bağlantısı DergiPark üzerinden sağlanmıştır.
DergiPark Kaydı ↗
✨

Yapay Zeka Hızlı Bakış & Çıkarımlar

AI Insight
⚡ TL;DR(1 Cümlede Hızlı Çıkarım)

The results indicate that LLMs can support teachers in formative assessment with properly structured rubrics, but ethical oversight and pedagogical responsibility remain indispensable in final decisions.

📌 3 Maddede Bu Makale

🎯
Amaç & Hipotez:This study examines the performance of large language models (LLMs) in Turkish short-answer assessments within the measurement and evaluation theory framework.
📊
Temel Bulgular & Veri:Formulating rubrics with clear and concrete performance indicators increases model-human alignment and assessment fairness.
💡
Sonuç & Katkı:The results indicate that LLMs can support teachers in formative assessment with properly structured rubrics, but ethical oversight and pedagogical responsibility remain indispensable in final decisions.
🔬Tür:Modelleme & Sayısal Simülasyon
📍Kapsam:Genişletilmiş Akademik Örneklem
⚙️Değişkenler:Artificial intelligence in education · Automated scoring · Evaluation rubric

Abstract

This study examines the performance of large language models (LLMs) in Turkish short-answer assessments within the measurement and evaluation theory framework. The GPT, Gemini, Gemma, and LLaMA models were evaluated under zero-shot and one-shot conditions with rubric support. The results show that LLMs have high internal consistency, but decision reliability can vary depending on prompt format and example sensitivity. Formulating rubrics with clear and concrete performance indicators increases model-human alignment and assessment fairness. Furthermore, error direction analyses revealed that models can exhibit systematic low-scoring tendencies. The results indicate that LLMs can support teachers in formative assessment with properly structured rubrics, but ethical oversight and pedagogical responsibility remain indispensable in final decisions.

🔖

Bu Makaleye Atıf Yap

Kara, A., Yıldırım, S. (2026). Evaluating Large Language Models in Turkish Short Answer Scoring: Validity, Reliability, and Fairness Perspectives. Sakarya University Journal of Computer and Information Sciences, Volume: 9 Issue: 3. https://doi.org/10.35377/saucis...1835608

📄 Tam Metin PDF Okuyucu

📄

PDF görüntüleyici hazırlanıyor...

Kaynakça (61)

  1. A. Çınar, E. Ince, M. Gezer, and O. Yılmaz, “Machine learning algorithm for grading open-ended physics questions in Turkish,” Education and Information Technologies, vol. 25, no. 5, pp. 3821–3844, 2020.
  2. N. I. Kurbanoğlu and M. Olcaytürk, “Investigation of the exam question types attitude scale for secondary school students: Development, validity, and reliability,” Sakarya University Journal of Education, vol. 13, no. 2, pp. 191–206, 2023.
  3. K. Noyes, R. L. McKay, M. Neumann, K. C. Haudek, and M. M. Cooper, “Developing computer resources to automate analysis of students’ explanations of London dispersion forces,” Journal of Chemical Education, vol. 97, no. 11, pp. 3923–3936, 2020.
  4. X. Zhu, H. Wu, and L. Zhang, “Automatic short-answer grading via BERT-based deep neural networks,” IEEE Transactions on Learning Technologies, vol. 15, no. 3, pp. 364–375, 2022.
  5. M. Ariely, T. Nazaretsky, and G. Alexandron, “Personalized automated formative feedback can support students in generating causal explanations in biology,” in Proc. 16th Int. Conf. Learning Sciences (ICLS 2022), 2022, pp. 953–956.
  6. L. Morjaria et al., “Examining the efficacy of ChatGPT in marking short-answer assessments in an undergraduate medical program,” International Medical Education, vol. 3, no. 1, pp. 32–43, 2024.
  7. M. Uto and Y. Uchida, “Automated short-answer grading using deep neural networks and item response theory,” in Artificial Intelligence in Education, AIED 2020, pp. 334–339.
  8. S. Nath, B. Parsaeifard, and E. Werlen, “Automated short answer grading using BERT on German datasets,” in EARLI 2023, 2023.
  9. S. Gombert et al., “Coding energy knowledge in constructed responses with explainable NLP models,” Journal of Computer Assisted Learning, vol. 39, no. 3, pp. 767–786, 2023.
  10. M. Abdul-Salam, M. A. El-Fatah, and N. F. Hassan, “Automatic grading for Arabic short answer questions using optimized deep learning model,” PLOS One, vol. 17, no. 8, p. e0272269, 2022.
  11. W. Westera, M. Dascalu, H. Kurvers, S. Ruseti, and S. Trausan-Matu, “Automated essay scoring in applied games: Reducing the teacher bandwidth problem in online training,” Computers & Education, vol. 123, pp. 212–224, 2018.
  12. O. Henkel, L. Hills, B. Roberts, and J. McGrane, “Can LLMs grade short-answer reading comprehension questions: An empirical study with a novel dataset,” arXiv:2310.18373, 2023.
  13. J. Burstein et al., “Computer analysis of essay content for automated score prediction: A prototype automated scoring system for GMAT analytical writing assessment essays,” ETS Research Report Series, vol. 1998, no. 1, pp. i–67, 1998.
  14. T. K. Landauer, “Latent semantic analysis: A theory of the psychology of language and mind,” Discourse Processes, vol. 27, pp. 303–310, 1999.
  15. C. Leacock and M. Chodorow, “C-rater: Automated scoring of short-answer questions,” Computers and the Humanities, vol. 37, pp. 389–405, 2003.
  16. S. Burrows, I. Gurevych, and B. Stein, “The eras and trends of automatic short answer grading,” International Journal of Artificial Intelligence in Education, vol. 25, pp. 60–117, 2015.
  17. H. A. Ghavidel, A. Zouaq, and M. C. Desmarais, “Using BERT and XLNET for the automatic short answer grading task,” in Proc. CSEDU, 2020, pp. 58–67.
  18. G. Mardini et al., “A deep-learning-based grading system (ASAG) for reading comprehension assessment by using aphorisms as open-answer questions,” Education and Information Technologies, vol. 29, no. 4, pp. 4565–4590, 2024.
  19. J. Sawatzki, T. Schlippe, and M. Benner-Wickner, “Deep learning techniques for automatic short answer grading,” in Proc. Int. Conf. Artificial Intelligence in Education Technology, Singapore, 2021, pp. 65–75.
  20. F. Sun, N. Li, K. Wang, and L. Goette, “Large language models are overconfident and amplify human bias,” arXiv:2505.02151, 2025.
  21. M. Weber and M. Reichardt, “Evaluation is all you need: Prompting generative large language models for annotation tasks in the social sciences,” arXiv:2401.00284, 2023.
  22. T. Brown et al., “Language models are few-shot learners,” in Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 1877–1901.
  23. J. Wei et al., “Chain-of-thought prompting elicits reasoning in large language models,” arXiv:2201.11903, 2023.
  24. N. Pangakis, S. Wolken, and N. Fasching, “Automated annotation with generative AI requires validation,” arXiv:2306.00176, 2023.
  25. R. Thalken, E. H. Stiglitz, D. Mimno, and M. Wilkens, “Modeling legal reasoning: LM annotation at the edge of human agreement,” arXiv:2310.18440, 2023.
  26. E. Ollion, R. Shen, A. Macanovic, and A. Chatelain, “ChatGPT for text annotation? Mind the hype!,” OSF Preprints, 2023.
  27. A. Spirling, “Why open-source generative AI models are an ethical way forward for science,” Nature, vol. 616, p. 413, 2023.
  28. M. T. Kane, “Validation,” in Educational Measurement, 4th ed., R. L. Brennan, Ed. ACE/Praeger, 2006, pp. 17–64.
  29. S. Messick, “Validity,” in Educational Measurement, 3rd ed., R. L. Linn, Ed. Macmillan, 1989, pp. 13–103.
  30. D. M. Williamson, X. Xi, and F. J. Breyer, “A framework for evaluation and use of automated scoring,” Educational Measurement: Issues and Practice, vol. 31, no. 1, pp. 2–13, 2012.
  31. S. Y. Yoon, “Short answer grading using one-shot prompting and text similarity scoring model,” arXiv:2305.18638, 2023.
  32. C. Senanayake and D. Asanka, “Rubric-based automated short answer scoring using large language models (LLMs),” pp. 1–6, Apr. 2024.
  33. J. Rodriguez et al., “Cross-context adaptability of GPT-4 for automated grading across diverse disciplinary scenarios,” in Proc. Educational Data Mining, 2024.
  34. H. Li, Y. Chu, K. Yang, Y. Copur-Gencturk, and J. Tang, “LLM-based automated grading with human-in-the-Loop,” arXiv:2504.05239, 2025.
  35. Z. Wang and C. Ormerod, “Generative language models with retrieval augmented generation for automated short answer scoring,” arXiv:2408.03811, 2024.
  36. M. Fateen, B. Wang, and T. Suenori Mine, “Beyond scores: A modular RAG-based system for automatic short answer scoring with feedback,” arXiv:2409.20042, 2024.
  37. S. Haldar, S. Sengupta, and A. K. Das, “RUBRA: An agentic AI system for automatic short answer grading using LLMs and RAG,” pp. 1–6, Jul. 2025, doi: 10.1109/ciacon65473.2025.11189329.
  38. A. Kundu and D. Barbosa, “Are large language models good essay graders?,” arXiv:2409.13120, 2024.
  39. G. Kortemeyer, “Performance of the pre-trained large language model GPT-4 on automated short answer grading,” Discover Artificial Intelligence, vol. 4, p. 47, 2024.
  40. R. Gao et al., “Towards scalable automated grading: Leveraging large language models for conceptual question evaluation in engineering,” arXiv:2411.03659, 2024.
  41. P. C. Mendonça, F. Quintal, and F. Mendonça, “Evaluating LLMs for automated scoring in formative assessments,” Applied Sciences, vol. 15, no. 5, p. 2787, Mar. 2025, doi: 10.3390/app15052787.
  42. C. Grevisse, “LLM-based automatic short answer grading in undergraduate medical education,” BMC Medical Education, vol. 24, no. 1, p. 1060, 2024.
  43. W. Li and H. Liu, “Applying large language models for automated essay scoring for non-native Japanese,” Humanities and Social Sciences Communications, vol. 11, pp. 1–15, 2024.
  44. X. Wu et al., “Unveiling scoring processes: Dissecting the differences between LLMs and human graders in automatic scoring,” Technology, Knowledge and Learning, pp. 1–16, 2025.
  45. P. Lai et al., “SAS-Bench: A fine-grained benchmark for evaluating short answer scoring with large language models,” arXiv:2505.07247, 2025.
  46. J. Cohen, “Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit,” Psychological Bulletin, vol. 70, no. 4, pp. 213–220, 1968.
  47. R. Neumann and P. Graeff, “Quantitative approaches to comparative analyses: Data properties and their implications for theory, measurement and modeling,” European Political Science, vol. 14, pp. 385–393, 2015.
  48. T. G. Dietterich, “An experimental comparison of three methods for constructing ensembles of decision trees: Bagging, boosting, and randomization,” Machine Learning, vol. 40, pp. 139–157, 2000.
  49. J. Dodge et al., “Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping,” arXiv:2002.06305, 2020.
  50. C. Yeung et al., “A zero-shot LLM framework for automatic assignment grading in higher education,” arXiv:2501.14305, 2025.
  51. A. A. Khan, O. Chaudhari, and R. Chandra, “A review of ensemble learning and data augmentation models for class imbalanced problems,” Expert Systems with Applications, vol. 244, p. 122778, 2024.
  52. B. Pecher, I. Srba, M. Bielikova, and J. Vanschoren, “Automatic combination of sample selection strategies for few-shot learning,” arXiv:2402.03038, 2024.
  53. L. Yoshida, “The impact of example selection in few-shot prompting on automated essay scoring using GPT models,” in International Conference on Artificial Intelligence in Education, 2024, pp. 61–73.
  54. G. Meyer, P. Breuer, and J. Fürst, “ASAG2024: A combined benchmark for short answer grading,” in Proc. 2024 ACM Virtual Global Computing Education Conf. V.2, 2024, pp. 322–323.
  55. A. Condor, “Automatic grading and feedback for students’ short written responses,” Doctoral dissertation, University of California, Berkeley, 2024.
  56. N. Doewes, A. Kurdhi, and A. Saxena, “Evaluating quadratic weighted kappa as the standard performance metric for automated essay scoring,” in Proc. 16th Int. Conf. Educational Data Mining (EDM 2023), pp. 103–113, 2023.
  57. A. Jonsson and G. Svingby, “The use of scoring rubrics: Reliability, validity and educational consequences,” Educational Research Review, vol. 2, pp. 130–144, 2007.
  58. C. Xavier et al., “Empowering instructors with AI: Evaluating the impact of an AI-driven feedback tool in learning analytics,” IEEE Transactions on Learning Technologies, 2025.
  59. I. Chamieh, T. Zesch, and K. Giebermann, “LLMs in short answer scoring: Limitations and promise of zero-shot and few-shot approaches,” in Proc. 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), 2024, pp. 309–315.
  60. L. H. Chang and F. Ginter, “Automatic short answer grading for Finnish with ChatGPT,” in Proc. AAAI Conf. Artificial Intelligence, vol. 38, no. 21, 2024, pp. 23173–23181.
  61. M. Mynlieff, A. L. Manogaran, M. St. Maurice, and T. J. Eddinger, “Writing assignments with a metacognitive component enhance learning in a large introductory biology course,” CBE—Life Sciences Education, vol. 13, no. 2, pp. 311–321, 2014.