SavedAdd your journal

Journals / Anadolu Journal of Educational Sciences International / 2026 / Cilt: 15 Sayı: 1

ChatGPT: Is It Reliable as an Automated Writing Evaluation Tool?

ChatGPT: Otomatik Bir Yazma Değerlendirme Aracı olarak Güvenilir mi?

Saliha Toscu(ÇANKAYA ÜNİVERSİTESİ)0000-0002-8179-5444
Pages
329–349
Publication Date
2025-03-22
🏛️Bibliographic metadata and full text access for this article are provided via DergiPark.
DergiPark Record ↗
✨

AI Summary & Key Takeaways

AI Insight
⚡ TL;DR(Too Long; Didn't Read)

The test results indicated no statistically significant differences in the scores the bot or the teacher gave.

📌 3 Key Takeaways

🎯
Objective & Hypothesis:This study primarily aims to give an understanding of whether or not teachers could rely on AI technology, specifically ChatGPT, to score students’ writings.
📊
Key Findings & Data:The students were assigned different writing tasks for five weeks, and the tasks were scored by a teacher and ChatGPT separately.
💡
Conclusion & Impact:The test results indicated no statistically significant differences in the scores the bot or the teacher gave.
🔬Design:Araştırma Makalesi
📍Scope:Comprehensive Sample
⚙️Variables:yapay zeka · ChatGPT · değerlendirme

Abstract

This study primarily aims to give an understanding of whether or not teachers could rely on AI technology, specifically ChatGPT, to score students’ writings. The study was conducted with the participation of EFL university students. The students were assigned different writing tasks for five weeks, and the tasks were scored by a teacher and ChatGPT separately. Then, their scores were compared to see the extent to which ChatGPT and teacher scores differed on the SPSS. The test results indicated no statistically significant differences in the scores the bot or the teacher gave. Additionally, the results were supported by the qualitative analysis of the teachers’ perception of ChatGPT use for automated writing evaluation. The teachers’ perceptions indicated their positive attitudes towards its use for the evaluation process and general use for enhancing instruction and learning, together with the concerns and suggestions to make the most of ChatGPT. The study gives insights into the integration of ChatGPT into the assessment process and its effectiveness for class practices.

Özet

Bu çalışma öncelikle öğretmenlerin öğrencilerin yazılarını puanlamak için yapay zeka teknolojisine, özellikle chatgpt'ye güvenip güvenemeyeceğine dair bir anlayış kazandırmayı amaçlamaktadır. Yarı deneysel bir araştırma tasarımına dayanan çalışma, EFL üniversitesi öğrencilerinin katılımıyla gerçekleştirilmiştir. Öğrencilere beş hafta boyunca farklı yazma ödevleri verildi ve ödevler bir öğretmen ve ChatGPT tarafından ayrı ayrı puanlandı. Daha sonra, ChatGPT ve bir öğretmenin puanlarının SPSS’de ne ölçüde farklı olduğunu görmek için puanları birbirleriyle karşılaştırıldı. Testlerden elde edilen sonuçlar, ChatGPT veya öğretmen tarafından verilen puanlarda istatistiksel olarak anlamlı bir farklılık göstermedi. Ek olarak, sonuçlar öğretmenlerin otomatik yazma değerlendirmesi için ChatGPT kullanımı algısının nitel analizi ile desteklenmiştir. Öğretmenlerin algıları, değerlendirme sürecinde kullanımına ve öğretimi ve öğrenmeyi geliştirmek için genel kullanıma yönelik olumlu tutumlarını birlikte göstermiştir.

🔖

Cite This Article

Toscu, S. (2025). ChatGPT: Is It Reliable as an Automated Writing Evaluation Tool?. Anadolu Journal of Educational Sciences International, Cilt: 15 Sayı: 1, 329–349. https://doi.org/10.18039/ajesi.1463503

📄 Full Text PDF Viewer

📄

PDF görüntüleyici hazırlanıyor...

References (38)

  1. Adamopoulou, E., & Moussiades, L. (2020). Chatbots: History, technology, and applications. Machine Learning with Applications, 2, 1-18. https://doi.org/10.1016/j.mlwa.2020.100006
  2. Alberth. (2023). The use of ChatGPT in academic writing: A blessing or a curse in English. TEFLIN Journal, 34(2), 337-352. http://dx.doi.org/10.15639/teflinjournal.v34i2/337-352
  3. Algaraady, J., & Mahyoob, M. (2023). ChatGPTs capabilities in spotting and analyzing writing errors experienced by EFL learners. Arab World English Journal, 9, 3-17. https://dx.doi.org/10.24093/awej/call9.1
  4. Avsheniuk, N., Lutsenko, O., Svyrydiuk, T., & Seminikhyna, N. (2024). Empowering language learners’ critical thinking: Evaluating ChatGPT ‘s role in English course implementation. Arab World English Journal (AWEJ), 210-224. https://dx.doi.org/10.24093/awej/ChatGPT.14
  5. Ayan, A. D., & Erdemir, N. (2023). EFL teachers’ perceptions of automated written corrective feedback and Grammarly. Ahmet Keleşoğlu Eğitim Fakültesi Dergisi (AKEF), 5(3), 1183-198. https://doi.org/10.38151/akef.2023.106
  6. Bin-Hady, W. R. A., Al-Kadi, A., Hazaea, A., Ali, J. K. M. (2023). Exploring the dimensions of ChatGPT in English language learning: A global perspective. Library Hi Tech. https://dx.doi.org/10.1108/LHT-05-2023-0200
  7. Chiu, M., Hwang, G., Hsia, L., & Shyu, F. (2022). Artificial intelligence-supported art education: A deep learning-based system for promoting university students’ artwork appreciation and painting outcomes. Interactive Learning Environments. https://doi.org/10.1080/10494820.2022.2100426
  8. Demir, S. (2023). Investigation of ChatGPT and real raters in scoring open-ended items in terms of inter-rater reliability. International Journal of Turkish Educational Studies, 11(21), 1072-1099. https://doi.org/10.46778/goputeb.1345752
  9. Dilekçi, A., & Karatay, H. (2023). The effects of the 21st century skills curriculum on the development of students’ creative thinking skills. Thinking Skills and Creativity, 47, 101229. https://doi.org/10.1016/j.tsc.2022.101229
  10. Dikli, S. (2006). An overview of automated scoring of essays. The Journal of Technology, Learning, and Assessment, 5(1),1–35.
  11. Escalante, J., Pack, A., & Barrett, A. (2023). AI-generated feedback on writing: Insights into efficacy and ENL student preference. International Journal of Educational Technology in Higher Education, 1-20. https://doi.org/10.1186/s41239-023-00425-2
  12. Evmenova, A. S., Borup, J., & Shin, J. K. (2024). Harnessing the power of generative AI to support ALL learners. TechTrends, 68, 820-831. https://doi.org/10.1007/s11528-024-00966-x
  13. Guo, K., & Wang, D. (2024). To resist it or to embrace it? Examining ChatGPT’s potent to support teacher feedback in EFL writing. Education and Information Technologies, 29, 8435-8463. https://doi.org/ 10.1007/s10639-023-12146-0
  14. Halaweh, M. (2023). ChatGPT in education: Strategies for responsible implementation. Contemporary Educational Technology, 15(2), 1-11. https://doi.org/10.30935/cedtech/13036
  15. Harunasari, S. Y. (2022). Examining the effectiveness of AI-integrated approach in EFL writing: A case of ChatGPT. International Journal of Progressive Sciences and Technologies (IJPSAT), 39(2), 357-368.
  16. Ho, P. X. P. (2024). Using ChatGPT in English language learning: A study on I.T. students’ attitudes, habits, and perceptions. International Journal of TESOL & Education, 4(1), 55-68. https://doi.org/10.54855/ijte.24414
  17. Hussein, M. A., Hassan, H., & Nassef, M. (2019). Automated language essay scoring systems: A literature review. Peer Computer Science, 5, 1-16. https://doi.org/10.7717/peerj-cs.208
  18. Kucuk, T. (2024). ChatGPT integrated grammar teaching and learning in EFL classes: A study on Tishk international university students in Erbil, Iraq. Arab World English Journal, 100-111. https://dx.doi.org/10.24093/awej/ChatGPT.6
  19. Latif, E., & Zhai, X. (2024). Fine-tuning ChatGPT for automatic scoring. Computers and Education, 6, 1-10. https://doi.org/10.1016/j.caeai.2024.100210
  20. Li, J., Link, S., & Hegelheimer, V. (2015). Rethinking the role of automated writing evaluation (AWE) feedback in ESL writing instruction. Journal of Second Language Writing, 27, 1-18. https://doi.org/10.1016/j.jslw.2014.10.004
  21. Liu, X., Zhao, A., Sun, X., Zhang, K., Kou, F., & Gai, J. (2023). The rise of ChatGPT: Unlocking its potential in education. In D. Kumar, P. Loskot, & Q. Chen (Eds.), The 3rd International Conference on Internet, Education and Information Technology (IETS 2023) (pp.1230-1236). Springer. https://doi.org/10.2991/978-94-6463-230-9_148
  22. Mizumoto, A., & Eguchi, M. (2023). Exploring the potential of using an AI language model for automated essay scoring. Research Methods in Applied Linguistics, 2(2), 1-13. https://doi.org/10.1016/j.rmal.2023.100050
  23. Mohamed, A. M. (2024). Exploring the potential of an AI-ChatBot (ChatGPT) in enhancing English as a foreign language (EFL) teaching: Perceptions of EFL faculty members. Education and Information Technologies, 29(3), 3195-3217. https://doi.org/10.1007/s10639-023-11917-z
  24. Moussalli, S., & Cardoso, W. (2020). Intelligent personal assistants: Can they understand and be understood by accented L2 learners? Computer Assisted Language Learning, 33(8), 865-890. https://doi.org/10.1080/09588221.2019.1595664
  25. Qu, K., & Wu, X. (2024). ChatGPT as a CALL tool in language education: A study of hedonic motivation adoption models in English learning Environments. Education and Information Technologies. https://doi.org/10.1007/s10639-024-12598-y
  26. Ranalli, J. (2018). Automated written corrective feedback: How well can students make use of it? Computer Assisted Language Learning, 31(7), 653-674. https://doi.org/10.1080/09588221.2018.1428994
  27. Solak, E. (2024). Revolutionizing language learning: How ChatGPT and AI are changing the way we learn languages. International Journal of Technology in Education (IJTE), 7(2), 353-372. https://doi.org/10.46328/ijte.732
  28. Stevenson, M., & Phakiti, A. (2014). The effects of computer-generated feedback on the quality of writing. Assessing Writing, 19, 51–65. https://doi.org/10.1016/j.asw.2013.11.007
  29. Susanto, D. A., Priyolistiyanto, A., Pinandhita, F., Prabowo, A. B., & Bimo, D. S. (2024). Utilizing ChatGPT on designing English language teaching (ELT) materials in Indonesia: Opportunities and challenges. Journal of Culture, English Language, Teaching & Literature, 24(1), 157-181. https://doi.org/10.24167/celt.v24i1
  30. Tai, T., & Chen, H. H. (2023). The impact of Google Assistant on adolescent EFL learners’ willingness to communicate. Interactive Learning Environments, 31(3), 1485-1502. https://doi.org/10.1080/10494820.2020.1841801
  31. Tsai, C., Lİ, Y., & Brown, I. K. (2024). Impacts of ChatGPT-assisted writing for EFL English majors: Feasibility and challenges. Education and Information Technologies. https://doi.org/10.1007/s10639-024-12722-y
  32. Wang, Y., Shang, H., & Briody, P. (2012). Exploring the impact of using automated writing evaluation in English as a foreign language university students’ writing. Computer Assisted Language Learning, 26(3), 1–24. https://doi.org/10.1080/09588221.2012.655300
  33. Wang, X., Pang, H., Wallace, M. P., Wang, Q., & Chen, W. (2022). Learners’ perceived AI presences in AI-supported language learning: A study of AI as a humanized agent from community of inquiry. Computer Assisted Language Learning. https://doi.org/10.1080/09588221.2022.2056203
  34. Wang, E. L., Matsumura, L. C., Litman, D., Correnti, R., Zhang, H., Rahimi, Z., Kisa, Z., Magooda, A., Howe, E., Quintana, R. (2022). Contributions to research on automated writing scoring and feedback systems. RAND Corporation. https://doi.org/10.7249/RBA1062-1
  35. Warschauer, M., & Ware, P. (2006). Automated writing evaluation: Defining the classroom research agenda. Language Teaching Research, 10(2), 157–180. https://doi.org/10.1191/1362168806lr190oa
  36. Wilson, J., Huang, Y., Palermo, C., Beard, G., & MacArthur, C. A. (2021). Automated feedback and automated scoring in the elementary grades: Usage, Attitudes, and associations with writing outcomes in a districtwide implementation of MI write. International Journal of Artificial Intelligence in Education, 31, 234-276. https://doi.org/10.1007/s40593-020-00236-w
  37. Zhang, B., & Mao, J. (2023). On the teaching and learning in the information age for “big data + Internet?” – Some thoughts on the application of ChatGPT in teaching. In C. F. Peng, A. Asmawi, & C. Zhao (Eds.), Proceedings of the 2023 2nd International Conference on Educational Innovation and Multimedia Technology (EIMT 2023) (pp.1005-1016). Atlantis Press. https://doi.org/10.2991/978-94-6463-192-0_131
  38. Zheng, L., Niu, J., Zhong, L., & Gyasi, J. F. (2021): The effectiveness of artificial intelligence on learning achievement and learning perception: A meta-analysis. Interactive Learning Environments. https://doi.org/10.1080/10494820.2021.2015693