SavedAdd your journal

Journals / EMO Bilimsel Dergi / 2026 / Cilt: 16 Sayı: 3

LLM-as-a-Judge for Multilingual Question Answering Evaluation: An Experimental Study on an English–Turkish Bilingual SQuAD Dataset

Çok Dilli Soru-Cevap Değerlendirmesinde Hakem Olarak LLM: İngilizce–Türkçe İki Dilli SQuAD Veri Kümesi Üzerine Deneysel Bir İnceleme

İlkay Yelmen(ISTINYE UNIVERSITY)
Pages
81–92
Publication Date
2026-09-30
DOI
—
🏛️Bibliographic metadata and full text access for this article are provided via DergiPark.
DergiPark Record ↗
✨

AI Summary & Key Takeaways

AI Insight
⚡ TL;DR(Too Long; Didn't Read)

The findings revealed that all models exhibited a high level of rank correlation with the hybrid reference scores.

📌 3 Key Takeaways

🎯
Objective & Hypothesis:This study conducts a comparative analysis of the feasibility of using large language models as automatic evaluators in English and Turkish question-answering assessments.
📊
Key Findings & Data:In this context, the study analyzes the alignment of five different large language models with a hybrid reference scoring system consisting of partial human annotation and rule-based proxy scores, as well as their classification performance, cross-lingual consistency, and prompt sensitivity.
💡
Conclusion & Impact:The findings revealed that all models exhibited a high level of rank correlation with the hybrid reference scores.
🔬Design:Modeling & Simulation
📍Scope:Comprehensive Sample
⚙️Variables:Büyük dil modeli · değerlendirici olarak büyük dil modelleri · soru-cevap değerlendirmesi

Abstract

This study conducts a comparative analysis of the feasibility of using large language models as automatic evaluators in English and Turkish question-answering assessments. In this context, the study analyzes the alignment of five different large language models with a hybrid reference scoring system consisting of partial human annotation and rule-based proxy scores, as well as their classification performance, cross-lingual consistency, and prompt sensitivity. For 100 examples selected from the parallel English–Turkish SQuAD dataset, three different candidate responses, namely correct, partially correct, and incorrect responses, were generated, and these responses were evaluated using two prompt strategies: a simple prompt and a rubric-based prompt. The model outputs were analyzed using Spearman’s rank correlation, accuracy, macro-F1, Cohen’s kappa coefficient, and the Wilcoxon signed-rank test. The findings revealed that all models exhibited a high level of rank correlation with the hybrid reference scores. While GPT-4o-mini produced the highest correlation in English, Claude-Sonnet-4.6 and gpt-5.4-mini shared the highest correlation in Turkish; in the combined English–Turkish results, Claude-Sonnet-4.6 achieved the highest correlation. Claude-Sonnet-4.6 emerged as the most balanced model in terms of classification performance and prompt stability. The results indicate that the LLM-as-a-Judge approach shows promise for evaluating Turkish question-answering systems, but that model selection, language effects, prompt design, and score calibration must be addressed collectively.

Özet

Bu çalışma, büyük dil modellerinin İngilizce ve Türkçe soru-cevap değerlendirmesinde otomatik değerlendirici olarak kullanılabilirliğini karşılaştırmalı olarak incelemektedir. Bu kapsamda, beş farklı büyük dil modelinin kısmi insan anotasyonu ve kural tabanlı vekil skorlardan oluşan karma referans puanlama yapısıyla uyumu, sınıflandırma başarımı, diller arası tutarlılığı ve prompt duyarlılığı analiz edilmiştir. Paralel İngilizce–Türkçe SQuAD veri kümesinden seçilen 100 örnek için doğru, kısmen doğru ve yanlış olmak üzere üç farklı aday yanıt oluşturulmuş, bu yanıtlar basit ve rubrik tabanlı iki prompt stratejisiyle değerlendirilmiştir. Model çıktıları Spearman sıra korelasyonu, doğruluk, makro-F1, Cohen’in kappa katsayısı ve Wilcoxon işaretli sıralar testi kullanılarak incelenmiştir. Bulgular, tüm modellerin karma referans puanlarıyla yüksek düzeyde sıra korelasyonu gösterdiğini ortaya koymuştur. GPT-4o-mini İngilizcede en yüksek korelasyonu üretirken, Türkçe değerlendirmede Claude-Sonnet-4.6 ve gpt-5.4-mini en yüksek korelasyonu paylaşmış; birleşik İngilizce–Türkçe sonuçlarda ise Claude-Sonnet-4.6 en yüksek korelasyonu göstermiştir. Claude-Sonnet-4.6, sınıflandırma başarımı ve prompt kararlılığı açısından en dengeli model olarak öne çıkmıştır. Sonuçlar, LLM-as-a-Judge yaklaşımının Türkçe soru-cevap değerlendirmesi için umut verici olduğunu ancak model seçimi, dil etkisi, prompt tasarımı ve puan kalibrasyonunun birlikte ele alınması gerektiğini göstermektedir.

🔖

Cite This Article

Yelmen, İ. (2026). LLM-as-a-Judge for Multilingual Question Answering Evaluation: An Experimental Study on an English–Turkish Bilingual SQuAD Dataset. EMO Bilimsel Dergi, Cilt: 16 Sayı: 3, 81–92. https://ajindex.com/articles/1390246

📄 Full Text PDF Viewer

📄

PDF görüntüleyici hazırlanıyor...

References (13)

  1. D. Li, B. Jiang, L. Huang, A. Beigi, C. Zhao, Z. Tan, A. Bhattacharjee, Y. Jiang, C. Chen, T. Wu, K. Shu, L. Cheng, and H. Liu, “From generation to judgment: Opportunities and challenges of LLM-as-a-Judge,” in Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025, pp. 2757–2791. doi: 10.18653/v1/2025.emnlp-main.138.
  2. A. Türkmen and K. E. Keleş, “Whom to trust? Analyzing the divergence between user satisfaction and LLM-as-a-Judge in e-commerce RAG systems,” in Proceedings of the First Workshop on Multilingual Multicultural Evaluation, 2026. pp. 189–195. doi: 10.18653/v1/2026.mme-main.12.
  3. A. Szymanski, N. Ziems, H. A. Eicher-Miller, T. J.-J. Li, M. Jiang, and R. A. Metoyer, “Limitations of the LLM-as-a-Judge approach for evaluating LLM outputs in expert knowledge tasks,” in Proceedings of the 30th International Conference on Intelligent User Interfaces (IUI ’25), 2025. doi: 10.1145/3708359.3712091.
  4. C. Toraman et al., “TurkBench: A benchmark for evaluating Turkish large language models,” in Proceedings of the Second Workshop on Natural Language Processing for Turkic Languages, 2026, pp. 126–154. doi: 10.18653/v1/2026.sigturk-1.12.
  5. J. Yu, S. Sun, X. Hu, J. Yan, K. Yu, and X. Li, “Improve LLM-as-a-Judge ability as a general ability,” in Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025, pp. 14099–14115. doi: 10.18653/v1/2025.emnlp-main.712.
  6. H. Huang, X. Bu, H. Zhou, Y. Qu, J. Liu, M. Yang, B. Xu, and T. Zhao, “An empirical study of LLM-as-a-Judge for LLM evaluation: Fine-tuned judge model is not a general substitute for GPT-4,” in Findings of the Association for Computational Linguistics: ACL 2025, 2025, pp. 5880–5895. doi: 10.18653/v1/2025.findings-acl.306.
  7. C. Yamasaki, K. Sugahara, Y. Nagi, and K. Okamoto, “Function-based labels for complementary recommendation: Definition, annotation, and LLM-as-a-Judge,” Pattern Recognition Letters, vol. 200, pp. 8–15, 2026. doi: 10.1016/j.patrec.2025.11.042.
  8. T. Khan, M. Rafiq, and A. Michalas, “AI on Trial: LLM-as-a-Judge for private and reliable clinical decision-making,” in 2025 Annual Computer Security Applications Conference Workshops (ACSAC Workshops), 2025, pp. 256–262. doi: 10.1109/ACSACW69556.2025.00033.
  9. B. Bellard, X. C. Liu, R. Lohani, S. Raza, P. E. Velmovitksy, Q. Pham, and S. Saha, “Development and evaluation of an agentic RAG with embedded LLM-as-a-Judge for heart failure management: A pilot study,” in 2025 IEEE International Conference on Big Data (BigData), 2025, pp. 7770–7774. doi: 10.1109/BigData66926.2025.11402164.
  10. R. Mahadik, A. Mandal, S. Tripathi, and S. Jaiswal, “LLM evaluations for emotional expressiveness and factual consistency in medical dialogue systems using LLM-as-a-Judge,” in 2025 International Conference on Responsible, Generative and Explainable AI (ResGenXAI), 2025. doi: 10.1109/ResgenXAI64788.2025.11344029.
  11. M. I. H. Abir, N. Ferdous, A. Tasnim, and N. Mustaqeem, “Assessing large language models on zero-shot poetry summarization using ROUGE, BERT, and LLM-as-a-Judge metrics,” in 2026 5th International Conference on Electrical, Computer & Telecommunication Engineering (ICECTE), 2026. doi: 10.1109/ICECTE69292.2026.11429378.
  12. G. Crupi, R. Tufano, A. Velasco, A. Mastropaolo, D. Poshyvanyk, and G. Bavota, “On the effectiveness of LLM-as-a-Judge for code generation and summarization,” IEEE Transactions on Software Engineering, vol. 51, no. 8, pp. 2329–2345, 2025. doi: 10.1109/TSE.2025.3586082.
  13. D. Chen, R. Chen, S. Zhang, Y. Wang, Y. Liu, H. Zhou, Q. Zhang, Y. Wan, P. Zhou, and L. Sun, “MLLM-as-a-Judge: Assessing multimodal LLM-as-a-Judge with vision-language benchmark,” in Proceedings of the 41st International Conference on Machine Learning, vol. 235, 2024, pp. 6562–6595. doi: 10.5555/3692070.3692324.