SavedAdd your journal

Journals / Muş Alparslan Üniversitesi Sağlık Bilimleri Dergisi / 2026 / Cilt: 6 Sayı: 2

Evaluating Large Language Models as Educational Tools: A Multi-Model Analysis of Performance on the Turkish Medical Specialty Examination (TUS) Ophthalmology Questions Özet:

Büyük Dil Modellerinin Eğitsel Araçlar Olarak Değerlendirilmesi: Türk Tıpta Uzmanlık Sınavı (TUS) Göz Hastalıkları Soruları Üzerinde Çoklu Model Analizi

Burhan Başkan(SBÜ Van Eğitim ve Araştırma Hastanesi)0000-0002-2481-9892
Yusuf Evcimen(SBÜ Van Eğitim ve Araştırma Hastanesi)0000-0002-9934-2881
Özgür Bülent Timuçin(SBÜ Van Eğitim ve Araştırma Hastanesi)0000-0003-0257-0819
İffet Yarımağa(SBÜ Van Eğitim ve Araştırma Hastanesi)0000-0003-0131-2316
Celal Sağmış(SBÜ Van Eğitim ve Araştırma Hastanesi)0009-0001-2899-4727
Pages
51–65
Publication Date
2026-07-31
🏛️Bibliographic metadata and full text access for this article are provided via DergiPark.
DergiPark Record ↗
✨

AI Summary & Key Takeaways

AI Insight
⚡ TL;DR(Too Long; Didn't Read)

Performance was statistically compared based on overall accuracy, question type (case-based vs. knowledge-based), chronological period (old: 2015-2019 vs. new: 2020-2024), and clinical subspecialty.

📌 3 Key Takeaways

🎯
Objective & Hypothesis:This study aimed to comprehensively evaluate the performance of seven leading large language models (LLMs) from the 2024-2025 period on Turkish Medical Specialty Examination (TUS) ophthalmology questions, analyzing factors such as question type, chronology, and clinical area, while critically assessing the risk of data contamination and the technology's realistic potential in medical education.
📊
Key Findings & Data:A set of 210 TUS ophthalmology questions (2015-2024) was presented to seven LLMs (OpenAI o1-preview, Claude 4.5 Sonnet, GPT-4o, Gemini 2.5 Pro, DeepSeek-R1, Llama 3 70B, Command R+) using a standardized protocol.
💡
Conclusion & Impact:Performance was statistically compared based on overall accuracy, question type (case-based vs. knowledge-based), chronological period (old: 2015-2019 vs. new: 2020-2024), and clinical subspecialty.
🔬Design:Review & Systematic Survey
📍Scope:2024-2025 Verileri
⚙️Variables:İngilizce dışı diller · klinik akıl yürütme · OpenAI o1

Abstract

This study aimed to comprehensively evaluate the performance of seven leading large language models (LLMs) from the 2024-2025 period on Turkish Medical Specialty Examination (TUS) ophthalmology questions, analyzing factors such as question type, chronology, and clinical area, while critically assessing the risk of data contamination and the technology's realistic potential in medical education. A set of 210 TUS ophthalmology questions (2015-2024) was presented to seven LLMs (OpenAI o1-preview, Claude 4.5 Sonnet, GPT-4o, Gemini 2.5 Pro, DeepSeek-R1, Llama 3 70B, Command R+) using a standardized protocol. Performance was statistically compared based on overall accuracy, question type (case-based vs. knowledge-based), chronological period (old: 2015-2019 vs. new: 2020-2024), and clinical subspecialty. All responses underwent blinded, expert qualitative coding for error type and confidence. OpenAI o1-preview achieved the highest overall accuracy (95.2%). Claude 4.5 Sonnet and GPT-4o demonstrated 100% accuracy on case-based questions. A significant performance gap existed between closed-source (average 91.0%) and open-source models (average 66.1%) (p<0.001, OR=5.14). Chronological analysis revealed a significant performance drop on new questions for several models (e.g., Command R+, DeepSeek-R1), suggesting potential data contamination. Qualitative analysis showed error concentration (over 65%) in complex clinical areas like glaucoma and retina. Hallucination rates were substantially higher in open-source models (42.5% vs. 12.8%). A weak, non-significant correlation was found between model confidence and answer accuracy (rho=0.21, p=0.09). Advanced LLMs show near-expert-level textual performance on TUS questions, highlighting their potential as complementary educational tools. However, significant performance variations, vulnerabilities in nuanced clinical judgment, suspected data contamination, high hallucination rates in open-source models, and the confidence-accuracy disconnect necessitate a critical approach. This text-based performance must not be equated with real-world, visually-intensive clinical diagnostic competence. Implementation requires rigorous human oversight, AI literacy education, and ethical frameworks.

Özet

Bu Bu çalışma, 2024-2025 dönemine ait yedi büyük dil modelinin (BDM) Türkçe TUS Göz Hastalıkları sorularındaki performansını kapsamlı bir şekilde değerlendirmeyi, soru tipi, kronoloji ve klinik alan gibi faktörleri analiz etmeyi, veri kirliliği riskini eleştirel olarak incelemeyi ve teknolojinin tıp eğitimindeki gerçekçi potansiyelini ortaya koymayı amaçlamıştır. İki yüz on TUS Göz Hastalıkları sorusu (2015-2024) yedi BDM'ye (OpenAI o1-preview, Claude 4.5 Sonnet, GPT-4o, Gemini 2.5 Pro, DeepSeek-R1, Llama 3 70B, Command R+) standart bir protokol kullanılarak yöneltildi. Performans; genel doğruluk, soru tipi (olgu temelli vs. bilgi temelli), kronolojik dönem (eski: 2015-2019 vs. yeni: 2020-2024) ve klinik yan dal bazında istatistiksel olarak karşılaştırıldı. Tüm yanıtlar, hata tipi ve güven düzeyi için körlenmiş, uzman nitel kodlamaya tabi tutuldu. OpenAI o1-preview en yüksek genel doğruluğa (%95.2) ulaştı. Claude 4.5 Sonnet ve GPT-4o, olgu temelli sorularda %100 doğruluk gösterdi. Kapalı kaynak (ortalama %91.0) ve açık kaynak modeller (ortalama %66.1) arasında anlamlı bir performans farkı saptandı (p<0.001, OR=5.14). Kronolojik analiz, bazı modellerde (örn. Command R+, DeepSeek-R1) yeni sorularda anlamlı performans düşüşü ortaya koyarak potansiyel veri kirliliğine işaret etti. Nitel analiz, hataların %65'inin glokom ve retina gibi karmaşık klinik alanlarda yoğunlaştığını gösterdi. Halüsinasyon oranları açık kaynak modellerde anlamlı derecede yüksekti (%42.5'e karşı %12.8). Model güveni ile cevap doğruluğu arasında zayıf, anlamsız bir korelasyon bulundu (rho=0.21, p=0.09). Gelişmiş BDM'ler, TUS sorularında uzman düzeyine yakın metinsel performans sergileyerek tamamlayıcı eğitim araçları olarak potansiyellerini vurgulamaktadır. Ancak anlamlı performans farklılıkları, nüanslı klinik yargıdaki zayıflıklar, şüpheli veri kirliliği, açık kaynak modellerdeki yüksek halüsinasyon oranları ve güven-doğruluk kopukluğu eleştirel bir yaklaşımı zorunlu kılmaktadır. Bu metin temelli performans, gerçek dünyadaki görsel-yoğun klinik tanı yetkinliği ile eş tutulmamalıdır. Uygulama, titiz insan denetimi, yapay zeka okuryazarlığı eğitimi ve etik çerçeveler gerektirmektedir.

🔖

Cite This Article

Başkan, B., Evcimen, Y., Timuçin, Ö. B., Yarımağa, İ., Sağmış, C. (2026). Evaluating Large Language Models as Educational Tools: A Multi-Model Analysis of Performance on the Turkish Medical Specialty Examination (TUS) Ophthalmology Questions Özet:. Muş Alparslan Üniversitesi Sağlık Bilimleri Dergisi, Cilt: 6 Sayı: 2, 51–65. https://doi.org/10.71255/maunsbd.1913060

📄 Full Text PDF Viewer

📄

PDF görüntüleyici hazırlanıyor...

References (30)

  1. Antaki, F., Touma, S., Milad, D., El-Khoury, J., & Duval, R. (2023). Evaluating the Performance of ChatGPT in Ophthalmology: An Analysis of Its Successes and Shortcomings. Ophthalmology Science, *4*(4), 100424. doi.org/10.1016/j.xops.2023.100324
  2. Aygul, Y., Olucoglu, M., & Alpkocak, A. (2024). Are Large Language Models More Successful than Humans in the Turkish Medical Specialty Exam (TUS)? arXiv preprint arXiv:2408.12305. doi: 10.48550/arXiv.2408.12305
  3. Balci, A.S., Yazar, Z., Ozturk, B.T. et al. (2024). Performance of Chatgpt in ophthalmology exam; human versus AI. International Ophthalmology, *44*, 413. doi.org/10.1007/s10792-024-03353-w
  4. Bozkurt, B., Coskun, K., & Bakal, G. (2024). Building a challenging medical dataset for comparative evaluation of classifier capabilities. Computers in Biology and Medicine, *178*, 108721. doi: 10.1016/j.compbiomed.2024.108721
  5. Carbonari, V., Veltri, P., & Guzzi, P. H. (2025). Decoding Rarity: Large Language Models in the Diagnosis of Rare Diseases. arXiv preprint arXiv:2505.17065. doi: 10.48550/arXiv.2505.17065
  6. Gilson, A., Safranek, C. W., Huang, T., et al. (2023). How Does ChatGPT Perform on the United States Medical Licensing Examination? The Implications of Large Language Models for Medical Education and Knowledge Assessment. JMIR Medical Education, *9*, e45312. doi: 10.2196/45312
  7. Gokkurt Yilmaz, B. N., et al. (2025). Evaluation of the performance of different large language models on head and neck anatomy questions in the dentistry specialization exam in Turkey. Surgical and Radiologic Anatomy, *47*(1), 211. https://doi.org/10.1007/s00276-025-03723-8
  8. Greig, E. C., Duker, J. S., & Waheed, N. K. (2020). A practical guide to optical coherence tomography angiography interpretation. International Journal of Retina and Vitreous, *6*(1), 55. https://doi.org/10.1186/s40942-020-00262-9
  9. Ji, Z., Lee, N., Frieske, R., et al. (2023). Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, *55*(12), 1-38. doi: 10.1145/3571730
  10. Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G., & King, D. (2019). Key Challenges for Delivering Clinical Impact with Artificial Intelligence. BMC Medicine, *17*(1), 195. doi: 10.1186/s12916-019-1426-2
  11. Kim, Y., Jeong, H., Chen, S., Li, S. S., Park, C., Lu, M., & Breazeal, C. (2025). Medical hallucinations in foundation models and their impact on healthcare. arXiv preprint arXiv:2503.05777. doi: 10.48550/arXiv.2503.05777
  12. Kung, T. H., Cheatham, M., Medenilla, A., et al. (2023). Performance of ChatGPT on USMLE: Potential for AI-Assisted Medical Education Using Large Language Models. PLOS Digital Health, *2*(2), e0000198. doi: 10.1371/journal.pdig.0000198
  13. Mihalache, A., Popovic, M. M., & Muni, R. H. (2023). Performance of an Artificial Intelligence Chatbot in Ophthalmic Knowledge Assessment. JAMA Ophthalmology, *141*(6), 589-597. doi: 10.1001/jamaophthalmol.2023.1144
  14. Pham, A. T., Pan, A. A., & Yohannan, J. (2024). Big data in visual field testing for glaucoma. Taiwan Journal of Ophthalmology, *14*(3), 289-298. doi.10.4103/tjo.TJO-D-24-00059
  15. Shvartz, E., Zur, O., Attal, L., Nujeidat, Z., Plopsky, G., & Bahir, D. (2025). GPT-4o vs. residency exams in ophthalmology---a performance analysis. Journal of Medical Artificial Intelligence, *8*. doi: 10.21037/jmai-24-42
  16. Singhal, K., Azizi, S., Tu, T., et al. (2022). Large Language Models Encode Clinical Knowledge. arXiv preprint arXiv:2212.13138. doi.org/10.48550/arXiv.2212.13138
  17. Terzi, R., Kılıç, A. E., Karaahmetoğlu, G., & Özdemir, O. B. (2025). The digital eye for mammography: deep transfer learning and model ensemble based open-source toolkit for mass detection and classification. Signal, Image and Video Processing, *19*(2), 170. doi.org/10.1007/s11760-024-03737-6
  18. Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023). Large Language Models in Medicine. Nature Medicine, *29*(8), 1930-1940. doi: 10.1038/s41591-023-02448-8
  19. Topol, E. J. (2019). High-performance medicine: the convergence of human and artificial intelligence. Nature Medicine, *25*(1), 44-56. doi: 10.1038/s41591-018-0300-7
  20. Türkmen, H., Dikenelli, O., Eraslan, C., Callı, M. C., & Özbek, S. S. (2023). Bioberturk: Exploring turkish biomedical language model development strategies in low-resource setting. Journal of Healthcare Informatics Research, *7*(4), 433-446. doi.org/10.1007/s41666-023-00140-7
  21. Wang, Y., Wu, S., Zhang, Y., Yan, S., Liu, Z., Luo, J., & Fei, H. (2025). Multimodal chain-of-thought reasoning: A comprehensive survey. arXiv preprint arXiv:2503.12605. doi.org/10.48550/arXiv.2503.12605
  22. Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., et al. (2024). Harnessing the power of llms in practice: a survey on chatgpt and beyond. ACM Transactions on Knowledge Discovery from Data, *18*(6), 1-32. doi.org/10.1145/3649506
  23. Zeng, Z., Liu, J., Chen, S., He, T., Liao, Y., Tian, Y., et al. (2025). Futurex: an advanced live benchmark for llm agents in future prediction. arXiv preprint arXiv:2508.11987. doi: 10.48550/arXiv.2508.11987
  24. Zhang, Q., Wang, S., Wang, X., Xu, C., Liang, J., & Liu, Z. (2025). Advancing ophthalmology with large language models: applications, challenges, and future directions. Survey of Ophthalmology. https://doi.org/10.1016/j.survophthal.2025.02.009
  25. Zhang, X., Li, S., Hauer, B., Shi, N., & Kondrak, G. (2023). Don't trust ChatGPT when your question is not in English: a study of multilingual abilities and types of LLMs. arXiv preprint arXiv:2305.16339. doi: 10.48550/arXiv.2305.16339
  26. Zhang, Z., Zhang, H., Pan, Z., Bi, Z., Wan, Y., Song, X., & Fan, X. (2025). Evaluating large language models in ophthalmology: systematic review. Journal of Medical Internet Research, *27*, e76947. doi: 10.2196/76947
  27. Zheng, T., Chen, Y., Li, C., Li, C., Zong, Q., Shi, H.,. & See, S. (2025). The curse of cot: On the limitations of chain-of-thought in in-context learning. arXiv preprint arXiv:2504.05081. doi.org/10.48550/arXiv.2504.05081
  28. Zhong, Q., Ding, L., Liu, J., Du, B., & Tao, D. (2023). Can chatgpt understand too? a comparative study on chatgpt and fine-tuned bert. arXiv preprint arXiv:2302.10198. doi: 10.48550/arXiv.2302.10198
  29. Zhu, Y., Du, S., Li, B., Luo, Y., & Tang, N. (2024). Are large language models good statisticians? Advances in Neural Information Processing Systems, *37*, 62697-62731. doi.10.52202/079017-2005
  30. Zheng, T., Chen, Y., Li, C., Li, C., Zong, Q., Shi, H.,. & See, S. (2025). The curse of cot: On the limitations of chain-of-thought in in-context learning. arXiv preprint arXiv:2504.05081. doi.org/10.48550/arXiv.2504.05081