SavedAdd your journal

Journals / Uludağ Üniversitesi Tıp Fakültesi Dergisi / 2025 / Cilt: 52

From Hallucination to Precision: A Longitudinal Analysis of Reference Accuracy and Plagiarism in AI-Generated Medical Literature (2024–2026)

Halüsinasyondan Hassasiyete: Yapay Zeka Tarafından Oluşturulan Tıbbi Literatürde Referans Doğruluğu ve İntihal Üzerine Boylamsal Bir Değerlendirme (2024–2026)

Mevlüt Okan Aydin(Department of Medical Education, Faculty of Medicine, Bursa Uludağ University, Bursa, Türkiye.)0000-0002-8060-8803
Alper Vatansever(Department of Anatomy, Faculty of Medicine, Bursa Uludağ University, Bursa, Türkiye.)0000-0002-3632-1020
Sezer Erer Kafa(Department of Medical Ethics and History of Medicine, Faculty of Medicine, Bursa Uludağ University, Bursa, Türkiye.)0000-0001-8926-996X
Pages
—
Publication Date
2026-03-17
🏛️Bibliographic metadata and full text access for this article are provided via DergiPark.
DergiPark Record ↗
✨

AI Summary & Key Takeaways

AI Insight
⚡ TL;DR(Too Long; Didn't Read)

Findings revealed that in November 2024, general-purpose Large Language Models exhibited high hallucination rates, with ChatGPT and Claude providing zero authentic references for certain prompts.

📌 3 Key Takeaways

🎯
Objective & Hypothesis:This study aims to evaluate the longitudinal development of general-purpose and specialized artificial intelligence tools in terms of reliability in academic writing and citation accuracy.
📊
Key Findings & Data:Eight platforms (ChatGPT, Gemini, QuillBot, Claude, Microsoft Copilot, Elicit, Consensus, and SciSpace) were analyzed using five standardized medical prompts in November 2024 and January 2026.
💡
Conclusion & Impact:Findings revealed that in November 2024, general-purpose Large Language Models exhibited high hallucination rates, with ChatGPT and Claude providing zero authentic references for certain prompts.
🔬Design:Modeling & Simulation
📍Scope:2024 Verileri
⚙️Variables:Yapay zeka · akademik yazım · yazarlık

Abstract

This study aims to evaluate the longitudinal development of general-purpose and specialized artificial intelligence tools in terms of reliability in academic writing and citation accuracy. Eight platforms (ChatGPT, Gemini, QuillBot, Claude, Microsoft Copilot, Elicit, Consensus, and SciSpace) were analyzed using five standardized medical prompts in November 2024 and January 2026. The generated introductions were assessed for reference authenticity using PubMed, Google Scholar, and Web of Science, and for plagiarism using iThenticate. Findings revealed that in November 2024, general-purpose Large Language Models exhibited high hallucination rates, with ChatGPT and Claude providing zero authentic references for certain prompts. Conversely, specialized academic tools like Elicit and SciSpace maintained near-perfect accuracy from the outset. By January 2026, a dramatic improvement was observed, with general-purpose tools like ChatGPT achieving 100% reference accuracy across all categories. Although plagiarism rates were typically below 15%, Gemini recorded a peak of 45% in 2024 before stabilizing. Specialized tools demonstrated superior capacity to manage larger citation volumes, such as SciSpace, which provided 31 verified references in a single output in 2026. While both general and specialized tools have matured significantly, researchers should still exercise caution and use verification protocols. The results indicate that artificial intelligence tools have rapidly transitioned from being prone to academic hallucinations to becoming highly reliable instruments for scholarly literature synthesis.

Özet

Bu çalışma, genel amaçlı ve uzmanlaşmış yapay zeka araçlarının akademik yazım ve atıf doğruluğu konusundaki güvenilirliğinin zamana bağlı gelişimini değerlendirmeyi amaçlamaktadır. Kasım 2024 ve Ocak 2026 tarihlerinde sekiz farklı platform (ChatGPT, Gemini, QuillBot, Claude, Microsoft Copilot, Elicit, Consensus ve SciSpace), tıp odaklı beş standart istem kullanılarak analiz edilmiştir. Üretilen giriş bölümleri, PubMed, Google Scholar ve Web of Science üzerinden referans doğruluğu ve iThenticate aracılığıyla intihal oranları açısından incelenmiştir. Bulgular, Kasım 2024'te ChatGPT ve Claude gibi genel amaçlı modellerin bazı istemlerde sıfır doğru referans vererek yüksek halüsinasyon oranları sergilediğini, Elicit ve SciSpace gibi akademik odaklı araçların ise başlangıçtan itibaren tam doğruluk sağladığını göstermiştir. Ocak 2026 itibarıyla, ChatGPT'nin tüm istemlerde tam doğruluğa ulaşmasıyla genel amaçlı araçlarda büyük bir iyileşme gözlemlenmiştir. İntihal oranları genellikle %15'in altında kalsa da, Gemini 2024 yılında bir istemde %45 ile en yüksek seviyeye ulaşmıştır. SciSpace gibi akademik araçlar, 2026'da tek bir metinde 31 doğrulanmış referans sunarak yüksek hacimli atıf yönetiminde üstünlük göstermiştir. Hem genel hem de uzmanlaşmış yapay zeka araçları önemli ölçüde olgunlaşmış olsa da, araştırmacıların doğrulama protokollerini kullanmaya devam etmeleri önerilmektedir. Sonuç olarak, yapay zeka araçları akademik halüsinasyonlara eğilimli bir yapıdan, bilimsel literatür sentezi için oldukça güvenilir enstrümanlara hızla dönüşmüştür.

🔖

Cite This Article

Aydin, M. O., Vatansever, A., Kafa, S. E., *, M. O. A. (2026). From Hallucination to Precision: A Longitudinal Analysis of Reference Accuracy and Plagiarism in AI-Generated Medical Literature (2024–2026). Uludağ Üniversitesi Tıp Fakültesi Dergisi, Cilt: 52. https://doi.org/10.32708/uutfd.1870116

📄 Full Text PDF Viewer

📄

PDF görüntüleyici hazırlanıyor...

References (31)

  1. Busse CE, Anderson EW, Endale T, et al. Strengthening research capacity: a systematic review of manuscript writing and publishing interventions for researchers in low-income and middle-income countries. BMJ Glob Heal. 2022;7(2):8059. doi:10.1136/BMJGH-2021-008059
  2. Aronson JK. When I use a word. “Publish or perish”: adverse effects. BMJ. 2025;390. doi:10.1136/BMJ.R1577
  3. Jefferies D, McNally S, Roberts K, et al. The importance of academic literacy for undergraduate nursing students and its relationship to future professional clinical practice: A systematic review. Nurse Educ Today. 2018;60:84-91. doi:10.1016/J.NEDT.2017.09.020
  4. Sweet L, Borrelli S, Matvienko-Sikar K, Kuipers Y, Fox D. Writing for publication: The basics. Women and Birth. 2025;38(2):101878. doi:10.1016/J.WOMBI.2025.101878
  5. Zhang Y, Eto H, Cui J. Linguistic challenges of writing papers in English for scholarly publication: Perceptions of Chinese academics in science and engineering. PLoS One. 2025;20(5):e0324760. doi:10.1371/JOURNAL.PONE.0324760
  6. Huang J, Tan M. The role of ChatGPT in scientific communication: writing better scientific review articles. Am J Cancer Res. 2023;13(4):1148. Accessed January 21, 2026. https://pmc.ncbi.nlm.nih.gov/articles/PMC10164801/
  7. Liebrenz M, Schleifer R, Buadze A, Bhugra D, Smith A. Generating scholarly content with ChatGPT: ethical challenges for medical publishing. Lancet Digit Heal. 2023;5(3):e105-e106. doi:10.1016/S2589-7500(23)00019-5
  8. Giglio A Del, da Costa MUP. The use of artificial intelligence to improve the scientific writing of non-native english speakers. Rev Assoc Med Bras. 2023;69(9):e20230560. doi:10.1590/1806-9282.20230560
  9. Elali FR, Rachid LN. AI-generated research paper fabrication and plagiarism in the scientific community. Patterns. 2023;4(3):100706. doi:10.1016/j.patter.2023.100706
  10. Chen Z, Chen C, Yang G, et al. Research integrity in the era of artificial intelligence: Challenges and responses. Med (United States). 2024;103(27):e38811. doi:10.1097/MD.0000000000038811
  11. Agrawal A. Fairness in AI-Driven Oncology: Investigating Racial and Gender Biases in Large Language Models. Cureus. 2024;16(9):e69541. doi:10.7759/CUREUS.69541
  12. Kar SK, Bansal T, Modi S, Singh A. How Sensitive Are the Free AI-detector Tools in Detecting AI-generated Texts? A Comparison of Popular AI-detector Tools. Indian J Psychol Med. 2025;47(3):275-278. doi:10.1177/02537176241247934
  13. Sajid M, Sanaullah M, Fuzail M, Malik TS, Shuhidan SM. Comparative analysis of text-based plagiarism detection techniques. PLoS One. 2025;20(4):e0319551. doi:10.1371/JOURNAL.PONE.0319551
  14. Ganjavi C, Eppler MB, Pekcan A, et al. Publishers’ and journals’ instructions to authors on use of generative artificial intelligence in academic and scientific publishing: bibliometric analysis. BMJ. 2024;384. doi:10.1136/BMJ-2023-077192
  15. Salvagno M, Taccone FS, Gerli AG. Can artificial intelligence help for scientific writing? Crit Care 2023 271. 2023;27(1):75-. doi:10.1186/S13054-023-04380-2
  16. Leung TI, de Azevedo Cardoso T, Mavragani A, Eysenbach G. Best Practices for Using AI Tools as an Author, Peer Reviewer, or Editor. J Med Internet Res. 2023;25(1):e51584. doi:10.2196/51584
  17. Xu X, Chen Y, Miao J. Opportunities, challenges, and future directions of large language models, including ChatGPT in medical education: a systematic scoping review. J Educ Eval Health Prof. 2024;21. doi:10.3352/JEEHP.2024.21.6
  18. Kacena MA, Plotkin LI, Fehrenbacher JC. The Use of Artificial Intelligence in Writing Scientific Review Articles. Curr Osteoporos Reports 2024 221. 2024;22(1):115-121. doi:10.1007/S11914-023-00852-0
  19. Margetts TJ, Karnik SJ, Wang HS, et al. Use of AI Language Engine ChatGPT 4.0 to Write a Scientific Review Article Examining the Intersection of Alzheimer’s Disease and Bone. Curr Osteoporos Reports 2024 221. 2024;22(1):177-181. doi:10.1007/S11914-023-00853-Z
  20. Omar M, Nassar S, Hijazi K, Glicksberg BS, Nadkarni GN, Klang E. Generating credible referenced medical research: A comparative study of openAI’s GPT-4 and Google’s gemini. Comput Biol Med. 2025;185:109545. doi:10.1016/J.COMPBIOMED.2024.109545
  21. Walters WH, Wilder EI. Fabrication and errors in the bibliographic citations generated by ChatGPT. Sci Reports 2023 131. 2023;13(1):14045-. doi:10.1038/s41598-023-41032-5
  22. Chelli M, Descamps J, Lavoué V, et al. Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis. J Med Internet Res. 2024;26(1):e53164. doi:10.2196/53164
  23. Bender EM, Gebru T, McMillan-Major A, Shmitchell S. On the dangers of stochastic parrots: Can language models be too big? FAccT 2021 - Proc 2021 ACM Conf Fairness, Accountability, Transpar. 2021;21:610-623. doi:10.1145/3442188.3445922
  24. Kampa RK, Padhan DK, Karna N, Gouda J. Identifying the factors influencing plagiarism in higher education: An evidence-based review of the literature. Account Res. 2025;32(2):83-98. doi:10.1080/08989621.2024.2311212
  25. Jenkins R, Lin P. AI-Assisted Authorship: How to Assign Credit in Synthetic Scholarship. SSRN Electron J. Published online 30 January 2023. doi:10.2139/SSRN.4342909
  26. Flanagin A, Bibbins-Domingo K, Berkwits M, Christiansen SL. Nonhuman “Authors” and Implications for the Integrity of Scientific Publication and Medical Knowledge. JAMA. 2023;329(8):637-639. doi:10.1001/JAMA.2023.1344
  27. Oermann MH, Owens JK, Carter-Templeton H, Peterson G, Bailey HE. Using Artificial Intelligence for Scholarly Writing. Am J Nurs. 2025;125(11):52-55. doi:10.1097/AJN.0000000000000179
  28. Lee JY. Can an artificial intelligence chatbot be the author of a scholarly article? J Educ Eval Health Prof. 2023;20. doi:10.3352/JEEHP.2023.20.6
  29. Bobier C, Rodger D, Hurst D. Artificial intelligence policies in bioethics and health humanities: a comparative analysis of publishers and journals. BMC Med Ethics 2025 261. 2025;26(1):79-. doi:10.1186/S12910-025-01239-9
  30. Tang A, Li KK, Kwok KO, Cao L, Luong S, Tam W. The importance of transparency: Declaring the use of generative artificial intelligence (AI) in academic writing. J Nurs Scholarsh. 2024;56(2):314-318. doi:10.1111/JNU.12938
  31. Dergaa I, Chamari K, Zmijewski P, Saad H Ben. From human writing to artificial intelligence generated text: examining the prospects and potential threats of ChatGPT in academic writing. Biol Sport. 2023;40(2):615-622. doi:10.5114/BIOLSPORT.2023.125623