KaydedilenlerDerginizi ekleyin

Dergiler / Sakarya University Journal of Computer and Information Sciences / 2026 / Volume: 9 Issue: 4

Evaluating Large Language Models for Biomedical Text Summarization: A Study of Cardiovascular Research

Burcu Baştürk(IZMIR KATIP CELEBI UNIVERSITY)0009-0005-4781-353X
Aytuğ Onan(IZMIR KATIP CELEBI UNIVERSITY)0000-0002-9434-5880
Sayfa
—
Yayın Tarihi
2026-09-30
🏛️Bu makalenin bibliyografik verileri ve tam metin bağlantısı DergiPark üzerinden sağlanmıştır.
DergiPark Kaydı ↗
✨

Yapay Zeka Hızlı Bakış & Çıkarımlar

AI Insight
⚡ TL;DR(1 Cümlede Hızlı Çıkarım)

This work highlights critical differences in LLM behavior depending on the summarization method and offers a benchmark dataset and evaluation pipeline for future research on AI-assisted biomedical summarization.

📌 3 Maddede Bu Makale

🎯
Amaç & Hipotez:In this study, a comprehensive evaluation of abstractive and extractive summarization performance across three prominent large language models (LLMs): ChatGPT, DeepSeek, and Gemini is presented.
📊
Temel Bulgular & Veri:A total of 8,000 cardiovascular-related research abstracts were collected from PubMed and summarized using two distinct prompting strategies: abstractive and extractive.
💡
Sonuç & Katkı:This work highlights critical differences in LLM behavior depending on the summarization method and offers a benchmark dataset and evaluation pipeline for future research on AI-assisted biomedical summarization.
🔬Tür:Modelleme & Sayısal Simülasyon
📍Kapsam:Genişletilmiş Akademik Örneklem
⚙️Değişkenler:Abstractive summarization · Cardiovascular research · Extractive summarization

Abstract

In this study, a comprehensive evaluation of abstractive and extractive summarization performance across three prominent large language models (LLMs): ChatGPT, DeepSeek, and Gemini is presented. A total of 8,000 cardiovascular-related research abstracts were collected from PubMed and summarized using two distinct prompting strategies: abstractive and extractive. This process yielded a dataset of 48,000 summaries. To assess summarization quality, applied a multi-metric evaluation framework including semantic similarity (SBERT cosine), BLEU, GLEU, ROUGE-1 F1, ROUGE-2 F1, ROUGE-L F1, and METEOR. The results indicate that extractive summaries, particularly those generated by ChatGPT, consistently achieve higher scores across most metrics, suggesting stronger lexical fidelity and sequence retention. While Gemini shows balanced performance between abstraction and extraction, DeepSeek yields lower scores in both approaches. This work highlights critical differences in LLM behavior depending on the summarization method and offers a benchmark dataset and evaluation pipeline for future research on AI-assisted biomedical summarization.

🔖

Bu Makaleye Atıf Yap

Baştürk, B., Onan, A. (2026). Evaluating Large Language Models for Biomedical Text Summarization: A Study of Cardiovascular Research. Sakarya University Journal of Computer and Information Sciences, Volume: 9 Issue: 4. https://doi.org/10.35377/saucis...1780353

📄 Tam Metin PDF Okuyucu

📄

PDF görüntüleyici hazırlanıyor...

Kaynakça (28)

  1. U.S. National Library of Medicine, “Search results for cardiovascular diseases,” [Online]. Available: https://pubmed.ncbi.nlm.nih.gov/?term=cardiovascular+diseases. [Accessed: 27-Jun-2025].
  2. Z. Lu, “PubMed and beyond: a survey of web tools for searching biomedical literature,” Database (Oxford), 2011, Art. no. baq036, doi: 10.1093/database/baq036.
  3. B. C. Wallace, J. Kuiper, A. Sharma, M. B. Zhu, and I. J. Marshall, “Extracting PICO sentences from clinical trial reports using supervised distant supervision,” Journal of Machine Learning Research, vol. 17, p. 132, 2016.
  4. A. Rahman et al., “Comparative analysis based on DeepSeek, ChatGPT, and Google Gemini: Features, techniques, performance, future prospects,” arXiv preprint, arXiv:2503.04783, 2025, doi: 10.48550/arXiv.2503.04783.
  5. H. Zhang, X. Liu, and J. Zhang, “Extractive summarization via ChatGPT for faithful summary generation,” Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, Dec. 6–10, 2023.
  6. M. Wang, M. Wang, F. Yu, Y. Yang, J. Walker, and J. Mostafa, “A systematic review of automatic text summarization for biomedical literature and EHRs,” Journal of the American Medical Informatics Association, vol. 28, no. 10, pp. 2287–2297, 2021, doi: 10.1093/jamia/ocab143.
  7. A. Chaves, C. Kesiku, and B. Garcia-Zapirain, “Automatic text summarization of biomedical text data: A systematic review,” Information, vol. 13, no. 8, p. 393, 2022, doi: 10.3390/info13080393.
  8. N. Kemaloğlu Alagöz and E. U. Küçüksille, “System of automatic scientific article summarization in Turkish,” Pamukkale University Journal of Engineering Sciences, vol. 30, no. 4, pp. 470–481, 2024.
  9. G. Quer and E. J. Topol, “The potential for large language models to transform cardiovascular medicine,” The Lancet Digital Health, vol. 6, no. 10, pp. e767–e771, 2024.
  10. Q. Xie, Z. Luo, B. Wang, and S. Ananiadou, “A survey for biomedical text summarization: From pre-trained to large language models,” arXiv preprint, arXiv:2304.08763, 2023, doi: 10.48550/arXiv.2304.08763.
  11. A. Givchi, R. Ramezani, and A. Baraani-Dastjerdi, “Graph-based abstractive biomedical text summarization,” Journal of Biomedical Informatics, vol. 132, p. 104099, 2022, doi: 10.1016/j.jbi.2022.104099.
  12. L. Bednarczyk et al., “Scientific evidence for clinical text summarization using large language models: Scoping review,” Journal of Medical Internet Research, vol. 27, p. e68998, 2025, doi: 10.2196/68998.
  13. L. Tang et al., “Evaluating large language models on medical evidence summarization,” NPJ Digital Medicine, vol. 6, no. 1, p. 158, 2023, doi: 10.1038/s41746-023-00896-7.
  14. E. Croxford et al., “Evaluation of large language models for summarization tasks in the medical domain: A narrative review,” arXiv preprint, arXiv:2409.18170, 2024, doi: 10.48550/arXiv.2409.18170.
  15. D. Van Veen et al., “Adapted large language models can outperform medical experts in clinical text summarization,” Nature Medicine, vol. 30, pp. 1134–1142, 2024, doi: 10.1038/s41591-024-02855-5.
  16. U.S. National Library of Medicine, “PubMed overview,” [Online]. Available: https://pubmed.ncbi.nlm.nih.gov/about/. [Accessed: 30-Jun-2025].
  17. OpenAI, “GPT-4 technical report,” [Online]. Available: https://openai.com/research/gpt-4. [Accessed: 30-Jun-2025].
  18. S. Mercer, S. Spillard, and D. P. Martin, “Brief analysis of DeepSeek R1 and its implications for generative AI,” arXiv preprint, arXiv:2502.02523, 2025, doi: 10.48550/arXiv.2502.02523.
  19. M. E. de Carvalho Souza and L. Weigang, “Grok, Gemini, ChatGPT and DeepSeek: Comparison and applications in conversational artificial intelligence,” Inteligencia Artificial, vol. 2, no. 1, 2025.
  20. D. Jurafsky and J. H. Martin, Speech and Language Processing, 3rd ed. Stanford, CA, USA: Stanford University, 2021.
  21. S. Bird, E. Klein, and E. Loper, Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit. Sebastopol, CA, USA: O’Reilly Media, Inc., 2009.
  22. S. Kannan, V. Gurusamy, S. Vijayarani, J. Ilamathi, and M. Nithya, “Preprocessing techniques for text mining,” International Journal of Computer Science & Communication Networks, vol. 5, no. 1, pp. 7–16, 2014.
  23. N. Reimers and I. Gurevych, “Sentence-BERT: Sentence embeddings using siamese BERT-networks,” arXiv preprint, arXiv:1908.10084, 2019, doi: 10.48550/arXiv.1908.10084.
  24. N. Reimers and I. Gurevych, “Making monolingual sentence embeddings multilingual using knowledge distillation,” arXiv preprint, arXiv:2004.09813, 2020, doi: 10.48550/arXiv.2004.09813.
  25. K. Papineni, S. Roukos, T. Ward, and W. J. Zhu, “BLEU: A method for automatic evaluation of machine translation,” in Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 311–318, Jul. 2002, doi: 10.3115/1073083.1073135.
  26. Y. Wu et al., “Google’s neural machine translation system: Bridging the gap between human and machine translation,” arXiv preprint, arXiv:1609.08144, 2016, doi: 10.48550/arXiv.1609.08144.
  27. C. Y. Lin, “ROUGE: A package for automatic evaluation of summaries,” in Text Summarization Branches Out Workshop, pp. 74–81, Jul. 2004.
  28. S. Banerjee and A. Lavie, “METEOR: An automatic metric for MT evaluation with improved correlation with human judgments,” in Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 65–72, Jun. 2005.