SavedAdd your journal

Journals / Scientific Journal of Mehmet Akif Ersoy University / 2025 / Volume: 8 Issue: 2

Machine Learning Based Identification of LLM Generated Scientific Research Article Abstracts

Burcu Baştürk(IZMIR KATIP CELEBI UNIVERSITY)0009-0005-4781-353X
Aytuğ Onan(IZMIR KATIP CELEBI UNIVERSITY)0000-0002-9434-5880
Pages
—
Publication Date
2025-12-26
🏛️Bibliographic metadata and full text access for this article are provided via DergiPark.
DergiPark Record ↗
✨

AI Summary & Key Takeaways

AI Insight
⚡ TL;DR(Too Long; Didn't Read)

This study highlights the significant potential of ML techniques in ensuring transparency and reliability in AI-driven medical decision-making, especially in the area of heart disease diagnosis.

📌 3 Key Takeaways

🎯
Objective & Hypothesis:Heart disease is one of the leading causes of death worldwide, making early detection and diagnosis essential for effective treatment.
📊
Key Findings & Data:As AI systems become more complex, it becomes important to distinguish between abstracts generated by AI algorithms and those prepared by human experts.
💡
Conclusion & Impact:This study highlights the significant potential of ML techniques in ensuring transparency and reliability in AI-driven medical decision-making, especially in the area of heart disease diagnosis.
🔬Design:Modeling & Simulation
📍Scope:Comprehensive Sample
⚙️Variables:Machine Learning · Artificial Intelligence · Natural Language Processing

Abstract

Heart disease is one of the leading causes of death worldwide, making early detection and diagnosis essential for effective treatment. With advancements in machine learning (ML) and artificial intelligence (AI), these technologies are being increasingly applied in the medical field, particularly for detecting and predicting heart disease. As AI systems become more complex, it becomes important to distinguish between abstracts generated by AI algorithms and those prepared by human experts. This study aims to develop and assess ML approaches to distinguish between human-written and AI-generated (ChatGPT and NLTK) heart disease abstracts. Using a dataset of 15,000 abstracts (5,000 written by humans, 5,000 reworded by ChatGPT, and 5,000 generated using NLTK), various Natural Language Processing (NLP) techniques, such as tokenization, stop word removal, stemming and lemmatization were applied. The text data was transformed into numerical form using TF-IDF vectorization. Different ML models, including K-nearest neighbors (KNN), support vector machines (SVMs), logistic regression, random forest, decision tree were trained and tested for their classification accuracy. This study highlights the significant potential of ML techniques in ensuring transparency and reliability in AI-driven medical decision-making, especially in the area of heart disease diagnosis.

🔖

Cite This Article

Baştürk, B., Onan, A. (2025). Machine Learning Based Identification of LLM Generated Scientific Research Article Abstracts. Scientific Journal of Mehmet Akif Ersoy University, Volume: 8 Issue: 2. https://doi.org/10.70030/sjmakeu.1730246

📄 Full Text PDF Viewer

📄

PDF görüntüleyici hazırlanıyor...

References (42)

  1. Internet: World diseases(CVDs), Health Organization, Cardiovascular https://www.who.int/news-room/fact sheets/detail/cardiovascular-diseases-(cvds), 22.09.2024.
  2. LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep Learning. Nature, 521(7553), 436–444. https://doi.org/10.1038/nature14539
  3. Bhatt, A. (2020). Healthcare predictive analytics using machine learning and deep learning techniques: a survey. Journal of Electrical Systems and Information Technology, 7(2), 13–19.
  4. Russell, S., & Norvig, P. (2010). Artificial Intelligence: A Modern Approach (3rd ed.). Prentice Hall, Upper Saddle River, NJ.
  5. Mitchell, T. M. (1997). Machine Learning. McGraw-Hill, New York.
  6. Jurafsky, D., & Martin, J. H. (2021). Speech and Language Processing (3rd ed.). Pearson, San Francisco, CA.
  7. Krittanawong, C., Zhang, H., Wang, Z., et al. (2017). Artificial Intelligence in Precision Cardiovascular Medicine. Journal of the American College of Cardiology, 69(21), 2657–2664. https://doi.org/10.1016/j.jacc.2017.03.571
  8. Ouyang, D., He, B., Ghorbani, A., Yuan, N., Ebinger, J., Langlotz, P., Heidenreich, P. A., Harrington, R. A., Liang, D. H., Ashley, E. A., & Zou, J. Y. (2020). Video-based AI for beat-to-beat assessment of cardiac function. Nature, 580(7802), 252–256. https://doi.org/10.1038/s41586-020-2145-8
  9. Goodfellow, I., Shlens, J., & Szegedy, C. (2015). Explaining and Harnessing Adversarial Examples. International Conference on Learning Representations (ICLR), 1–9.
  10. Cingillioğlu, İ. (2023). Detecting AI-generated essays: the ChatGPT challenge. International Journal of Information and Learning Technology, 40(3), 259–268. https://doi.org/10.1108/IJILT-03-2023-0043
  11. Ribeiro, M., Singh, S., & Guestrin, C. (2016). Why Should I Trust You? Explaining the Predictions of Any Classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '16), 1135–1144. https://doi.org/10.1145/2939672.2939778
  12. Sebastiani, A. (2002). Machine Learning in Automated Text Categorization. ACM Computing Surveys, 34(1), 1–47. https://doi.org/10.1145/505282.505283
  13. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., & Amodei, D. (2020). Language models are few-shot learners (arXiv:2005.14165). Retrieved from https://arxiv.org/abs/2005.14165
  14. Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space (arXiv:1301.3781). Retrieved from https://arxiv.org/abs/1301.3781
  15. Joachims, T. (1998). Text categorization with support vector machines: Learning with many relevant features. In Nédellec, C., & Rouveirol, C. (eds.), Machine Learning: ECML-98 (Vol. 1398, pp. 137–142). Springer, Berlin, Heidelberg.
  16. Mathur, P., Srivastava, M., Kulshreshtha, A. D., Gulati, P., & Singh, A. (2020). Artificial intelligence, machine learning, and cardiovascular disease. Clinical Medicine Insights: Cardiology, 14, 1–7. https://doi.org/10.1177/1179546820927404.
  17. Gadde, S. S., & Kalli, V. D. R. (2020). Applications of artificial intelligence in medical devices and healthcare. International Journal of Computer Science Trends and Technology, 8, 182–188.
  18. Mishra, V., Sarraju, A., Kalwani, N. M., & Dexter, J. P. (2024). Evaluation of prompts to simplify cardiovascular disease information generated using a large language model: Cross-sectional study. Journal of Medical Internet Research, 26. https://doi.org/10.2196/51795
  19. Bhattaru, A., Yanamala, N., & Sengupta, P. (2024). Revolutionizing cardiology with words: Unveiling the impact of large language models in medical science writing. Canadian Journal of Cardiology.
  20. Liao, W., Liu, Z., Dai, H., Xu, S., Wu, Z., Zhang, Y., Huang, X., Zhu, D., Cai, H., Li, Q., Liu, T., & Li, X. (2023). Differentiating ChatGPT-generated and human-written medical texts: Quantitative study. JMIR Medical Education, 9(1), e48904. https://doi.org/10.2196/48904
  21. Theocharopoulos, P. C., Anagnostou, P., Tsoukala, A., Georgakopoulos, S. V., Tasoulis, S. K., & Plagianakos, V. P. (2023). Detection of fake generated scientific abstracts (arXiv preprint No. 2304.06148). Retrieved from https://arxiv.org/abs/2304.06148
  22. Kumar, V., Bharti, A., Verma, D., & Bhatnagar, V. (2023). Deep dive into language traits of AI-generated abstracts (arXiv preprint No. 2312.10617). Retrieved from https://arxiv.org/abs/2312.10617
  23. Doru, B., Maier, C., Busse, J., Lücke, T., Schönhoff, J., Enax-Krumova, E., Hessler, S., Berger, M., & Tokic, M. (2025). Detecting artificial intelligence–generated versus human-written medical student essays: Semirandomized controlled study. JMIR Medical Education, 11, e62779. https://doi.org/10.2196/62779
  24. Mallapaty, S. (2025). Signs of AI-generated text found in 14% of biomedical abstracts. Nature. https://www.nature.com/articles/d41586-025-02097-6
  25. U.S. National Library of Medicine. PubMed, from https://pubmed.ncbi.nlm.nih.gov, accessed on 2024-09-26.
  26. Biswas, S. S. (2023). Role of Chat GPT in public health. Annals of Biomedical Engineering, 51(5), 868–869. https://doi.org/10.1007/s10439-023-03172-7
  27. Miller, G. A. (1995). WordNet: A lexical database for English. Communications of the ACM, 38, 39–41. https://doi.org/10.1145/219717.219748
  28. Grefenstette, G. (1999). Tokenization. In Voutilainen, A., Heikkilä, J., & Anttila, A. (eds.), Syntactic wordclass tagging, Springer, Dordrecht, pp. 117–133.
  29. Kaur, J., & Buttar, P. K. (2018). A systematic review on stopword removal algorithms. International Journal on Future Revolution in Computer Science & Communication Engineering, 4(4), 207–210.
  30. Kannan, S., & Gurusamy, V. (2014). Preprocessing techniques for text mining. International Journal of Computer Science & Communication Networks, 5(1), 7–16.
  31. Sparck Jones, K. (1972). A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28(1), 11–21. https://doi.org/10.1108/eb026526
  32. Kohavi, R. (1995). A study of cross-validation and bootstrap for accuracy estimation and model selection. Proceedings of the 14th International Joint Conference on Artificial Intelligence (IJCAI), Montreal, Canada, pp. 1137–1145.
  33. Hosmer Jr, D. W., Lemeshow, S., & Sturdivant, R. X. (2013). Applied logistic regression. John Wiley & Sons, New York.
  34. Charbuty, B., & Abdulazeez, A. (2021). Classification based on decision tree algorithm for machine learning. Journal of Applied Science and Technology Trends, 2(1), 20–28. https://doi.org/10.38094/jastt20165
  35. Breiman, L. (2017). Classification and regression trees. Routledge, New York.
  36. Kulkarni, V. Y., & Sinha, P. K. (2012). Pruning of random forest classifiers: A survey and future directions. Proceedings of the International Conference on Data Science & Engineering (ICDSE), pp. 64–68. IEEE.
  37. Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273–297. https://doi.org/10.1007/BF00994018
  38. Abdel Hady, M. F., & Schwenker, F. (2010). Combining committee-based semi-supervised learning and active learning. Journal of Computer Science and Technology, 25(4), 681–698. https://doi.org/10.1007/s11390-010-9357-6 dl.acm.org+8
  39. Han, J., Kamber, M., & Pei, J. (2011). Data Mining: Concepts and Techniques (3rd ed.). Morgan Kaufmann, San Francisco. https://doi.org/10.1016/C2009-0-61819-5
  40. Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). Springer, New York.
  41. Manning, C. D., Raghavan, P., & Schütze, H. (2008). Introduction to information retrieval. Cambridge University Press, Cambridge.
  42. Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7, 1–30.