SavedAdd your journal

Journals / Yalvaç Akademi Dergisi / 2026 / Cilt: 11 Sayı: 1

Leakage-Aware Cross-Dataset Evaluation of Prompt Injection Detection Using Classical Machine Learning and Transformer Models

Klasik Makine Öğrenmesi ve Transformer Modelleriyle İstem Enjeksiyonu Tespiti: Veri Sızıntısı Duyarlı ve Veri Kümeleri Arası Bir Değerlendirme

Oğuzhan Kilim(ISPARTA UYGULAMALI BİLİMLER ÜNİVERSİTESİ, KEÇİBORLU MESLEK YÜKSEKOKULU)0000-0003-3365-7327
Pages
102–131
Publication Date
2026-09-02
🏛️Bibliographic metadata and full text access for this article are provided via DergiPark.
DergiPark Record ↗
✨

AI Summary & Key Takeaways

AI Insight
⚡ TL;DR(Too Long; Didn't Read)

The results show that internal validation results are not enough for prompt injection detection.

📌 3 Key Takeaways

🎯
Objective & Hypothesis:The widespread adoption of systems based on Large Language Models has made the reliable detection of prompt injection attacks a critical requirement.
📊
Key Findings & Data:However, high performance achieved on training and test splits generated from the same data source does not guarantee that models can generalize to prompts from different sources.
💡
Conclusion & Impact:The results show that internal validation results are not enough for prompt injection detection.
🔬Design:Modeling & Simulation
📍Scope:Comprehensive Sample
⚙️Variables:İstem enjeksiyonu · Büyük Dil Modelleri · Veri sızıntısı

Abstract

The widespread adoption of systems based on Large Language Models has made the reliable detection of prompt injection attacks a critical requirement. However, high performance achieved on training and test splits generated from the same data source does not guarantee that models can generalize to prompts from different sources. In this study, a leak-aware cross-dataset evaluation framework is presented to examine the robustness of classical machine learning and Transformer-based prompt injection detection models in the face of data source changes. During the data preparation process, empty records, duplicate prompts, conflicting labels, and text overlaps between datasets were checked. In this context, 38,184 duplicate records and 14 instances with conflicting labels were removed, and the 198 common prompts identified between the training and external test sets were removed only from the training set. Using WordHash and CharHash representations, SGD Logistic and Linear SVM models, as well as DistilBERT and DeBERTa-v3-small, were evaluated on an internal dataset consisting of 426,073 cleaned requests; the models were also tested on an independent dataset of 5,000 examples. While the models achieved performance in the range of approximately 0.997–1.000 in the internal evaluation, significant performance losses were observed in the external evaluation. DeBERTa-v3-small delivered the most balanced results, with an accuracy of 0.7360, a balanced accuracy of 0.7320, a Macro-F1 of 0.7284, and an attack sensitivity of 0.7119. The domain classifier achieved an ROC-AUC of 0.9770 hence indicating a remarkable shift in the distribution across data sources. The results show that internal validation results are not enough for prompt injection detection. Independent external validation, data leakage verification and domain shift analysis should be fundamental parts of reliable model evaluation.

Özet

Büyük Dil Modellerine dayalı sistemlerin yaygınlaşması, prompt injection saldırılarının güvenilir biçimde tespit edilmesini kritik bir gereksinim hâline getirmiştir. Bununla birlikte, aynı veri kaynağından oluşturulan eğitim ve test bölmeleri üzerinde elde edilen yüksek performans, modellerin farklı kaynaklardan gelen istemlere genellenebildiğini garanti etmemektedir. Bu çalışmada, klasik makine öğrenmesi ve Transformer tabanlı prompt injection tespit modellerinin veri kaynağı değişimi karşısındaki dayanıklılığını incelemek amacıyla sızıntı farkındalıklı bir çapraz veri kümesi değerlendirme çerçevesi sunulmuştur. Veri hazırlama sürecinde boş kayıtlar, yinelenen istemler, çelişkili etiketler ve veri kümeleri arasındaki metin örtüşmeleri denetlenmiştir. Bu kapsamda 38.184 yinelenen kayıt ile çelişkili etiketlere sahip 14 örnek çıkarılmış, eğitim ve dış test kümeleri arasında belirlenen 198 ortak istem yalnızca eğitim kümesinden kaldırılmıştır. Temizlenmiş 426.073 istemden oluşan iç veri kümesi üzerinde WordHash ve CharHash temsilleriyle SGD Logistic ve Linear SVM modelleri ile DistilBERT ve DeBERTa-v3-small değerlendirilmiş; modeller ayrıca 5.000 örnekten oluşan bağımsız bir veri kümesinde sınanmıştır. İç değerlendirmede modeller yaklaşık 0,997–1,000 aralığında performans üretirken, dış değerlendirmede belirgin performans kayıpları gözlenmiştir. DeBERTa-v3-small, 0,7360 doğruluk, 0,7320 dengeli doğruluk, 0,7284 Macro-F1 ve 0,7119 saldırı duyarlılığı ile en dengeli sonucu sağlamıştır. Alan sınıflandırıcısının 0,9770 ROC-AUC elde etmesi, veri kaynakları arasında belirgin bir dağılım kayması bulunduğunu göstermiştir. Bulgular, prompt injection tespitinde iç doğrulama sonuçlarının tek başına yeterli olmadığını; bağımsız dış doğrulama, veri sızıntısı denetimi ve alan kayması analizinin güvenilir model değerlendirmesinin temel bileşenleri olması gerektiğini ortaya koymaktadır.

🔖

Cite This Article

Kilim, O., *, O. K. (2026). Leakage-Aware Cross-Dataset Evaluation of Prompt Injection Detection Using Classical Machine Learning and Transformer Models. Yalvaç Akademi Dergisi, Cilt: 11 Sayı: 1, 102–131. https://doi.org/10.57120/yalvac.2001116

📄 Full Text PDF Viewer

📄

PDF görüntüleyici hazırlanıyor...

References (32)

  1. Yao, Y., Duan, J., Xu, K., Cai, Y., Sun, Z., & Zhang, Y. (2024). A survey on large language model (llm) security and privacy: The good, the bad, and the ugly. High-Confidence Computing, 4(2), 100211., doi: 10.1016/j.hcc.2024.100211.
  2. Raza, M., Jahangir, Z., Riaz, M. B., Saeed, M. J., & Sattar, M. A. (2025). Industrial applications of large language models. Scientific Reports, 15(1), 13755, doi: 10.1038/s41598-025-98483-1.
  3. Milani, A., Franzoni, V., & Florindi, E. (2026). Indirect prompt injection in large language models. Neural Computing and Applications, 38(13), 530, doi: 10.1007/s00521-026-12266-x.
  4. Berini, A. D. E., Jamil, N., Benrazek, A. E., Lakas, A., Ismail, L., Ferrag, M. A., & Lam, K. Y. (2026). Security and privacy in LLMs: A comprehensive survey of threats and mitigation strategies. Information Fusion, 104241, doi: 10.1016/j.inffus.2026.104241.
  5. Alzahrani, A. (2026). PromptGuard a structured framework for injection resilient language models. Scientific Reports, 16(1), 1277, doi: 10.1038/s41598-025-31086-y.
  6. Srinivasan, J., Regi, S. A., Anbarasan, A. K., A, S., T, V., & Venu, S. (2026). Detection and analysis of prompt injection in indian multilingual large language models. Scientific Reports, doi: 10.1038/s41598-026-43883-0.
  7. Alahmari, S. S., Hall, L., Mouton, P. R., & Goldgof, D. (2025). Large language models robustness against perturbation: S. Alahmari et al. Scientific Reports, doi: 10.1038/s41598-025-29770-0.
  8. Kim, J., Podlasek, A., Shidara, K., Liu, F., Alaa, A., & Bernardo, D. (2025). Limitations of large language models in clinical problem-solving arising from inflexible reasoning. Scientific reports, 15(1), 39426, doi: 10.1038/s41598-025-22940-0.
  9. J. Zhang, H. Bu, H. Wen, et al., “When LLMs meet cybersecurity: A systematic literature review,” Cybersecurity, vol. 8, Art. no. 55, 2025, doi: 10.1186/s42400-025-00361-w.
  10. Y. Dong, R. Mu, Y. Zhang, et al., “Safeguarding large language models: A survey,” Artificial Intelligence Review, vol. 58, Art. no. 382, 2025, doi: 10.1007/s10462-025-11389-2.
  11. T. Geng, Z. Xu, Y. Qu, and W. E. Wong, “Prompt injection attacks on large language models: A survey of attack methods, root causes, and defense strategies,” Computers, Materials & Continua, vol. 87, no. 1, Art. no. 4, 2026, doi: 10.32604/cmc.2025.074081.
  12. T. Ivănușcă and C.-I. Irimia, “The impact of prompting techniques on the security of the LLMs and the systems to which they belong,” Applied Sciences, vol. 14, no. 19, Art. no. 8711, 2024, doi: 10.3390/app14198711.
  13. R. Lapid, R. Langberg, and M. Sipper, “Open Sesame! Universal black-box jailbreaking of large language models,” Applied Sciences, vol. 14, no. 16, Art. no. 7150, 2024, doi: 10.3390/app14167150.
  14. N. Dzhaliuk, D. Sabodashko, V. Khoma, et al., “Comparative evaluation of machine learning methods for protecting LLMs from prompt injection attacks,” International Journal of Information Security, vol. 25, Art. no. 109, 2026, doi: 10.1007/s10207-026-01264-8.
  15. J. Clusmann, D. Ferber, I. C. Wiest, et al., “Prompt injection attacks on vision language models in oncology,” Nature Communications, vol. 16, Art. no. 1239, 2025, doi: 10.1038/s41467-024-55631-x.
  16. Y. Yang, Q. Jin, F. Huang, et al., “Adversarial prompt and fine-tuning attacks threaten medical large language models,” Nature Communications, vol. 16, Art. no. 9011, 2025, doi: 10.1038/s41467-025-64062-1.
  17. S. S. Karlsen, M. M. Yamin, E. Hashmi, B. Katt, and M. Ullah, “Securing large language models: A quantitative assurance framework approach,” Journal of Information Security and Applications, vol. 97, Art. no. 104351, 2026, doi: 10.1016/j.jisa.2025.104351.
  18. G. Chuk, Benign–Malicious Prompt Classification Dataset, Hugging Face Datasets, 2024. Available: Hugging Face dataset card.
  19. Rogue Security, Prompt Injections Benchmark Dataset, Hugging Face Datasets, 2025.
  20. C. Cortes and V. Vapnik, “Support-vector networks,” Machine Learning, vol. 20, no. 3, pp. 273–297, 1995.
  21. F. Pedregosa et al., “Scikit-learn: Machine Learning in Python,” Journal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011.
  22. V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “DistilBERT: A distilled version of BERT: Smaller, faster, cheaper and lighter,” arXiv preprint arXiv:1910.01108, 2019.
  23. P. He, X. Liu, J. Gao, and W. Chen, “DeBERTa: Decoding-enhanced BERT with disentangled attention,” in International Conference on Learning Representations (ICLR), 2021.
  24. B. Efron and R. Tibshirani, An Introduction to the Bootstrap. Chapman & Hall, 1993.
  25. L. Zhou, W. Schellaert, F. Martínez-Plumed, Y. Moros-Daval, C. Ferri, and J. Hernández-Orallo, “Larger and more instructable language models become less reliable,” Nature, vol. 634, pp. 61–68, 2024, doi: 10.1038/s41586-024-07930-y.
  26. Y. Ektefaie, A. Shen, D. Bykova, et al., “Evaluating generalizability of artificial intelligence models for molecular datasets,” Nature Machine Intelligence, vol. 6, pp. 1512–1524, 2024, doi: 10.1038/s42256-024-00931-6.
  27. C. Ong Ly, B. Unnikrishnan, T. Tadic, et al., “Shortcut learning in medical AI hinders generalization: Method for estimating AI model generalization without external data,” npj Digital Medicine, vol. 7, Art. no. 124, 2024, doi: 10.1038/s41746-024-01118-4.
  28. X. Huang, W. Ruan, W. Huang, et al., “A survey of safety and trustworthiness of large language models through the lens of verification and validation,” Artificial Intelligence Review, vol. 57, Art. no. 175, 2024, doi: 10.1007/s10462-024-10824-0.
  29. S. Pelekis, T. Koutroubas, A. Blika, et al., “Adversarial machine learning: A review of methods, tools, and critical industry sectors,” Artificial Intelligence Review, vol. 58, Art. no. 226, 2025, doi: 10.1007/s10462-025-11147-4.
  30. Y. Cai, “Prompt injection attacks on educational large language models for higher and vocational education,” Scientific Reports, vol. 16, Art. no. 15594, 2026, doi: 10.1038/s41598-026-46563-1.
  31. M. A. Ferrag, N. Tihanyi, D. Hamouda, L. Maglaras, A. Lakas, and M. Debbah, “From prompt injections to protocol exploits: Threats in LLM-powered AI agents workflows,” ICT Express, vol. 12, no. 2, pp. 353–383, 2026, doi: 10.1016/j.icte.2025.12.001.
  32. D. A. Alber, Z. Yang, A. Alyakin, et al., “Medical large language models are vulnerable to data-poisoning attacks,” Nature Medicine, vol. 31, pp. 618–626, 2025, doi: 10.1038/s41591-024-03445-1.