Journals / Yalvaç Akademi Dergisi / 2026 / Cilt: 11 Sayı: 1

Machine Learning-Based Anomaly Detection in Post-Quantum TLS 1.3 Traffic: A Comparative Analysis on the CIC-PQC_OAV v1 Dataset

Post-Kuantum TLS 1.3 Trafiğinde Makine Öğrenmesi Tabanlı Anomali Tespiti: CIC-PQC_OAV v1 Veri Seti Üzerinde Karşılaştırmalı Bir Analiz

Serkan Keskin(MEHMET AKİF ERSOY ÜNİVERSİTESİ, TEKNİK BİLİMLER MESLEK YÜKSEKOKULU)0000-0001-9404-5039
Pages
47–62
Publication Date
2026-09-02
🏛️Bibliographic metadata and full text access for this article are provided via DergiPark.
DergiPark Record ↗
✨

AI Summary & Key Takeaways

AI Insight
⚡ TL;DR(Too Long; Didn't Read)

The experimental results show that Decision Tree and Random Forest achieved the most balanced performance, with an Accuracy of 97.79% and an F1-Score of 90.31%.

📌 3 Key Takeaways

🎯
Objective & Hypothesis:The integration of post-quantum cryptographic algorithms into the TLS 1.3 protocol may alter the behavioral characteristics of network traffic and affect the performance of existing anomaly detection systems.
📊
Key Findings & Data:While secure communication in earlier uses of TLS largely relied on classical key exchange mechanisms, the inclusion of post-quantum algorithms in TLS 1.3 sessions may lead to different traffic behaviors.
💡
Conclusion & Impact:The experimental results show that Decision Tree and Random Forest achieved the most balanced performance, with an Accuracy of 97.79% and an F1-Score of 90.31%.
🔬Design:Modeling & Simulation
📍Scope:Comprehensive Sample
⚙️Variables:Post-kuantum kriptografi · TLS 1.3 · CIC-PQC_OAV v1

Abstract

The integration of post-quantum cryptographic algorithms into the TLS 1.3 protocol may alter the behavioral characteristics of network traffic and affect the performance of existing anomaly detection systems. While secure communication in earlier uses of TLS largely relied on classical key exchange mechanisms, the inclusion of post-quantum algorithms in TLS 1.3 sessions may lead to different traffic behaviors. Considering this change, this study performs machine learning-based anomaly detection on the CIC-PQC_OAV v1 dataset. Decision Tree, K-Nearest Neighbor, Support Vector Machine, Random Forest, and XGBoost algorithms were applied using the same training, validation, and test partitions. The models were compared in terms of Accuracy, Precision, Recall, F1-Score, and ROC-AUC. The experimental results show that Decision Tree and Random Forest achieved the most balanced performance, with an Accuracy of 97.79% and an F1-Score of 90.31%. XGBoost stood out in terms of class separability, achieving a ROC-AUC value of 98.35%. The findings indicate that post-quantum TLS 1.3 traffic presents a different and more challenging classification structure than traditional network traffic. Therefore, anomaly detection systems should be re-evaluated by considering the traffic characteristics specific to post-quantum network environments.

Özet

Post-kuantum kriptografik algoritmaların TLS 1.3 protokolüne entegrasyonu, ağ trafiğinin davranışsal özelliklerini değiştirebilmekte ve mevcut anomali tespit sistemlerinin başarısını etkileyebilmektedir. TLS protokolünün önceki kullanımlarında güvenli iletişim büyük ölçüde klasik anahtar değişim mekanizmalarına dayanırken, post-kuantum algoritmaların TLS 1.3 oturumlarına dâhil edilmesi ağ trafiğinin daha farklı bir davranış göstermesine neden olabilmektedir. Bu çalışmada, söz konusu değişim dikkate alınarak CIC-PQC_OAV v1 veri seti üzerinde makine öğrenmesi tabanlı anomali tespiti gerçekleştirilmiştir. Çalışma kapsamında Decision Tree, K-Nearest Neighbor, Support Vector Machine, Random Forest ve XGBoost algoritmaları aynı eğitim, doğrulama ve test ayrımları üzerinde uygulanmıştır. Modellerin performansları doğruluk, kesinlik, duyarlılık, F1-Skoru ve ROC-AUC ölçütleriyle karşılaştırılmıştır. Deneysel sonuçlar, Decision Tree ve Random Forest algoritmalarının %97,79 doğruluk ve %90,31 F1-Skoru değerleriyle en dengeli performansı sunduğunu göstermiştir. XGBoost ise %98,35 ROC-AUC değeriyle sınıflar arasındaki ayırt edicilik bakımından öne çıkmıştır. Bulgular, post-kuantum TLS 1.3 trafiğinin geleneksel ağ trafiğine göre daha farklı ve zorlayıcı bir sınıflandırma yapısı sunduğunu göstermektedir. Bu nedenle, anomali tespit sistemlerinin post-kuantum ağ ortamlarına özgü trafik özellikleri dikkate alınarak yeniden değerlendirilmesi gerektiği sonucuna ulaşılmıştır.

🔖

Cite This Article

Keskin, S., *, S. K. (2026). Machine Learning-Based Anomaly Detection in Post-Quantum TLS 1.3 Traffic: A Comparative Analysis on the CIC-PQC_OAV v1 Dataset. Yalvaç Akademi Dergisi, Cilt: 11 Sayı: 1, 47–62. https://doi.org/10.57120/yalvac.1987078

📄 Full Text PDF Viewer

📄

PDF görüntüleyici hazırlanıyor...

References (23)

  1. K. Sasikumar and S. Nagarajan, “Comprehensive Review and Analysis of Cryptography Techniques in Cloud Computing,” IEEE Access, vol. 12, pp. 52325–52351, 2024, doi: 10.1109/ACCESS.2024.3385449.
  2. E. Rescorla, “The Transport Layer Security (TLS) Protocol Version 1.3,” Aug. 2018, doi: 10.17487/RFC8446.
  3. G. M. Raimondo and L. E. Locascio, “Module-Lattice-Based Key-Encapsulation Mechanism Standard,” Aug. 2024, doi: 10.6028/NIST.FIPS.203.
  4. E. Camacho-Ruiz, P. Navarro-Torrero, and A. C. Aldaya, “A Framework for designing High-Order Side-Channel Protected Hardware Implementations of ML-KEM,” IACR Transactions on Cryptographic Hardware and Embedded Systems, vol. 2026, no. 2, pp. 272–295, Apr. 2026, doi: 10.46586/TCHES.V2026.I2.272-295.
  5. B. Yan et al., “Factoring integers with sublinear resources on a superconducting quantum processor,” Dec. 2022, Accessed: Jul. 03, 2026. [Online]. Available: http://arxiv.org/abs/2212.12372
  6. E. Zeydan, Y. Turk, B. Aksoy, and S. B. Ozturk, “Recent Advances in Post-Quantum Cryptography for Networks: A Survey,” Proceedings of the 2022 7th International Conference on Mobile and Secure Services, MobiSecServ 2022, 2022, doi: 10.1109/MOBISECSERV50855.2022.9727214.
  7. D. Joseph et al., “Transitioning organizations to post-quantum cryptography,” Nature 2022 605:7909, vol. 605, no. 7909, pp. 237–243, May 2022, doi: 10.1038/s41586-022-04623-2.
  8. C. Rubio Garcia, A. Cano Aguilera, C. Stan, J. Jose Vegas Olmos, S. Rommel, and I. Tafur Monroy, “Enhanced Network Security Protocols for the Quantum Era: Combining Classical and Post-Quantum Cryptography, and Quantum Key Distribution,” IEEE Journal on Selected Areas in Communications, vol. 43, no. 8, pp. 2765–2781, 2025, doi: 10.1109/JSAC.2025.3568011.
  9. R. Xie et al., “Rosetta: Enabling Robust TLS Encrypted Traffic Classification in Diverse Network Environments with TCP-Aware Traffic Augmentation,” Proceedings of ACM Turing Award Celebration Conference, CHINA 2023, vol. 1, pp. 131–132, Jul. 2023, doi: 10.1145/3603165.3607437.
  10. “PQC-OAV-dataset-2025 | Datasets | Research | Canadian Institute for Cybersecurity | UNB.” Accessed: Jun. 20, 2026. [Online]. Available: https://www.unb.ca/cic/datasets/pqc-oav-dataset-2025.html
  11. H. M. Mujlid and R. Alshahrani, “Quantum-driven security evolution in IoT: AI-powered cryptography and anomaly detection,” Journal of Supercomputing, vol. 81, no. 9, pp. 1087-, Jun. 2025, doi: 10.1007/S11227-025-07582-3/TABLES/2.
  12. K. S. Adewole, A. Jacobsson, and P. Davidsson, “Intrusion Detection Framework for Internet of Things with Rule Induction for Model Explanation,” Sensors 2025, Vol. 25, Page 1845, vol. 25, no. 6, p. 1845, Mar. 2025, doi: 10.3390/S25061845.
  13. M. A. Talukder et al., “Machine learning-based network intrusion detection for big and imbalanced data using oversampling, stacking feature embedding and feature extraction,” J. Big Data, vol. 11, no. 1, pp. 33-, Dec. 2024, doi: 10.1186/S40537-024-00886-W/TABLES/16.
  14. S. More et al., “Enhanced Intrusion Detection Systems Performance with UNSW-NB15 Data Analysis,” Algorithms 2024, Vol. 17, Page 64, vol. 17, no. 2, p. 64, Feb. 2024, doi: 10.3390/A17020064.
  15. R. Jablaoui, O. Cheikhrouhou, M. Hamdi, and N. Liouane, “Deep learning enabled intrusion detection system for IoT security,” EURASIP J. Wirel. Commun. Netw., vol. 2025, no. 1, pp. 66-, Dec. 2025, doi: 10.1186/S13638-025-02477-6/TABLES/8.
  16. S. Fischer and M. H. Alsulami, “Residual Dense Optimization-Based Multi-Attention Transformer to Detect Network Intrusion against Cyber Attacks,” Applied Sciences 2024, Vol. 14, Page 7763, vol. 14, no. 17, p. 7763, Sep. 2024, doi: 10.3390/APP14177763.
  17. “PQC-OAV-dataset-2025 | Datasets | Research | Canadian Institute for Cybersecurity | UNB.” Accessed: Jul. 03, 2026. [Online]. Available: https://www.unb.ca/cic/datasets/pqc-oav-dataset-2025.html
  18. J. Singh Kushwah, A. Kumar, S. Patel, R. Soni, A. Gawande, and S. Gupta, “Comparative study of regressor and classifier with decision tree using modern tools,” Mater. Today Proc., vol. 56, pp. 3571–3576, Jan. 2022, doi: 10.1016/J.MATPR.2021.11.635.
  19. R. K. Halder, M. N. Uddin, M. A. Uddin, S. Aryal, and A. Khraisat, “Enhancing K-nearest neighbor algorithm: a comprehensive review and performance analysis of modifications,” Journal of Big Data 2024 11:1, vol. 11, no. 1, pp. 113-, Aug. 2024, doi: 10.1186/S40537-024-00973-Y.
  20. M. Bansal, A. Goyal, and A. Choudhary, “A comparative analysis of K-Nearest Neighbor, Genetic, Support Vector Machine, Decision Tree, and Long Short Term Memory algorithms in machine learning,” Decision Analytics Journal, vol. 3, p. 100071, Jun. 2022, doi: 10.1016/J.DAJOUR.2022.100071.
  21. E. Açıkgözoğlu, “COMPARISON OF MACHINE LEARNING ALGORITHMS FOR DETECTION OF DATA EXFILTRATION OVER DNS,” Yalvaç Akademi Dergisi, vol. 9, no. 2, pp. 61–70, Oct. 2024, doi: 10.57120/YALVAC.1507402.
  22. Z. F. Faruq, T. Mantoro, M. A. Catur Bhakti, and Wandy, “Random Forest Classifier Evaluation in DDoS Detection System for Cyber Defence Preparation,” 2022 IEEE 8th International Conference on Computing, Engineering and Design, ICCED 2022, 2022, doi: 10.1109/ICCED56140.2022.10010341.
  23. M. Kumar, C. Kondaiah, A. R. Pais, and R. S. Rao, “Machine learning models for phishing detection from TLS traffic,” Cluster Comput., vol. 26, no. 5, pp. 3263–3277, Oct. 2023, doi: 10.1007/S10586-023-04042-6/TABLES/11.