SavedAdd your journal

Journals / Sakarya University Journal of Computer and Information Sciences / 2026 / Volume: 9 Issue: 4

A Comparative Performance Evaluation of Classification Algorithms on Imbalanced Datasets

Necati Vardar(KTO KARATAY ÜNİVERSİTESİ)0000-0002-5017-9788
Mehmet Fatih Ören(KTO KARATAY UNIVERSITY)0009-0001-0683-0000
Pages
—
Publication Date
2026-09-30
🏛️Bibliographic metadata and full text access for this article are provided via DergiPark.
DergiPark Record ↗
✨

AI Summary & Key Takeaways

AI Insight
⚡ TL;DR(Too Long; Didn't Read)

This study contributes to the literature by demonstrating that algorithmic superiority is context-dependent and providing a data-driven framework for selecting classifiers based on structural characteristics such as dimensionality, categorical complexity, and sample size.

📌 3 Key Takeaways

🎯
Objective & Hypothesis:Class imbalance remains a critical challenge in supervised learning, often biasing classifiers toward majority classes.
📊
Key Findings & Data:Unlike previous studies relying on default parameters, this research employs a rigorous Grid Search strategy to optimize hyperparameters for each algorithm within a rigorous SMOTE-balanced stratified cross-validation pipeline to ensure robust evaluation.
💡
Conclusion & Impact:This study contributes to the literature by demonstrating that algorithmic superiority is context-dependent and providing a data-driven framework for selecting classifiers based on structural characteristics such as dimensionality, categorical complexity, and sample size.
🔬Design:Modeling & Simulation
📍Scope:Comprehensive Sample
⚙️Variables:Imbalanced data · Classification algorithms · Naive Bayes

Abstract

Class imbalance remains a critical challenge in supervised learning, often biasing classifiers toward majority classes. While resampling techniques like Synthetic Minority Oversampling Technique (SMOTE) are widely used, the combined effect of data balancing and hyperparameter optimization across diverse datasets is rarely systematically explored. This study presents a comprehensive comparative analysis of four classification algorithms—Naive Bayes (NB), K-Nearest Neighbors (K-NN), Artificial Neural Networks (ANN), and Random Forest (RF)—across ten benchmark datasets from the UCI Machine Learning Repository. Unlike previous studies relying on default parameters, this research employs a rigorous Grid Search strategy to optimize hyperparameters for each algorithm within a rigorous SMOTE-balanced stratified cross-validation pipeline to ensure robust evaluation. Performance was assessed using a wide range of metrics, including Accuracy, Precision, Recall, F1-score, and Area Under the Curve (AUC). Experimental results reveal that ANN achieved the highest robustness in high-dimensional and complex categorical datasets (e.g., Car Evaluation F1-score: 0.990), significantly outperforming traditional models. Conversely, RF demonstrated superior stability in datasets with high feature dimensionality (e.g., Arrhythmia F1-score: 0.600) and chemical interactions (e.g., QSAR Fish Toxicity F1-score: 0.839). While K-NN remained competitive in low-dimensional spaces, NB struggled with complex feature dependencies. This study contributes to the literature by demonstrating that algorithmic superiority is context-dependent and providing a data-driven framework for selecting classifiers based on structural characteristics such as dimensionality, categorical complexity, and sample size.

🔖

Cite This Article

Vardar, N., Ören, M. F. (2026). A Comparative Performance Evaluation of Classification Algorithms on Imbalanced Datasets. Sakarya University Journal of Computer and Information Sciences, Volume: 9 Issue: 4. https://doi.org/10.35377/saucis...1833058

📄 Full Text PDF Viewer

📄

PDF görüntüleyici hazırlanıyor...

References (37)

  1. A. Tharwat, “Classification assessment methods,” Appl. Comput. Informat., vol. 17, no. 1, pp. 168–192, 2021, doi: 10.1016/j.aci.2018.08.003. [Online]. Available: https://doi.org/10.1016/j.aci.2018.08.003
  2. T. Saito and M. Rehmsmeier, “The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets,” PLOS ONE, vol. 10, no. 3, Art. no. e0118432, 2015, doi: 10.1371/journal.pone.0118432. [Online]. Available: https://doi.org/10.1371/journal.pone.0118432
  3. N. V. Chawla, N. Japkowicz, and A. Kotcz, “Editorial: Special issue on learning from imbalanced data sets,” ACM SIGKDD Explor. Newslett., vol. 6, no. 1, pp. 1–6, 2004, doi: 10.1145/1007730.1007733. [Online]. Available: https://doi.org/10.1145/1007730.1007733
  4. J. M. Johnson and T. M. Khoshgoftaar, “Survey on deep learning with class imbalance,” J. Big Data, vol. 6, no. 1, Art. no. 27, 2019, doi: 10.1186/s40537-019-0192-5. [Online]. Available: https://doi.org/10.1186/s40537-019-0192-5
  5. W. Chen, K. Yang, Z. Yu, Y. Shi, and C. L. P. Chen, “A survey on imbalanced learning: Latest research, applications and future directions,” Artif. Intell. Rev., vol. 57, Art. no. 137, 2024, doi: 10.1007/s10462-024-10759-6. [Online]. Available: https://doi.org/10.1007/s10462-024-10759-6
  6. M. Salmi et al., “Handling imbalanced medical datasets: Review of a decade of research,” Artif. Intell. Rev., vol. 57, Art. no. 273, 2024, doi: 10.1007/s10462-024-10884-2. [Online]. Available: https://doi.org/10.1007/s10462-024-10884-2
  7. H. Kaur, H. S. Pannu, and A. K. Malhi, “A systematic review on imbalanced data challenges in machine learning: Applications and solutions,” ACM Comput. Surv., vol. 52, no. 4, pp. 1–36, 2019, doi: 10.1145/3343440. [Online]. Available: https://doi.org/10.1145/3343440
  8. N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “SMOTE: Synthetic minority over-sampling technique,” J. Artif. Intell. Res., vol. 16, pp. 321–357, 2002, doi: 10.1613/jair.953. [Online]. Available: https://doi.org/10.1613/jair.953
  9. A. Fernández, S. García, M. Galar, R. C. Prati, B. Krawczyk, and F. Herrera, Learning from Imbalanced Data Sets. Cham, Switzerland: Springer, 2018, doi: 10.1007/978-3-319-98074-4. [Online]. Available: https://doi.org/10.1007/978-3-319-98074-4
  10. D. Dua and C. Graff, “UCI machine learning repository,” School Inf. Comput. Sci., Univ. California, Irvine, CA, USA, 2019. [Online]. Available: https://archive.ics.uci.edu
  11. G. James, D. Witten, T. Hastie, and R. Tibshirani, An Introduction to Statistical Learning: With Applications in R. New York, NY, USA: Springer, 2013.
  12. H. Zhang, “The optimality of Naive Bayes,” in Proc. 17th Int. Florida Artif. Intell. Res. Soc. Conf. (FLAIRS), Menlo Park, CA, USA: AAAI Press, 2004, pp. 562–567.
  13. I. Rish, “An empirical study of the Naive Bayes classifier,” in Proc. IJCAI Workshop Empirical Methods Artif. Intell., vol. 3, no. 22, 2001, pp. 41–46.
  14. J. Han, M. Kamber, and J. Pei, Data Mining: Concepts and Techniques, 3rd ed. Waltham, MA, USA: Morgan Kaufmann, 2012.
  15. T. Cover and P. Hart, “Nearest neighbor pattern classification,” IEEE Trans. Inf. Theory, vol. IT-13, no. 1, pp. 21–27, Jan. 1967.
  16. S. Haykin, Neural Networks: A Comprehensive Foundation, 2nd ed. Englewood Cliffs, NJ, USA: Prentice-Hall, 1999.
  17. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning representations by back-propagating errors,” Nature, vol. 323, no. 6088, pp. 533–536, 1986.
  18. L. Breiman, “Random forests,” Mach. Learn., vol. 45, no. 1, pp. 5–32, 2001.
  19. C. Chen, A. Liaw, and L. Breiman, “Using random forest to learn imbalanced data,” Dept. Statist., Univ. California, Berkeley, CA, USA, Tech. Rep., 2004.
  20. A. A. Khan, O. Chaudhari, and R. Chandra, “A review of ensemble learning and data augmentation models for class imbalanced problems: Combination, implementation and evaluation,” Expert Syst. Appl., vol. 244, Art. no. 122778, 2024, doi: 10.1016/j.eswa.2023.122778. [Online]. Available: https://doi.org/10.1016/j.eswa.2023.122778
  21. G. E. A. P. A. Batista, R. C. Prati, and M. C. Monard, “A study of the behavior of several methods for balancing machine learning training data,” ACM SIGKDD Explor. Newslett., vol. 6, no. 1, pp. 20–29, 2004, doi: 10.1145/1007730.1007735. [Online]. Available: https://doi.org/10.1145/1007730.1007735
  22. H. Guvenir, B. Acar, H. Muderrisoglu, and R. Quinlan, “Arrhythmia,” UCI Machine Learning Repository, 1997, doi: 10.24432/C5BS32. [Online]. Available: https://doi.org/10.24432/C5BS32
  23. R. Fisher, “Iris,” UCI Machine Learning Repository, 1936, doi: 10.24432/C56C76. [Online]. Available: https://doi.org/10.24432/C56C76
  24. M. Bohanec, “Car evaluation,” UCI Machine Learning Repository, 1988, doi: 10.24432/C5JP48. [Online]. Available: https://doi.org/10.24432/C5JP48
  25. “Labor relations,” UCI Machine Learning Repository, 1988, doi: 10.24432/C5CP4Q. [Online]. Available: https://doi.org/10.24432/C5CP4Q
  26. “Mushroom,” UCI Machine Learning Repository, 1981, doi: 10.24432/C5959T. [Online]. Available: https://doi.org/10.24432/C5959T
  27. Ş. Gül and F. Rahim, “Period changer,” UCI Machine Learning Repository, 2021, doi: 10.24432/C5B31D. [Online]. Available: https://doi.org/10.24432/C5B31D
  28. “Rice (Cammeo and Osmancik),” UCI Machine Learning Repository, 2019, doi: 10.24432/C5MW4Z. [Online]. Available: https://doi.org/10.24432/C5MW4Z
  29. D. Ballabio, M. Cassotti, V. Consonni, and R. Todeschini, “QSAR fish toxicity,” UCI Machine Learning Repository, 2015, doi: 10.24432/C5JG7B. [Online]. Available: https://doi.org/10.24432/C5JG7B
  30. E. Fokoue and N. Gunduz, “Türkiye student evaluation,” UCI Machine Learning Repository, 2013, doi: 10.24432/C5S02S. [Online]. Available: https://doi.org/10.24432/C5S02S
  31. S. Aeberhard and M. Forina, “Wine,” UCI Machine Learning Repository, 1992, doi: 10.24432/C5PC7J. [Online]. Available: https://doi.org/10.24432/C5PC7J
  32. B. Bischl et al., “Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges,” WIREs Data Mining Knowl. Discovery, vol. 13, no. 2, Art. no. e1484, 2023, doi: 10.1002/widm.1484. [Online]. Available: https://doi.org/10.1002/widm.1484
  33. P. Thölke et al., “Class imbalance should not throw you off balance: Choosing the right classifiers and performance metrics for brain decoding with imbalanced data,” NeuroImage, vol. 277, Art. no. 120253, 2023, doi: 10.1016/j.neuroimage.2023.120253. [Online]. Available: https://doi.org/10.1016/j.neuroimage.2023.120253
  34. D. H. Wolpert, “The lack of a priori distinctions between learning algorithms,” Neural Comput., vol. 8, no. 7, pp. 1341–1390, 1996, doi: 10.1162/neco.1996.8.7.1341. [Online]. Available: https://doi.org/10.1162/neco.1996.8.7.1341
  35. V. Borisov, T. Leemann, K. Seßler, J. Haug, M. Pawelczyk, and G. Kasneci, “Deep neural networks and tabular data: A survey,” IEEE Trans. Neural Netw. Learn. Syst., vol. 35, no. 6, pp. 7499–7519, Jun. 2024, doi: 10.1109/TNNLS.2022.3229161. [Online]. Available: https://doi.org/10.1109/TNNLS.2022.3229161
  36. H. He and E. A. Garcia, “Learning from imbalanced data,” IEEE Trans. Knowl. Data Eng., vol. 21, no. 9, pp. 1263–1284, Sep. 2009, doi: 10.1109/TKDE.2008.239. [Online]. Available: https://doi.org/10.1109/TKDE.2008.239
  37. A. K. Azlim Khan and N. H. Ahamed Hassain Malim, “Comparative studies on resampling techniques in machine learning and deep learning models for drug-target interaction prediction,” Molecules, vol. 28, no. 4, Art. no. 1663, 2023, doi: 10.3390/molecules28041663. [Online]. Available: https://doi.org/10.3390/molecules28041663