SavedAdd your journal

Journals / Dicle Üniversitesi Mühendislik Fakültesi Mühendislik Dergisi / 2026 / Cilt: 17 Sayı: 3

Comparative Analysis of Machine Learning and Deep Learning Models for Speech Emotion Recognition

Konuşma Sinyallerinden Duygu Tanıma İçin Makine Öğrenmesi ve Derin Öğrenme Modellerinin Karşılaştırmalı Analizi

Fatih Şahin(İstanbul Topkapı Üniversitesi)0000-0002-8036-3156
Pages
—
Publication Date
2026-10-17
🏛️Bibliographic metadata and full text access for this article are provided via DergiPark.
DergiPark Record ↗
✨

AI Summary & Key Takeaways

AI Insight
⚡ TL;DR(Too Long; Didn't Read)

The findings indicate that both the evaluation protocol and end-to-end cost measurement must be reported explicitly in SER studies.

📌 3 Key Takeaways

🎯
Objective & Hypothesis:Speech Emotion Recognition (SER) is a research topic of critical importance in areas such as human-computer interaction, health monitoring, and customer services.
📊
Key Findings & Data:In this study, four benchmark datasets (RAVDESS, CREMA-D, TESS, SAVEE) were merged into a corpus of 12,162 recordings from 121 speakers, and seven machine learning and deep learning models were evaluated comparatively.
💡
Conclusion & Impact:The findings indicate that both the evaluation protocol and end-to-end cost measurement must be reported explicitly in SER studies.
🔬Design:Modeling & Simulation
📍Scope:Comprehensive Sample
⚙️Variables:Konuşma duygu tanıma · derin öğrenme · makine öğrenmesi

Abstract

Speech Emotion Recognition (SER) is a research topic of critical importance in areas such as human-computer interaction, health monitoring, and customer services. In this study, four benchmark datasets (RAVDESS, CREMA-D, TESS, SAVEE) were merged into a corpus of 12,162 recordings from 121 speakers, and seven machine learning and deep learning models were evaluated comparatively. Evaluation was carried out under a speaker-independent protocol using speaker-wise stratified group cross-validation (5 folds × 3 repeats), complemented by a leave-one-dataset-out protocol in which each corpus served in turn as the test set. The principal finding is that the evaluation protocol matters more than the model architecture: random splitting that permits speaker overlap produces an optimistic bias of 14.61 percentage points on average across all seven models, which exceeds the 11.24-point gap between the best and worst model. Under the speaker-independent protocol the highest balanced accuracy was obtained by the MLP (52.39%), with no significant difference from CNN-BiLSTM-Attention; since the performance ranges of traditional machine learning and deep learning models overlap, no claim of superiority at the model-family level is supported. In the cross-corpus evaluation the best model reached only 29.3% balanced accuracy on the six-class problem, approaching the chance level (16.67%) for two of the four corpora. When computational cost is measured end to end, including feature extraction time, the ranking reverses: Random Forest becomes the slowest model (211.6 ms per recording) and the 1D-CNN the fastest (37.0 ms). The findings indicate that both the evaluation protocol and end-to-end cost measurement must be reported explicitly in SER studies.

Özet

Konuşma tabanlı duygu tanıma (Speech Emotion Recognition — SER), insan-bilgisayar etkileşimi, sağlık izleme ve müşteri hizmetleri gibi alanlarda kritik öneme sahip bir araştırma konusudur. Bu çalışmada dört referans veri seti (RAVDESS, CREMA-D, TESS, SAVEE) birleştirilerek 121 konuşmacıya ait 12.162 ses kaydı üzerinde yedi makine öğrenmesi ve derin öğrenme modeli karşılaştırmalı olarak değerlendirilmiştir. Değerlendirme, konuşmacıya göre gruplanmış katmanlı çapraz doğrulama (5 kat × 3 tekrar) ile konuşmacı-bağımsız olarak yürütülmüş; ayrıca her veri setinin sırayla test kümesi olarak kullanıldığı veri setleri arası protokol uygulanmıştır. Çalışmanın temel bulgusu, değerlendirme protokolünün model mimarisinden daha belirleyici olmasıdır: konuşmacı bağımlılığına izin veren rastgele bölme, yedi modelin tamamında ortalama 14,61 puanlık iyimser bir yanlılık üretmektedir ve bu, en iyi ile en kötü model arasındaki farktan (11,24 puan) büyüktür. Konuşmacı-bağımsız protokolde en yüksek dengeli doğruluğu MLP (%52,39) göstermiş, CNN-BiLSTM-Attention ile arasında anlamlı fark bulunmamıştır; geleneksel makine öğrenmesi ile derin öğrenme modellerinin başarım aralıkları örtüştüğünden model ailesi düzeyinde bir üstünlük iddiası desteklenmemektedir. Veri setleri arası değerlendirmede en iyi model altı sınıflı problemde %29,3 dengeli doğrulukta kalmış, iki veri seti için başarım şans düzeyine (%16,67) yaklaşmıştır. Hesaplama maliyeti özellik çıkarma süresi dâhil edilerek uçtan uca ölçüldüğünde sıralama tersine dönmekte, Random Forest en yavaş (211,6 ms/kayıt), 1D-CNN ise en hızlı (37,0 ms/kayıt) model olmaktadır. Bulgular, SER çalışmalarında protokol seçiminin ve uçtan uca maliyet ölçümünün açıkça raporlanması gerektiğini göstermektedir.

🔖

Cite This Article

Şahin, F. (2026). Comparative Analysis of Machine Learning and Deep Learning Models for Speech Emotion Recognition. Dicle Üniversitesi Mühendislik Fakültesi Mühendislik Dergisi, Cilt: 17 Sayı: 3. https://doi.org/10.24012/dumf.1966628

📄 Full Text PDF Viewer

📄

PDF görüntüleyici hazırlanıyor...

References (27)

  1. El Ayadi, M., Kamel, M. S., & Karray, F. (2011). Survey on speech emotion recognition: Features, classification schemes, and databases. Pattern Recognition, 44(3), 572-587.
  2. Akçay, M. B., & Oğuz, K. (2020). Speech emotion recognition: Emotional models, databases, features, preprocessing methods, supporting modalities, and classifiers. Speech Communication, 116, 56-76.
  3. Schuller, B. W. (2018). Speech emotion recognition: Two decades in a nutshell, benchmarks, and ongoing trends. Communications of the ACM, 61(5), 90-99.
  4. Zhao, J., Mao, X., & Chen, L. (2019). Speech emotion recognition using deep 1D & 2D CNN LSTM networks. Biomedical Signal Processing and Control, 47, 312-323.
  5. Lian, Z., Liu, B., & Tao, J. (2021). CTNet: Conversational transformer network for emotion recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29, 985-1000.
  6. Nwe, T. L., Foo, S. W., & De Silva, L. C. (2003). Speech emotion recognition using hidden Markov models. Speech Communication, 41(4), 603-623.
  7. Jiang, W., Wang, Z., Jin, J. S., Han, X., & Li, C. (2019). Speech emotion recognition with heterogeneous feature unification of deep neural network. Sensors, 19(12), 2730.
  8. Mao, Q., Dong, M., Huang, Z., & Zhan, Y. (2014). Learning salient features for speech emotion recognition using convolutional neural networks. IEEE Transactions on Multimedia, 16(8), 2203-2213.
  9. Badshah, A. M., Ahmad, J., Rahim, N., & Baik, S. W. (2017). Speech emotion recognition from spectrograms with deep convolutional neural network. In International Conference on Platform Technology and Service (pp. 1-5).
  10. Mirsamadi, S., Barsoum, E., & Zhang, C. (2017). Automatic speech emotion recognition using recurrent neural networks with local attention. In IEEE ICASSP (pp. 2227-2231).
  11. Luo, D., Zou, Y., & Huang, D. (2018). Investigation on joint representation learning for robust feature extraction in speech emotion recognition. In INTERSPEECH (pp. 152-156).
  12. Li, Y., Zhao, T., & Kawahara, T. (2019). Improved end-to-end speech emotion recognition using self attention mechanism and multitask learning. In INTERSPEECH (pp. 2803-2807).
  13. Livingstone, S. R., & Russo, F. A. (2018). The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS). PLoS ONE, 13(5), e0196391.
  14. Cao, H., Cooper, D. G., Keutmann, M. K., Gur, R. C., Nenkova, A., & Verma, R. (2014). CREMA-D: Crowd-sourced emotional multimodal actors dataset. IEEE Transactions on Affective Computing, 5(4), 377-390.
  15. Dupuis, K., & Pichora-Fuller, M. K. (2010). Toronto Emotional Speech Set (TESS). University of Toronto, Psychology Department.
  16. Haq, S., & Jackson, P. J. B. (2008). Speaker-dependent audio-visual emotion recognition. In International Conference on Auditory-Visual Speech Processing (pp. 53-58).
  17. Scherer, K. R. (2003). Vocal communication of emotion: A review of research paradigms. Speech Communication, 40(1-2), 227-256.
  18. Ahmed, M. R., Islam, S., Islam, A. K. M. M., & Shatabda, S. (2023). An ensemble 1D-CNN-LSTM-GRU model with data augmentation for speech emotion recognition. Expert Systems with Applications, 218, 119633.
  19. Ma, Z., Zheng, Z., Ye, J., Li, J., Gao, Z., Zhang, S., & Chen, X. (2024). emotion2vec: Self-supervised pre-training for speech emotion representation. In Findings of the Association for Computational Linguistics: ACL 2024 (pp. 15747-15760).
  20. Gao, Y., Wang, L., Liu, J., Dang, J., & Okada, S. (2024). Adversarial domain generalized transformer for cross-corpus speech emotion recognition. IEEE Transactions on Affective Computing, 15(2), 697-708.
  21. Baevski, A., Zhou, Y., Mohamed, A., & Auli, M. (2020). wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33, 12449-12460.
  22. Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., & Mohamed, A. (2021). HuBERT: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29, 3451-3460.
  23. Schuller, B., Vlasenko, B., Eyben, F., Wöllmer, M., Stuhlsatz, A., Wendemuth, A., & Rigoll, G. (2010). Cross-corpus acoustic emotion recognition: Variances and strategies. IEEE Transactions on Affective Computing, 1(2), 119-131.
  24. Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7, 1-30.
  25. Dietterich, T. G. (1998). Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation, 10(7), 1895-1923.
  26. Nadeau, C., & Bengio, Y. (2003). Inference for the generalization error. Machine Learning, 52(3), 239-281.
  27. Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65-70.