SavedAdd your journal

Journals / Sakarya University Journal of Computer and Information Sciences / 2022 / Cilt: 5 Sayı: 2

An Approach for Audio-Visual Content Understanding of Video using Multimodal Deep Learning Methodology

Emre Beray BOZTEPE
Bedirhan KARAKAYA
Bahadir KARASULU
İsmet ÜNLÜ
Pages
181–207
Publication Date
2022-08-31
DOI
—
🏛️Bibliographic metadata and full text access for this article are provided via DergiPark.
DergiPark Record ↗
✨

AI Summary & Key Takeaways

AI Insight
⚡ TL;DR(Too Long; Didn't Read)

Discussion and conclusion based on scientific findings are included in our study.

📌 3 Key Takeaways

🎯
Objective & Hypothesis:This study contains an approach for recognizing the sound environment class from a video to understand the spoken content with its sentimental context via some sort of analysis that is achieved by the processing of audio-visual content using multimodal deep learning methodology.
📊
Key Findings & Data:This approach begins with cutting the parts of a given video which the most action happened by using deep learning and this cutted parts get concanarated as a new video clip.
💡
Conclusion & Impact:Discussion and conclusion based on scientific findings are included in our study.
🔬Design:Modeling & Simulation
📍Scope:Comprehensive Sample
⚙️Variables:Multimodal Deep Learning · Association Rule Mining · Named Entity Recognition

Abstract

This study contains an approach for recognizing the sound environment class from a video to understand the spoken content with its sentimental context via some sort of analysis that is achieved by the processing of audio-visual content using multimodal deep learning methodology. This approach begins with cutting the parts of a given video which the most action happened by using deep learning and this cutted parts get concanarated as a new video clip. With the help of a deep learning network model which was trained before for sound recognition, a sound prediction process takes place. The model was trained by using different sound clips of ten different categories to predict sound classes. These categories have been selected by where the action could have happened the most. Then, to strengthen the result of sound recognition if there is a speech in the new video, this speech has been taken. By using Natural Language Processing (NLP) and Named Entity Recognition (NER) this speech has been categorized according to if the word of a speech has connotation of any of the ten categories. Sentiment analysis and Apriori Algorithm from Association Rule Mining (ARM) processes are preceded by identifying the frequent categories in the concanarated video and helps us to define the relationship between the categories owned. According to the highest performance evaluation values from our experiments, the accuracy for sound environment recognition for a given video's processed scene is 70%, average Bilingual Evaluation Understudy (BLEU) score for speech to text with VOSK speech recognition toolkit's English language model is 90% on average and for Turkish language model is 81% on average. Discussion and conclusion based on scientific findings are included in our study.

🔖

Cite This Article

BOZTEPE, E. B., KARAKAYA, B., KARASULU, B., ÜNLÜ, İ. (2022). An Approach for Audio-Visual Content Understanding of Video using Multimodal Deep Learning Methodology. Sakarya University Journal of Computer and Information Sciences, Cilt: 5 Sayı: 2, 181–207. https://ajindex.com/articles/946632

📄 Full Text PDF Viewer

📄

PDF görüntüleyici hazırlanıyor...

References (9)

  1. B. Karakaya, E.B. Boztepe, and B. Karasulu, "Development of a Deep Learning Based Model for Recognizing the Environmental Sounds in Videos," in The SETSCI Conference Proceedings Book, vol. 5, no. 1, pp. 53-58, 2022. B. Karasulu, “Çoklu Ortam Sistemleri İçin Siber Güvenlik Kapsamında Derin Öğrenme Kullanarak Ses Sahne ve Olaylarının Tespiti,” Acta Infologica, vol. 3, no. 2, pp. 60-82, 2019.
  2. E. A. Kıvrak, B. Karasulu, C. Sözbir ve A. Türkay, “Ses Özniteliklerini Kullanan Ses Duygu Durum Sınıflandırma İçin Derin Öğrenme Tabanlı Bir Yazılımsal Araç,” Veri Bilim Dergisi, vol. 4, no. 3, pp.14-27, 2021.
  3. S. Albawi, T. A. Mohammed, and S. Al-Zawi, “Understanding of a Convolutional Neural Network,” in Proceedings of the International Conference on Engineering and Technology (ICET), Antalya, Turkey, pp. 1-6, 2018.
  4. Y. Zhao, X. Jin, and X. Hu, “Recurrent Convolutional Neural Network for Speech Processing,” in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5300-5304, 2017.
  5. J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Y. Ng, “Multimodal Deep Learning,” in Proceedings of the 28th International Conference on Machine Learning (ICML11), Bellevue, Washington, USA, pp. 689–696, 2011.
  6. S. Bird, E. Loper, and J. Baldridge, "Multidisciplinary Instruction with the Natural Language Toolkit," in Proceedings of the Third Workshop on Issues in Teaching Computational Linguistics, Columbus, Ohio, pp. 62–70, 2008. J. Joseph, and J. R. Jeba, "Information Extraction Using Tokenization And Clustering Methods," International Journal of Recent Technology and Engineering, vol. 8 no. 4, pp. 3680-3692, 2019.
  7. H. van Halteren, J. Zavrel, and W. Daelemans, “Improving Accuracy in NLP Through Combination of Machine Learning Systems,” Computational Linguistics. vol. 27, no. 2, pp. 199–229, 2001. A. Roy, “Recent Trends in Named Entity Recognition (NER),” arXiv preprint arXiv:2101.11420 [cs.CL], 2021.
  8. K. Shaukat, S. Zaheer, and I. Nawaz, “Association Rule Mining: An Application Perspective,” International Journal of Computer Science and Innovation, vol. 2015, no. 1, pp.29-38, 2015. VOSK Offline Speech Recognition Library Website, 2022, [Online]. Available: https://alphacephei.com/vosk/. [Accessed: 01-July-2022]
  9. Ö. Şahinaslan, H. Dalyan ve E. Şahinaslan, "Naive Bayes Sınıflandırıcısı Kullanılarak YouTube Verileri Üzerinden Çok Dilli Duygu Analizi," Bilişim Teknolojileri Dergisi, vol. 15, no. 2, pp. 221-229, 2022. M.C. Yılmaz ve Z. Orman, "L