SavedAdd your journal

Journals / Psikoloji Çalışmaları / 2024 / Cilt: 44 Sayı: 3

Multimodal Communication in Virtual and Face-to-Face Settings: Gesture Production and Speech Disfluency

Çevrimiçi ve Yüz Yüze İletişim Ortamlarında Multimodal Dil Kullanımı: Jest Üretimi ve Konuşma Akıcılığı Üzerine Bir Araştırma

Burcu Arslan(Koç University, Department of Psychology)0000-0002-2465-360X
Can Avcı(Koç University, Department of Psychology)0009-0008-9204-3150
Demet Özer(Kadir Has University, Department of Psychology)0000-0003-3230-2874
Pages
349–363
Publication Date
2024-12-16
🏛️Bibliographic metadata and full text access for this article are provided via DergiPark.
DergiPark Record ↗
✨

AI Summary & Key Takeaways

AI Insight
⚡ TL;DR(Too Long; Didn't Read)

Our results have showed that the participants’ overall disfluency rate was higher for the speech communicated via videoconferencing than the speech communicated face-to-face.

📌 3 Key Takeaways

🎯
Objective & Hypothesis:The COVID-19 pandemic has made online data collection a popular choice.
📊
Key Findings & Data:It is important to evaluate how comparable online studies are to face-to-face studies, particularly in multimodal language research where modes of communication significantly impact the results.
💡
Conclusion & Impact:Our results have showed that the participants’ overall disfluency rate was higher for the speech communicated via videoconferencing than the speech communicated face-to-face.
🔬Design:Araştırma Makalesi
📍Scope:Comprehensive Sample
⚙️Variables:Jest kullanımı · konuşmanın akıcılığının bozulması · çevrimiçi iletişim

Abstract

The COVID-19 pandemic has made online data collection a popular choice. It is important to evaluate how comparable online studies are to face-to-face studies, particularly in multimodal language research where modes of communication significantly impact the results. In this study, we examined individuals’ rates and patterns of speech disfluency and gesture use across face-to-face and online videoconferencing settings as they described their daily routines (N = 64). We asked whether and how multimodal language is affected across different communication settings and gesture use, particularly iconic gestures, is associated with speech fluency regardless of the context. Our results have showed that the participants’ overall disfluency rate was higher for the speech communicated via videoconferencing than the speech communicated face-to-face. However, the type of disfluencies changed across contexts, such that filled pauses and repairs were more common in online communication, whereas silent pauses were more common in face-to-face communication. These f indings signal an interplay between the cognitive functions of different disfluency types and communicative strategies. Results indicate that the overall gesture frequency and iconic gesture use were similar in both settings. Furthermore, the use of iconic gestures was found to negatively predict the overall disfluency rate, regardless of the setting. This finding suggests that using iconic gestures might facilitate cognitive processes, paving the way for a more fluent speech. This study demonstrates that multimodal language and communication strategies may vary across different communication settings and nuanced understanding of the differences in multimodal language between online and face-to-face communication can be gained using different contexts. The findings contribute to understanding the impact of increasingly widespread online communication on multimodal language production processes and provide foundation for future research.

Özet

Çevrimiçi veri toplama, COVID-19 salgını nedeniyle öne çıkan bir seçenek haline gelmiştir. Çevrimiçi çalışmaların, özellikle bağlamın çok önemli bir etkiye sahip olduğu dil ve iletişim alanlarında, yüz yüze yapılan çalışmalarla ne ölçüde karşılaştırılabileceğini anlamak çok önemlidir. Bu çalışma, yüz yüze ve video konferans ortamlarında multimodal iletişimi araştırmak amacıyla, kişilerin (N= 64) günlük rutinlerini anlatırken kullandıkları konuşma akıcılıklarına ve sözlü dile eşlik eden jest üretimlerine odaklanmaktadır. Çalışmada, el jestlerinin ve sözlü dildeki akıcılığın farklı iletişim ortamlarında (çevrim içi ve yüz yüze) nasıl değiştiği ve iletişim ortamından bağımsız olarak jest kullanımının (özellikle ikonik jestlerin) konuşma akıcılığıyla ilişkisi araştırılmaktadır. Çalışmanın sonuçları, konuşma akışındaki bozulma oranının video konferans yoluyla iletişim kuranlarda, yüz yüze iletişim kuranlara göre daha yüksek olduğunu göstermiştir. Fakat konuşma akıcılığındaki farklı bozulma türlerinin iki ortamda farklılık gösterdiği bulunmuştur. Konuşmacıların video konferans ortamında daha fazla dolgulu duraksama ve onarım kullanırken, yüz yüze iletişim ortamında daha fazla sessiz duraksama kullandığı bulunmuştur. Bu bulgular, konuşmanın akıcılığındaki bozulmaların iletişim ortamına göre değişebileceğini ve farklı iletişim stratejileri doğurabileceğini göstermektedir. Bunun yanında, genel jest kullanımının ve özel olarak ikonik (temsili) jest kullanımının iki ortam arasında fark göstermediği bulunmuştur. Ayrıca, iletişim ortamından bağımsız olarak, ikonik jest kullanım sıklığının konuşma akıcılığını artırdığı bulunmuştur. Bu bulgu, özellikle ikonik jest kullanımının bilişsel süreçleri kolaylaştırarak daha akıcı bir konuşmaya zemin hazırlayabileceğini göstermektedir. Bu çalışma, multimodal dil ve iletişim stratejilerinin farklı iletişim ortamlarında değişebildiğini göstermekte ve bu anlamda bağlamın araştırılmasının önemini vurgulamaktadır. Sonuçlar, günümüzde özellikle yaygınlaşan çevrimiçi iletişimin multimodal dil üretim süreçleri üzerindeki etkilerini anlamaya katkı sağlamaktadır ve ileride yapılacak çalışmalar için önemli bir temel sunmaktadır.

🔖

Cite This Article

Arslan, B., Avcı, C., Özer, D. (2024). Multimodal Communication in Virtual and Face-to-Face Settings: Gesture Production and Speech Disfluency. Psikoloji Çalışmaları, Cilt: 44 Sayı: 3, 349–363. https://doi.org/10.26650/SP2024-1443983

📄 Full Text PDF Viewer

📄

PDF görüntüleyici hazırlanıyor...

References (41)

  1. Alibali, M. W. (2005). Gesture in spatial cognition: Expressing, communicating, and thinking about spatial information. Spatial Cognition and Computation, 5(4), 307-331. https://doi.org/10.1207/ s15427633scc0504_2 google scholar
  2. Archibald, M. M., Ambagtsheer, R. C., Casey, M. G., & Lawless, M. (2019). Using zoom videoconferencing for qualitative data collection: Perceptions and experiences of researchers and participants. International Journal of Qualitative Methods, 18, 1-8. https://doi.org/10.1177/ 1609406919874596 google scholar
  3. Arslan, B., & Göksun, T. (2021). Ageing, working memory, and mental imagery: Understanding gestural communication in younger and older adults. Quarterly Journal of Experimental Psychology, 74(1), 29-44. https://doi.org/10.1177/1747021820944696 google scholar
  4. Arslan, B., & Göksun, T. (2022). Aging, gesture production, and disfluency in speech: A comparison of younger and older adults. Cognitive Science, 46(2), Article e13098. https://doi.org/10.1111/ cogs.13098 google scholar
  5. Arslan, B., Avcı, C., Yılmaztekin, A., & Göksun, T. (2024). Do bilingual adults gesture when they are disfluent?: Understanding gesture-speech interaction across first and second languages. Language, Cognition and Neuroscience, 39(5), 571-583. https://doi.org/10.1080/23273798. 2024.2345306 google scholar
  6. Avcı, C., Arslan, B., & Göksun, T. (2022). Gesture and speech disfluency in narrative context: Disfluency rates in spontaneous, restricted, and encouraged gesture conditions. In Culbertson J., Perfors A., Rabagliati H., & Ramenzoni V. (Eds.), Proceedings of the 44th annual conference of the cognitive science society (pp. 1912-1917). Cognitive Science Society. google scholar
  7. Bailenson, J. N. (2021). Nonverbal overload: A theoretical argument for the causes of Zoom fatigue. Technology, Mind, and Behavior, 2(1), 1-6. https://doi.org/10.1037/tmb0000030 google scholar
  8. Bock, K. (1996). Language production: Methods and methodologies. Psychonomic Bulletin & Review, 3, 395-421. https://doi.org/10.3758/BF03214545 google scholar
  9. Bortfeld, H., Leon, S. D., Bloom, J. E., Schober, M. F., & Brennan, S. E. (2001). Disfluency rates in conversation: Effects of age, relationship, topic, role, and gender. Language and Speech, 44(2), 123-147. https://doi.org/10.1177/00238309010440020101 google scholar
  10. Corley, M., & Stewart, O. W. (2008). Hesitation disfluencies in spontaneous speech: The meaning of um. Language and Linguistics Compass, 2(4), 589-602. https://doi.org/10.1111/j.1749-818X. 2008.00068.x google scholar
  11. Cserzo, D. (2021). Discourses andpractices of attention in video chat. Multimodal Communication, 10(2), 143-156. https://doi.org/10.1515/mc-2020-0010 google scholar
  12. Fraundorf, S. H., & Watson, D. G. (2014). Alice’s adventures in um-derland: Psycholinguistic sources of variation in disfluency production. Language, Cognition and Neuroscience, 29(9), 1083-1096. https://doi.org/10.1080/01690965.2013.832785 google scholar
  13. Geller, V., & Shaver, P. (1976). Cognitive consequences of self-awareness. Journal of Experimental Social Psychology, 12(1), 99-108. https://doi.org/10.1016/0022-1031(76)90089-5 google scholar
  14. Glassmeyer, D. M., & Dibbs, R. A. (2012). Researching from a distance: Using live web conferencing to mediate data collection. International Journal of Qualitative Methods, 11(3), 292-302. https: //doi.org/10.1177/160940691201100308 google scholar
  15. Graziano, M., & Gullberg, M. (2018). When speech stops, gesture stops: Evidence from developmental and crosslinguistic comparisons. Frontiers in Psychology, 9, Article e00879. https://doi.org/10.3389/fpsyg.2018.00879 google scholar
  16. Hassell, M. D., & Cotton, J. L. (2017). Some things are better left unseen: Toward more effective communication and team performance in video-mediated interactions. Computers in Human Behavior, 73, 200-208. https://doi.org/10.1016/j.chb.2017.03.039 google scholar
  17. Hyusein G., & Göksun, T. (2023). The creative interplay between hand gestures, convergent thinking, and mental imagery. PLOS ONE, 18(4), Article e0283859. https://doi.org/10.1371/journal.pone. 0283859 google scholar
  18. Kandemir, S., Özer, D., & Aktan-Erciyes, A. (2023). Multimodal language in child-directed versus adult-directed speech. Quarterly Journal of Experimental Psychology, 77(4), 716-728. https: //doi.org/10.1177/17470218231188832 google scholar
  19. Kısa, Y. D., Goldin-Meadow, S., & Casasanto, D. (2022). Do gestures really facilitate speech production?. Journal of Experimental Psychology: General, 151(6), 1252-1271. https://doi.org/ 10.1037/xge0001135 google scholar
  20. Kita, S., Alibali, M. W., & Chu, M. (2017). How do gestures influence thinking and speaking? The gesture-for-conceptualization hypothesis. Psychological Review, 124(3), 245-266. https: //doi.org/10.1037/rev0000059 google scholar
  21. Kita, S., & Davies, T. S. (2009). Competing conceptual representations trigger co-speech representational gestures. Language and Cognitive Processes, 24(5), 761-775. https://doi.org/ 10.1080/01690960802327971 google scholar
  22. Kita, S., & Özyürek, A. (2003). What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language, 48(1), 16-32. https://doi.org/10.1016/ S0749-596X(02)00505-3 google scholar
  23. Krauss, R., Chen, Y., & Gottesman, R. (2000). Lexical gestures and lexical access: A process model. In D. McNeill (Ed.), Language and gesture: Window into thought and action (pp. 261-283). Cambridge University Press. google scholar
  24. Lausberg, H., & Sloetjes, H. (2009). Coding gestural behavior with the NEUROGES-ELAN system. Behavior Research Methods, 41(3), 841-849. https://doi.org/10.3758/BRM.41.3.841 google scholar
  25. Liebling, B. A., & Shaver, P. (1973). Evaluation, self-awareness, and task performance. Journal of Experimental Social Psychology, 9(4), 297-306. https://doi.org/10.1016/0022- 1031(73)90067- X google scholar
  26. Maclay, H., & Osgood, C. E. (1959). Hesitation phenomena in spontaneous English speech. Word, 15(1), 19-44. https://doi.org/10.1080/00437956.1959.11659682 google scholar
  27. McNeill, D. (1992). Hand and mind: What gestures reveal about thought. University of Chicago Press. google scholar
  28. Melinger, A., & Kita, S. (2007). Conceptualisation load triggers gesture production. Language and Cognitive Processes, 22(4), 473-500. https://doi.org/10.1080/01690960600696916 google scholar
  29. Morsella, E., & Krauss, R. M. (2004). The role of gestures in spatial working memory and speech. The American Journal of Psychology, 411-424. https://doi.org/10.2307/4149008 google scholar
  30. Oviatt, S. (1995). Predicting spoken disfluencies during human-computer interaction. Computer Speech and Language, 9(1), 19-36. doi:10.1006/csla.1995.0002 google scholar
  31. Özder, L. E., Özer D., & Göksun, T. (2022). Gesture use in L1-Turkish and L2-English: Evidence from emotional narrative retellings. Quarterly Journal of Experimental Psychology, 76(8), 1797-1816. https://doi.org/10.1177/17470218221126685 google scholar
  32. Özer, D., Tansan, M., Özer, E. E., Malykhina, K., Chatterjee, A., & Göksun, T. (2017). The effects of gesture restriction on spatial language in young and elderly adults. In G. Gunzelmann, A. Howes, T. Tenbrink, & E. Davelaar (Eds.), Proceedings of the 38th Annual Conference of the Cognitive Science Society (pp. 1471-1476). Cognitive Science Society. google scholar
  33. Rauscher, F. H., Krauss, R. M., & Chen, Y. (1996). Gesture, speech, and lexical access: The role of lexical movements in speech production. Psychological Science, 7(4), 226-231. https://doi.org/ 10.1111/j.1467-9280.1996.tb00364.x google scholar
  34. Seyfeddinipur, M. & Kita, S. (2001, August 29-31). Gesture as an indicator of early error detection in self-monitoring of speech. ISCA Tutorial and Research Workshop (ITRW) on Disfluency in Spontaneous Speech, Edinburgh, Scotland, UK. google scholar
  35. Silverman, S. W., & Ratner, N. B. (1997). Syntactic complexity, fluency, and accuracy of sentence imitation in adolescents. Journal of Speech, Language, and Hearing Research, 40(1), 95-106. https://doi.org/10.1044/jslhr.4001.95 google scholar
  36. Smith, V. L., & Clark, H. H. (1993). On the course of answering questions. Journal of Memory and Language, 32(1), 25-38. https://doi.org/10.1006/jmla.1993.1002 google scholar
  37. Ter Bekke, M., Drijvers, L., & Holler, J. (2024). Hand gestures have predictive potential during conversation: An investigation of the timing of gestures in relation to speech. Cognitive Science, 48(1), Article e13407. https://doi.org/10.1111/cogs.13407 google scholar
  38. Torrentira, M. C., Jr. (2020). Online data collection as adaption in conducting quantitative and qualitative research during the COVID-19 pandemic. European Journal of Education Studies, 7(11), 78-87. http://dx.doi.org/10.46827/ejes.v7i11.3336 google scholar
  39. Ünal, E., Manhardt, F., & Özyürek, A. (2022). Speaking and gesturing guide event perception during message conceptualization: Evidence from eye movements. Cognition, 225, Article e105127. https://doi.org/10.1016/j.cognition.2022.105127 google scholar
  40. Xu, Q., & Behring, D. (2014). The richer, the Better? Effects of modality on intercultural virtual collaboration. International Journal of Communication, 8, 2733-2754. google scholar
  41. Zoom Video Communications Inc. (2016). Security guide. Zoom Video Communications Inc. Retrieved from https://d24cgw3uvb9a9h.cloudfront.net/static/81625/doc/ Zoom- Security-White-Paper.pdf google scholar