Joint Online Seminar - How does the brain integrate auditory and visual signals in communication?

Face-to-face communication is inherently multimodal: next to the auditory speech signal, listeners are continuously exposed to rich visual signals, such as hand gestures. Despite the abundance of such visual signals in language, most models and theories on the neurobiology of language are based are based on characteristics of (clear) speech and text, and they rarely consider multimodal signals. In this talk, I will present a line of research that shows how such visual signals are integrated with speech, both within and between participants. I will present neuroscientific evidence that gestures are incorporated immediately by the brain, and shape how meaning is constructed from speech in both clear and adverse listening conditions. Finally, I will argue that we need a multimodal view on the neurobiology of language, that takes into account these audiovisual settings, but also naturalistic listening conditions (such as noisy, multi-person settings).