Joint Online Seminar - Methodological questions in multimodal research: What counts as a relevant gesture in the study of multimodal event expressions?

In this seminar I focus on methodological questions in multimodal research. The seminar explores how basic methodological decisions impact our analysis. As a case study we will consider the representation of caused motion events in speech and gesture, drawing on naturalistic language data, including from corpora compiled during endangered languages documentation projects.Gestures commonly temporally align with semantically co-expressive speech. However, when we consider not single concepts but whole events investigating this gets more complicated, since events are not mapped onto single lexical items but across larger stretches of spoken discourse. To research gesture-speech integration in event expressions we must decide which gestures are considered ‘relevant’ and to be included in our analysis. I explore the impact of such methodological choices on study results by applying different definitions of gesture inclusion and by comparing the resulting datasets. We find both language- and definition-specific differences but also comparable trends which are stable across different definitions. These results have a bearing on how confidently we can compare studies based on different methodologies and of different languages.