Can gestures speak louder than words? The effect of gestural discourse markers on discourse expectations

Abstract

Much of the research on discourse marking has focused on written text, thereby not considering signals available in the multimodal domain. It is therefore an open question to what extent comprehenders rely on nonlexical discourse signals, such as gestural discourse markers. We conducted a multimodal continuation study: 48 participants were presented with videos of a speaker narrating short stories including a contrast, a list, or an exception relation. In one condition, the target discourse relation was accompanied by a gestural discourse marker; in another condition it was not. The sound was cut out at the second relational argument, and participants were asked to provide a likely continuation. The results showed that comprehenders can infer discourse meaning from gestures, but gestural signals are not as strong as lexical connectives have been found to be. These findings contribute to our understanding of how people can achieve successful comprehension in multimodal communication.