Integrated processing in multimodal argumentation

Abstract

The question addressed in this paper is simple. If the argumentative function of a multimodal narrative text requires the integration of the information from different modes, among which verbal ones, what model for the order of processing and the integration of information do we need to adopt? Using discourse analysis as a method, we will argue that such a model needs to meet a number of requirements. We will argue that at least partially we are still in the realm of linguistic connectionism. By that we mean that the processing of the verbal modes is cognitively connected with and is influenced by the processing of other modes, perhaps similar to what has been argued that is the case in the multimodal processing of speech and gesture. The integration of for example a verbal speech voice-over mode with a pictorial mode is not a post-perception, late, dominantly conscious inference process that operates on more or less finalized mental representations of the separate modes. It depends on perceptual paring, on stipulated cross-modal congruence, and on cross-modal interactions between initial perceptions and knowledge based inferences. We will show how a unitary source assumption leads to cross-modal temporal paring as well as to the projection of the structure of one mode on the other mode. A model needs to allow such cross-modal connections to account for the fact that for example simultaneously appearing forms in two modes are interpreted as a crossmodal metaphor in which the structure receiving mode is taken to be the source domain.