Kress, G. (2009). Multimodality: A social semiotic approach to contemporary communication. New York, NY: Routledge.
Disclaimer: The highlights are from a book review rather than the book itself. Sounds like a worthwhile read. Getting the book soon…
Writing about multimodality requires defining “mode,” but this is notoriously difficult (cf. Forceville 2006, Elleström 2010). Kress advocates a dynamic view: “socially, what counts as mode is a matter for a community and its social representational needs. What a community decides to regard and use as mode is mode. … Formally, what counts as mode is a matter of what a social-semiotic theory of mode requires a mode to be and to do” (p. 87 (p. 1)
While this is a practical solution, it makes “mode” applicable to any dimension of mediated meaningmaking. (p. 1)
Kress’ concern with the freedom of the interpreter, and his (as such: correct) claim that the pictorial mode by and large enables more universal access than the verbal one leads to at least one untenable claim: “Saussure’s mistaken assumption that the relation of signifier and signified is an arbitrary one was, as is all theory, a product and realization of the social conditions of his time. … In Social Semiotics arbitrariness is replaced by motivation, in all instances of sign-making, for any kind of sign” (p. 65, p. 67). (p. 3)
Kress’ obsession with “freedom” (a symptom of having grown up in the sixties?) in my view also leads to an excessive inclination to link the intentions of a sign-maker to “relations of power” (p. 72). (p. 3)
With the support of Lakoff and Johnson’s TIME IS SPACE metaphor (Lakoff and Johnson 1980: chapter 4), and McNeill’s thorough, empirically supported work on gestures, it is for instance possible to answer Kress’ question “can we indicate past time or future time in the mode of gesture?” (p. 107) with a clear “yes!” since in most cultures the future is in front of us, and the past behind us” – and this is perfectly expressible via gestures. (p. 4)
Kress’ warning that the “sensory, affective and aesthetic dimension is too often ignored and treated as ancillary” (p. 78) in studies of communication is a pertinent one. Another good point: signal-makers have to select a mode, or a combination of modes, to bring across their message with the greatest chance of success, and this inevitably has consequences for the meaning that will or can be elicited. I also concur heartily with Kress stressing that “genre mediates between the social and the semiotic” (p. 116). (p. 5)