Gesture and Multimodal Interaction
I began to study gesture in 1986, motivated by the obvious importance of hand gestures to many turns at talk and the almost complete absence of naturalistic research on conversational gesture. My first robust observation was that some gestures are distinguished from others by the fact that, at the beginning of making them, speakers routinely glance at their own gesturing hands. The gestures they orient to and make relevant by their gaze are depictions, gestures with which speakers depict an object, event or scene, remembered, imagined, or abstract. By directing gaze at their own hands (and often also indexing the gesture in their talk), speakers display the communicative salience of the gesture. – An implication of this finding is that it is not only impossible to study gesture apart from talk, but also from other embodied actions: studying gesture becomes the study of multimodal interaction.
1988 The significance of gestures: How it is established. Papers in Pragmatics, 2, 1/2, 60-84
1993 Gesture as communication I: Its coordination with gaze and speech. Communication Monographs, 60 (December 1993), 275-299.
1994 Gesture as communication II: The audience as co-author. Research on Language and Social Interaction. 27 (3), (Special Issue: Is Gesture Communicative? Ed. by A.Kendon), 239-267
2002 Grammars, words, and embodied meanings. On the evolution and uses of so and like.
Journal of Communication, 52, 3, September 2002, 581-596
2008 Laborious intersubjectivity. Attentional struggle and embodied communication in an auto-shop. Embodied Communication in Humans and Machines. Ed. I.Wachsmuth, M. Lenzen & G. Knoblich. Oxford University Press, 202-228.
What communicative import hand gestures have in conversation has everything to do with when in relation to the talk—where in the turn and sequence—they are made. This placement is quite systematic, and a particularly frequent and interesting sequencing of gesture and talk puts the gesture before the bit of talk to which it relates—the gesture projects something about the next bit of talk: it depicts the object that is about to be named, previews the speech act that is about to be performed, or how many turn-units there will be. Gestures that are placed in this fashion project the next moment of talk and interaction.
1992 J. Streeck & U. Hartge. Previews: Gestures at the Transition Place. The Contextualization of Language. Eds. P. Auer & A. di Luzio. Amsterdam Benjamins B.V., 135-158
1995 On Projection. Interaction and Social Intelligence. Ed. E. Goody. Cambridge: Cambridge University Press, 84-110
2009 Forward-gesturing. Discourse Processes, 46, 2-3, March-June 2009, 161-179
When we observe people interacting in complex, activity-rich settings such as workplaces, we can often see how their hand gestures incorporate objects and develop, almost as a by-product, from the physical actions that they engage in: gestures make these actions and the setting in which they are carried out transparent and intelligible. Hand-gestures serve a much greater range of functions than can be observed in self-contained conversational or experimental settings. In Gesturecraft I have proposed an ecological perspective on gesture to differentiate between the various ways in which gesture supports social life and shared understanding. These ecologies of gesture are to be conceived as part of the fabric of multimodal interaction.
1996 How to do things with things: Objets trouvés and symbolization. Human Studies, 19, 365-384.
2009 Gesturecraft. The Manu-facture of Meaning. Amsterdam: John Benjamins. (Paperback edition 2011).
2010 Ecologies of gesture: Action and interaction. J Streeck (ed.). New Adventures in Language and Interaction. Amsterdam: John Benjamins. 221 - 240
2011 The changing meanings of things: Found objects and inscriptions in social interaction. J. Streeck, C. Goodwin & C. LeBaron (eds.). Embodied Interaction. Language and Body in the Material World. Cambridge: Cambridge University Press. 67-78
Gestures have not only been studied by scholars, but also by artists; the visual arts have a much longer history of systematic study of gesture and body motion and how it communicates meaning in social interaction than the sciences do. Artists since antiquity have treated hand gestures as both topic and resource, and how the problem of painters and sculptors—how to make a still image appear to move—compares to the problem of the gesturer—how to depict unmoving objects and abstract realities by movement.
2008 Depicting by gestures. Gesture 8:3, 285–301
2009 Depicting gestures. The representation of body motion in the visual arts of the West.
Gesture, 9, 1, 1-34
2003 A moment and its gestures. Review article on Leo Steinberg’s Leonardo’s Last Supper. Gesture, 3, 2, 2003, 213-236.
Studying gesture and embodiment in complex settings of physical and cognitive action has convinced me that the human body is insufficently conceptualized and under-theorized even among those who study interaction in natural settings. I believe that it is inadequate to treat the body solely as an instrument of the communicative will, and that we need to understand how the body’s specific, schematized sensory and enactive experiences and motor-memories contribute structure and meaning to embodied expressions such as getures of the hand. Such a non-dualist conception of the human body has been proposed, with some variations, by phenomenologists, Philosophical Anthropologists, and contemporary anthropologists who draw inspiration from them. In recent publications, I have advocated a broader notion of embodiment, which not only encompasses the expressive dimension of embodied action, but also its cognitive, intercorporeal, and place-specific dimensions, and is responsive to the new, ‘phenomenological’ conception of embodiment that is articulated by many neuroscientists.
2002 A body and its gestures. Gesture, 2, 1, 2002, 19-44.
2003 The body taken for granted: Lingering dualism in research on social interaction. Studies in Language and Social Interaction: In Honor of Robert Hopper. Eds. P. Glenn, C.D. LeBaron & J. Mandelbaum. Mahwah, NJ: Lawrence Erlbaum, 427-440.
to appear Interaction and the living body Journal of Pragmatics. Special issue on multimodal interaction. Ed.by A. Deppermann.