Multi-Modal Gesture
Charles Goodwin
Applied Linguistics UCLA
cgoodwin@humnet.ucla.edu
http://www.sscnet.ucla.edu/clic/cgoodwin.htm

A primordial site for the study of gesture consists of a situation in which multiple participants are carrying our courses of action together while attending to 1) each other; 2) the detailed organization of the talk in progress; 3) relevant phenomena in the environment and 4) the larger activities that they are engaged in. Within such a framework gesture does not stand alone as a self-contained system that can be analyzed in isolation from the other semiotic resources and meaning practices that participants are using to build action in concert with each other. Instead gesture occupies an interstitial position within a larger ecology of sign systems that build meaning and action by mutually elaborating each other. This is manifest in the organization of gesture in a variety of different ways. For example talk can be incomplete without being problematic by virtue of the way in which part of what is being said and treated as relevant within the utterance is accomplished through gesture. Such mutual interdependence between gestures and other co-occurring sign systems is relevant in other ways as well. For example the placement of gesture in relationship to other sign systems, such as the participation frameworks constituted through the visible orientation of the actors’ bodies, provides methods and practices for constituting the communicative or non-communicative status of a speaker’s hand movements.

In short the prototypical place where gesture emerges in the world is in the midst of a dense and consequential set of other semiotic practices being used by participants to build relevant action in concert with each other. This environment is not simply a haphazard collection of other semiotic resources that happen to be present when a gesture occurs, but instead a genuine ecology of sign systems that shape each other through their mutual interaction. Like other ecologies the organization of these sign systems in relationship to each other can dynamically change to maintain function and stability with confronted with a disturbance in both moment to moment interaction and on much larger time scales. Moreover, the way in which gesture is lodged within such an ecology demonstrates why it has a structure that is not only different from, but complementary to that of other sign systems such as language.

In the longer presentation at the Austin 2002 conference Gesture: the Living Medium these arguments about the structure and organization of gesture were demonstrated through analysis drawn from videotapes recorded in a number of different settings including 1) an archeological field excavation where gestures linking phenomena in the dirt being excavated to relevant talk were crucial to the constitution of the professional vision of new archaeologists, that is their mastery of how to see as an archaeologist as a form of public practice; 2) interaction between young girls playing hopscotch where contingencies in their interaction led to rapid changes in the structure of both gesture and the other sign systems it was linked to; 3) conversations in home of a man with severe aphasia where gesture was crucial but lodged within an organization of sign systems quite different that of fully fluent speakers (for example someone other than the gesturer produced the talk that elaborated the gesture and publicly established its locally relevant meaning). 4) lawyers gesturing at a videotape while making arguments to a jury about the guilt and innocence of the policemen seen on the tape beating a motorist, Rodney King; and 5) gestures within family interaction and interaction between friends talking on the street. Because of the present space limitations only a single, quite simple strip of data will be examined here.
The following was recorded in an American home. Talk is transcribed using the system created by Gail Jefferson (Sacks, Schegloff, & Jefferson, 1974: 731-733).

It is impossible to grasp just what the speaker is telling his recipient from the talk alone. Clearly a major reason for this is the use in the talk of deictic terms (this and that) that instruct the hearer to attend to phenomena beyond the stream of speech. Indeed each of these terms indexes a gesture. Characteristically gesture is analyzed by linking what a hand is doing to the structure of the talk in progress. Here however that is inadequate. When the gesturing hands alone are taken into account what exactly is being talked about is still not visible:

Figure 1 Gesturing Hands

To grasp what the speaker is saying and demonstrating a hearer must take into account an object being held by the speaker and being presented and demonstrated through the gesture. The object here is a pitcher for a blender that the speaker has ordered over the Internet. The speaker is telling his addressees that while the pitcher was shipped he did not receive either the top for the pitcher, or it’s screw-in base. While this is not made visible through gesture and its accompanying talk alone, it becomes vividly clear when a larger multi-modal sign complex that encompasses not only talk and gesture, but also objects in the world is taken into account (Streeck, 1996).

Figure 2 Gesture Linked to an Object in the World

As the speaker begins this utterance (more specifically over the word “sold”) his hands noticeably grasp the pitcher. He is not grasping the pitcher to hold it (it is already well supported by his other hand) but instead to prominently display the object to his addressees. One might think of this hand movement as a gestural practice for presenting or indicating something, that is as an action similar to a pointing gesture. However it is crucial to not restrict analytic focus to the gesturing hand, but to also take into account the object in the world being grasped. As is demonstrated a moment later this object forms a crucial part of the multi-modal signs that display the missing parts of the blender. The gesturing hands alone fail to make visible the absent base and lid (see Figure 1).

The co-occurring talk is equally crucial in that it formulates what is being done as describing something absent that can be inferred from the structure of the object being held. The general importance of the talk that elaborates a gesture is made particularly clear when the party producing the gesture can’t speak. Rather than being immediately, transparently clear, a gesture such as this unaccompanied by relevant talk can set off a long sequence devoted to figuring out what a speaker suffering from aphasia is trying to say to his interlocutors through the gesture (Goodwin, 1995, 2002a).

In short what one finds here is a small ecology in which different signs in different media (talk, the gesturing body and objects in the world) dynamically interact with each other. Each individual sign is partial and incomplete. However, as part of a larger complex of meaning making practices they mutually elaborate each other to create a whole, a clear statement, that is not only different from its individual parts, but greater than them in that no sign system in isolation makes clear what is being said.

Gestures coupled to phenomena in the environment are pervasive in many settings (archaeological field excavations, weather forecasts, pointing to overheads in academic talks, etc. -- consider how many computer screens are smeared with fingerprints). Gestures linked to the environment would thus seem to constitute a major class of gesture. However with a few notable exceptions (for example Goodwin, 2000, 2002b; Haviland, 1995; Haviland, 1998; Heath & Hindmarsh, 2000; Hutchins & Palen, 1997; LeBaron, 1998; LeBaron & Streeck, 2000; Nevile, 2001; Streeck, 1996) multi-modal sign complexes that encompass both gesture and phenomena in the world have been largely ignored by students of gesture. This neglect may result from the way in which such gestures slip beyond theoretical frameworks focused on either ties between gesture and psychological processes inside the mind of the individual speaker, or exclusively on the talk and bodies of participants in interaction. In essence an invisible analytic boundary is drawn at the skin of the participants. However, rather than being something that can be studied in isolation as a neat, self contained system, gesture is an intrinsically parasitic phenomenon, something that gets its meaning and organization from the way in which it is fluidly linked to the other meaning making practices and sign systems that are constituting the events of the moment. Human cognition and action are unique in the way in which they use as resources both the details of language, and physical and cultural environments that have been shaped by human action on an historical time scale. Typically these different kinds of phenomena are studied in isolation from each other by separate disciplines (for example linguistics and archaeology). However gesture’s interstitial position as something than links the details of language use to structure in the environment provides a key analytic point of entry for investigation of the rich interdigitiaton of different kinds of semiotic resources that human beings use to build relevant action in the consequential settings that define the lifeworld of a society.
========
This is a brief report from a larger analysis that was presented at the University of Texas at Austin conference Gesture: The Living Medium, June 5-8, 2002. The video data being analyzed here was collected by the UCLA Center on Everyday Lives of Families, a Center on Working Families funded by the Alfred P. Sloan Foundation. I thank Scott Phillabaum for the line drawings used here.

References Cited

Goodwin, C. (1995). Co-Constructing Meaning in Conversations with an Aphasic Man. Research on Language and Social Interaction, 28(3), 233-260.
Goodwin, C. (2000). Action and Embodiment Within Situated Human Interaction. Journal of Pragmatics, 32(1489-1522).
Goodwin, C. (2002a). Conversational Frameworks for the Accomplishment of Meaning in Aphasia. In C. Goodwin (Ed.), Situating Language Impairments Within Conversation (pp. 90+116). Oxford, New York: Oxford University Press.
Goodwin, C. (2002b). Pointing as Situated Practice. In S. Kita (Ed.), Pointing: Where Language, Culture and Cognition Meet (pp. 217-241). Hillsdale, NJ: Lawrence Erlbaum Associates.
Haviland, J. (1995). Mental Maps and Gesture Spaces: Land, Mind and Body in Two Contexts. Paper delived at the Conference on Gesture Compared Cross-Linguistically, Linguistic Institute, University of New Mexico, July 7-10, 1995.
Haviland, J. B. (1998). Early Pointing Gestures in Zincantán. Journal of Linguistic Anthropology, 8(2), 162-196.
Heath, C., & Hindmarsh, J. (2000). Configuring Action in Objects: From Mutual Space to Media Space. Mind, Culture and Activity, 7(1&2), 81-104.
Hutchins, E., & Palen, L. (1997). Constructing Meaning from Space, Gesture, and Speech. In L. Resnick & R. Säljö & C. Pontecorvo & B. Burge (Eds.), Discourse, Tools and Reasoning: Essays on Situated Cognition (pp. 23-40). Berlin, Heidelberg, New York: Springer-Verlag.
LeBaron, C. (1998). Building Communication: Architectural Gestures and the Embodiment of Ideas. Ph. D. dissertation, Department of Communition,The University of Texas at Austin.
LeBaron, C. D., & Streeck, J. (2000). Gestures, Knowledge, and the World. In D. McNeill (Ed.), Gestures in Action, Language, and Culture (pp. 118-138). Cambridge: Cambridge University Press.
Nevile, M. (2001). Beyond the Black Box: Talk-in-Interaction in the Airline Cockpit. Ph.D. dissertation, Department of Linguistics, Australian National University, Canberra.
Sacks, H., Schegloff, E. A., & Jefferson, G. (1974). A Simplest Systematics for the Organization of Turn-Taking for Conversation. Language, 50, 696-735.
Streeck, J. (1996). How to Do Things with Things. Human Studies, 19, 365-384.