Is speech-gesture production ballistic or interactive?
Nobuhiro Furuyama
National Institute of Informatics
(JAPAN)
David McNeill,
University of Chicago
(USA)
Mischa Park-Doob
University of Chicago
(USA)

Abstract

A “ballistic” model of speech and gesture argues that once the planning of a sentence and a gesture is completed in the conceptualizer, the rest of the production processes of speech and gesture are independent from each other. Observing Arrernte speakers who gesture at arm’s length, De Ruiter and Wilkins hypothesized that the preparation of their gestures should take longer than usual, and that the co-expressive speech should be delayed to the same degree as the gesture to maintain synchrony between the two if their production processes are interactive. The result was that the preparation phase of Arrernte gesture was longer than usual, but speech did not become delayed. They concluded the ballistic model was therefore correct. The present paper asks whether the same holds true for non-Arrernte speakers (e.g., English speakers) gesturing at arm’s length. Our preliminary analysis shows that the preparation phase does not necessarily take longer than usual and that in any case speech remains synchronized with the related gesture. We conclude that the production of speech and that of gesture are interactive throughout the entire process, and that De Ruiter and Wilkins’s findings likely resulted from factors other than the modularity of speechgesture production.

Introduction

This paper argues that speech production and production of the concurrent gesture are interactive throughout the entire process, regardless of where in the gesture space a gesture is produced, or how long the preparation phase of a gesture lasts. This is meant to be contrasted with the ballistic (or modular) model of speech-gesture production (e.g., Levelt, Richardson, and La Heij 1985; De Ruiter and Wilkins 1998) which argues that once the planning of a sentence and a gesture is completed in the conceptualizer speech production and gesture production are independent from each other.

Evidence in support of the ballistic model allegedly comes in part from a crosslinguistic (and cross-cultural) comparison of speech-gesture synchronization differences
between Arrernte speakers and Dutch speakers (De Ruiter and Wilkins 1998). Arrernte is a language spoken in central Australia. The key fact is that Arrernte speakers, unlike non-Arrernte speakers, perform gestures at arm’s length; such gestures, performed at the outer limit of the gesture space, require a longer preparation phase. The question De Ruiter and Wilkins ask is, does the co-expressive (‘affiliated’) speech become delayed to the same degree as the gesture to maintain the synchrony between gesture and speech? The finding was that the preparation phases of Arrernte gestures were longer than for Dutch gestures, but speech did not become delayed: gestures occurred after the co-expressive speech by an amount equal to the extra time needed for the gesture preparation. Thus, De Ruiter and Wilkins concluded, gesture and speech are ‘modular’ in that speech and gesture are on separate ballistic tracks; once launched they unfold independently of each other. This is schematically shown in (a) and (b) in Figure 1.

(a) Arrernte (large) gesture with long preparation phase
(b) non-Arrernte (small) gesture with short preparation phase

Figure 1. Relative timing between speech and the concurrent gesture (black = preparation phase, gray = stroke phase, or between the onset of stroke phase and the
offset of the entire gesture movement, slanted stripe = speech).

One may naturally wonder what the synchronization relationship of speech and gesture would be if non-Arrernte speakers—Dutch or English—could somehow be induced to make their gestures at arm’s length, just as Arrernte speakers do, albeit spontaneously. The present study examines this question, using English speakers as participants. A ballistic model of speech and gesture would predict that under these circumstances English speakers would exhibit behavior similar to that of Arrernte speakers: the preparation phase of English speakers’ gestures would take longer than normal, causing gesture strokes to become delayed with respect to the co-expressive speech. If this turns out to be the case, the ballistic model may prove to be correct. If not, as the present study argues, the result could show two things: one is that the production of speech and gesture are interactive throughout the duration of both processes, if despite an extension of the arms to the outer limit of the gesture space speech and gesture remain in synchrony. The other would be that the findings of De Ruiter and Wilkins’s experiment—that in Arrernte speech continues forward while the gesture becomes delayed by an amount equal to the extra time for gesture preparation—resulted from factors other than the modularity of speech-gesture production.

The Present Predictions

There are two possible scenarios that fit the interactive theory. One is that the preparation phases of large gestures by speakers of English do not become delayed (i.e., they are comparable to Dutch gestures), and that the relative timing between the onset of speech and that of the stroke phase is intact for large gestures.

The other possibility that also fits the interactive theory is that the preparation phases of large gestures by English speakers become longer than those of small (Dutch) gestures, yet the relative timing between the onset of speech and the stroke phase remains intact. These predictions are schematically shown in (c) and (d) of Figure 2, respectively. These diagrams are to be contrasted with the prediction that De Ruiter and Wilkins would make, as shown in (a) and (b) of Figure 1 above.

Method

To induce English speakers to produce gestures at the outer-limit of the gesture space —gestures that might induce longer preparation phases—we modified our standard procedure of eliciting gestures via a cartoon narration.

Eight native speakers of American English participated in the study. Pair 1 consisted of a female narrator and male listener, all four subjects in Pairs 2 and 4 were female, and both members of Pair 3 were male (the subjects volunteered in pairs—we conducted the experiment without regard to gender). They were all undergraduate students at the University of Chicago at the time of the experiment, and were recruited through the “study-pool” list, an email database of interested volunteers maintained by the Department of Psychology. The material used to elicit the narrative featured the well-known cartoon characters Sylvester and Tweety. The cartoon is entitled “Canary Row” (Warner Brothers, Inc.). A detailed description of the story and some aspects of the animated film can be found in the Appendices of McNeill (1992). We asked our subjects to recount the cartoon story and, as they spoke, to point at relevant photos (still image clips printed in color on standard US letter-sized paper, 23 in total) that were taken from each of the episodes of the cartoon and posted in a random arrangement on the walls of the experiment room. The participants were tested in pairs. The narrator watched the entire cartoon once in a separate room. The narrator then joined the listener in the experiment room and was given approximately one minute to familiarize herself/himself with the locations of the image clips. Then the narrator was instructed to recount the entire cartoon
story, and to point at the image clips posted on the walls of the experiment room whenever the narration came to a point where the image clips were relevant. The listener was instructed to attend carefully and attempt to remember the events recounted by the narrator. The cameras were then switched on and the pair was left alone in the experiment room until the task was completed. The narrator was videotaped such that her/his whole body would fit on the screen even if s/he produced a gesture at the outer-limit of the gesture space. The listener was excluded from the camera field in favor of having a wider space available for the narrator.

(c) First prediction from the interactive theory for large gestures
(d) Second prediction from the interactive theory for large gestures
Figure 2. The present predictions on speechgesture timing when a gesture is performed at the outer-limit of the gesture space.

Results

The gesture space can be segmented as shown in Figure 3. The present analysis is limited to pointing gestures for which the preparation started in the center gesture space and terminated at the extreme periphery. This coding restriction ensured that we could see the effects of large movements on preparation phase duration and speech-gesture timing. Analysis shows that in most cases speech either remains synchronized with the meaningfully related gesture or even waits until the pointing gesture is fully executed in the outerlimit of the gesture space.

Figure 4 shows the histogram of the duration of preparation phase when English speakers were induced to gesture at the outer-limit of the gesture space.
(1) The mean duration of the preparation phase of the English speakers is 631.11 msec.
(SD = 34.27 msec., the median is 566.67 msec.) This is more comparable to that of the Dutch speakers (Mean = 559 msec., SD unknown) than that of the Arrernte speakers (803 msec, SD unknown).
(2) The histogram also shows that the distribution of duration of preparation phase is not entirely normal. There are some cases of abnormally long preparation phases towards the right edge of the chart. Durations in the range 301-600 msec. have higher frequencies than other durations. These deviations from an otherwise relatively normal distribution imply that there may be several interacting factors involved. Crucially for the present purpose, however, performing the stroke phase of a gesture
at the outer-limit of the gesture space does not necessarily make the duration of preparation phases longer. The longest preparation phase in the data set was 2166.67 msec (=65 frames).
Figure 3. The gesture space (Pedelty 1987, cited in McNeill 1992).
When a preparation phase lasts as long as this, it typically is accompanied by a so-called pretroke hold between the onset of the preparation phase and the onset of the stroke phase. Our longest preparation phase is a case in point: there was a long pre-stroke hold. The shortest preparation phase for each subject was equal to or below 366.67 msec. (= 11 frames). The shortest preparation phase in the entire data set lasted 333.33 msec. (=10 frames).
As to the relative timing between the stroke phase and the meaningfully related speech segments, they remain robustly in synchrony for the English speakers regardless of the length of the preparation phase. This is clearly shown in examples (1) through (4) below:
Example (1): A gesture with one of the shortest prep. phases1
[there’s a picture / <h>of wha^t it looks like #]
Example (2): Another gesture with one of the shortest prep. phases
[Sylvester / on* / the picture’s on the door / goes u]p /
Example (3): A gesture with one of the mid-length prep. phases
and then<n> / the camera sorta [pulls back n’ we see that / driving the trolley #]
Example (4): A gesture with the longest prep. phase
[a<aa>nd / Tweety’s ¿cage? // ¿right? is sitting on the ¿window sill? /]
1The following symbols are used in the transcription: [ = onset of preparation phase; ] = offset of retraction; ^ =
super imposed beat; / = unfilled speech pause; Bold face = stroke phase & post stroke hold; _ = prestroke hold;
# = breath pause; < ... > = filled speech pause; * = aborted speech or “speech trouble”; ¿...? = rising intonational
contour.
366.67msec (11 frames)
333.33msec (10 frames)
566.67msec(17 frames)
2166.67msec (65 frames)
Figure 4. The histogram of the duration of preparation phase in msec
when English speakers were induced to gesture at the outer-limit of the
gesture space.


Discussion

Speech-gesture synchrony by non-Arrernte speakers would imply that, for the Arrernte, the temporal separation of speech and gesture is not just a mechanical effect of a larger gesture space. That is, there is something extra causing the separation of speech and gesture. This extra something could be a rhetorical use of gesture. Suppose that, in the Arrernte culture, there is active control of gesture such that gestures are timed to be deployed after the co-expressive speech. This control might be ‘rhetorical’ in that gestures are made to occur as‘reinforcements’ or echoes of what is said in speech.

Even if this hypothetical rhetorical use is not present in the Arrernte, as long as there is some form of active control of gesture, gesture could follow speech, as it were, by design. The large gesture space then would be an effect of the gesture-speech timing difference rather than a cause of it. The extra long preparation phase would be the instrument of the speaker’s active control over the timing of the gesture. Using the preparation phase as the instrument of control would automatically make the amount of gesture delay equal the extra length of the preparation phase, as De Ruiter and Wilkins report.

But whatever the extra factor in Arrernte gesture performance is, something extra in the control of gesture means that gesture is not ballistic. And this means that Arrernte speech and gestures cannot be taken as evidence of modularity. On the contrary, they would reveal the very opposite of modularity—a continuing on-line process by Arrernte speakers of controlling the relationship between speech and gesture, in which the gesture is aimed to occur at the moment that the semantically co-expressive speech ends.

Conclusion

The evidence shows decisively that separation of speech and gesture in the Arrernte manner is not the result of using a larger gesture space with its attendant longer preparation phase. While a ballistic model of speech and gesture would predict gesture to be delayed according to length of preparation phase, our evidence has shown that speakers can maintain tight synchrony between speech and gesture even as preparation phase length varies widely. Such behavior is possible only if speakers exert careful control over the temporal relationship between the various parts of their simultaneously unfolding speech and gestures.

References

De Ruiter, J.P. and Wilkins, D. (1998). The synchronization of gesture and speech in Dutch and Arrernte (an Australian Aboriginal language). In S. Santi, I. Guaïtella, C. Cavé and G. Konopczynski (eds.), Oralité et Gestualité, pp. 603-607. Paris: L’Hamattan.
Kita, S. (2000). How representational gestures help speaking. In McNeill (2000).
Levelt, W.J., Richardson, G. and La Heij, W. (1985). Pointing and voicing in deictic expressions. Journal of Memory and Language, 24:133-164.
McNeill, D. (1992). Hand and Mind: What Gestures Reveal about Thought. Chicago: University of Chicago Press.
McNeill, D. (2000). Language and Gesture. (Ed.). Cambridge: Cambridge University Press.
McNeill, D. and Duncan, S., (2000). Growth points in thinking-for-speaking. In McNeill. (2000).
Nobe, S. (1996). Representational Gestures, Cognitive Rhythms, and Acoustic Aspects of Speech: A Network/Threshold Model of Gesture Production. Unpublished Ph.D. Dissertation,Department of Psychology, The University of Chicago.