A Study on Coordination of
Speech, Gesture and Breathing Movements
Nobuhiro Furuyama
National Institute of Informatics (Japan)
Abstract
This paper describes an experiment in which the subjects were each instructed
to synchronize articulations of /ba/ and movements of the right-hand wrist at
increasing frequencies in two different modes of coordination. The means and
the standard deviations of relative phase among different movements were used
as indices of the degree of equilibrium of the coordination to discover what
control parameters contribute to the dynamics of the coordination characterized
by the relative phase as order parameters.
Introduction
People often gesture while articulating meaningfully related speech segments.
To explain such coordination between speech and gesture produced at different
parts of
the body, it is often either argued or merely assumed that the parameters determining
how articulation and gesticulation are coordinated are planned and specified
in the
command system usually associated with the central nervous system (CNS). Coordination
between speech and gesture can be observed not only within single individuals,
but also between individuals, however (Furuyama 2000). This simple observation
poses a fundamental question to the aforementioned explanation of speech-gesture
coordination because inter-personal coordination of speech and gesture is achieved
not by a system in which the sub-systems are entirely connected mechanically,
but by one in which the sub-systems are connected only functionally at best.
The present study attempts, from the viewpoint of a dynamical systems approach,
to test the hypothesis that coordination of speech and gesture, regardless of
whether it is done intra-personally or inter-personally, is based on the hierarchized
coordination of movements and/or actions among articulatory movements, limb
movements, and breathing movements rather than only on a set of motor commands
to specify every single degrees of freedom involved in the coordination.
The physiologist Nicolas Bernstein (1967) proposed the idea of coordination
to solve the “Degree-of-Freedom Problem” (Turvey 1990). This is a
problem with a system such as a human body of having too many degrees of freedom
to be each specified by the brain with a set of motor plans. As Bernstein proposes,
by coordinating different parts of our body, we can decrease the degrees of
freedom to a significantly smaller number. The coordinated structure consisting
of body parts, however, presents additional implications. Kelso conducted an
experiment in which the subject was asked to synchronize the oscillation of
her/his fingers at a frequency specified by a metronome pulse in either the
in-phase mode (two limbs are moving in synch such that they are in the same
position of a cycle simultaneously) or the anti-phase mode (two limbs are moving
out of synch such that they are in the opposite position of a cycle simultaneously)
(Figure 1). Kelso’s results show that a breakdown of the anti-phase mode,
but not the in-phase mode is found at higher frequencies of oscillation. The
breakdowns are sometimes so devastating such that the anti-phase mode undergoes
a “phase transition” to the in-phase mode. Similar results are obtained
in coordination between
Nobuhiro Furuyama
National Institute of Informatics
(JAPAN)
Hiroki Takase
Shinshu University
(JAPAN)
Koji Hayashi
The University of Tokyo
(JAPAN)
breathing movements where abdomen movements and chest movements constitute
one coordinated system, and in coordination between breathing movements and
limb movements (Takase et al. 1999). The finding of such a coordinated
structureof breathing movements and limb movements is extremely relevant to
our concern here with both intra- and inter-personal coordination of speech
and gesture. This is because the articulation of speech is a special kind of
breathing and the gesticulations are a special kind of limb and/or body movements
that assume a certain semiotic function. These studies suggest that the frequency
of oscillation conditions the way in which different parts of the body are coordinated,
and that different frequencies have different consequences for the in-phase
and anti-phase modes of coordination. Moreover, the finding that the coordinated
structure can and in fact does break down or shift entirely to the opposite
pattern suggests that coordinated limb movements (and breathing movements) can
and in fact do assume their own spontaneous and self-organizational regularity,
and that they sometimes even override the ‘intended motor plan’ made
and sent from the central nervous system that supposedly controls the entire
movement.
One might argue that if this kind of phenomenon is only found in intra-personal
coordination of movements of different body parts of a single individual, it
must still be under the complete control of the motor plan made by the central
nervous system. Crucial to the context of the present study, however, some of
the phase transition phenomena mentioned above are even replicated by inter-personal
coordination experiments in which two individuals sitting side by side are asked
to visually coordinate the oscillation of their lower legs in the sagittal plane
in either the in-phase mode or anti-phase mode (e.g., Schmidt et al. 1990).
Furthermore, similar phenomena are observed in inter-personal coordination between
breathing movements and limb movements (Takase et al., in press) If we consider
the already attested synchronous relationship between speech and gesture as
the coordinated oscillating movements of different body parts (Nobe 1996), a
dynamical systems approach to coordination of movements and actions opens up
wide possibilities for explaining inter-personal as well as intra-personal coordination
of speech and gesture without recourse to the system consisting entirely of
mechanically connected components, but by invoking the system consisting at
least in part of only functionally connected components.
Experiment
Purpose of the Study;
The present experiment attempts to examine the stability and variability among
articulatory movements, wrist movements and breathing movements at the ribcage
as
well as the abdomen. The experimental parameters manipulated were relative phase
mode and oscillatory frequency of the coordinated movements.
Method of Data Collection;
Subjects: Ten undergraduate/graduate students (5 male and 5 female) participated
in this experiment. They were all right-handed. None of them had any problems
with
breathing or motor control at any part of the body relevant to the present study.
Procedure: The subjects were each asked to synchronize the articulation
of /ba/ and extending/ bending wrist movements either in in-phase mode or anti-phase
mode of oscillation at the rate specified by cyclic sound pulses emitted from
a Macintosh computer. The rate of the sound pulses was set to increase in increments
of 0.2 Hz, covering eight frequencies in the range 0.6 Hz to 2.4 Hz, and there
were ten pulses for each frequency. In addition, the subjects were asked to
maintain constant amplitude and pitch of articulation while synchronizing with
their wrist movements, to make their limb movements as smooth as possible and
not accented, and to elongate the vowel of /ba/ as long as the corresponding
phase lasted. The subjects each performed five sets of two different tasks (i.e.,
two relative phase modes). Thus, they each performed ten trials in total. The
order of the trials was randomized within each set. There was a minimum interval
of two minutes between each trial.
Data Acquisition: The articulated speech sound was acquired with a condenser
microphone (SONY ECM-360) connected to an amplifier (SONY CFD-700). The wrist
movements were measured with an electro goniometer (Penny & Giles, Inc.)
attached to the right wrist of the subject. The chest and abdomen breathing
movements were measured with Respitrace (Ambulatory Monitoring, Inc.). All of
these data were recorded by a PC (EPSON) with a DAQ board (National Instruments,
AT-MIO-16). The entire experimental scene was videotaped throughout.
Data Analysis: The onset of articulation of /ba/ was manually
plotted on the waveform displayed in the analysis tool, which was designed and
made to be used in the present study with LabVIEW (National Instruments). The
onset of each articulation was defined to be a point from which the amplitude
increases abruptly, compared with the segment before, where no sound can be
heard. Since the target sound is a combination of the consonant /b/ and the
vowel /a/, the increment in amplitude occurs in two stages in many cases. In
such cases we used the first increment in amplitude as an index of the onset
of articulation, because that must be closer to the onset of articulation of
/ba/intended by the subject her/himself.
The time series data of wrist movements was smoothed by the triangular moving
average method (the average of five points; i.e., the point in question and
two
points before and after it). The time series øn of the relative phase
between wrist movement and articulated speech sound was computed by the following
formula:
1 n Wr n Wr
n Sp n Wr
extension peak of extension peak of on articulati of start of extension peak
of 2
+ • •
• •
-
-
• =
time time
time time
n þ • (1)
ba..., ba..., ba...
ba..., ba..., ba...
In-phase Mode Anti-phase Mode
Figure 9. In-phase mode and anti-phase mode of coordination of articulation
of /ba/ and bending and extending movements of the right-hand wrist.
where Wr is the peak of wrist movements (i.e., the turning point from the extending
movement to the bending movement when in-phase mode is intended and that from
the bending movement to the extending movement when anti-phase mode is intended)
and Sp is the onset of articulated speech sound.
The pattern of the coordination of movements was evaluated by computing the
means of phase difference between the wrist movements and the articulations
(mean ø)
in each oscillatory frequency. The stability of the coordination was evaluated
by computing standard deviation of the phase difference between the wrist movements
and the articulations (SD ø) in each oscillatory frequency. The first
data points after each change of metronome frequency were not included in the
data analyses so as to eliminate the instability introduced by each frequency
change.
Results;
The data of one of the ten subjects were entirely excluded from the quantitative
analyses below (i.e., (2)-(4)) because of a technical problem during the data
acquisition. Also excluded were all of the data points obtained at frequencies
beyond 2.4 Hz and all of the data points at which any of the cases (1a) through
(1c) below were observed.
(1a) The targeted articulation is missing. This was observed at least once with
four subjects when the oscillatory frequency was 1.6 Hz or above. Three of these
four
subjects failed to articulate almost every time the oscillatory frequency became
higher than a certain point.
(1b) The targeted articulation and the wrist movements are both missing. This
was observed twice within the same trial by one subject when the oscillatory
frequencies
were 2.0 Hz and 2.4 Hz.
(1c) The wrist movements become gradually delayed with respect to the articulations
to the extent that the delayed cycle reaches approximately one cycle or above.
This was observed with only one subject in four out of the five trials of anti-phase
mode. Having excluded all these data points from the entire data set, we conducted
statistical analyses. The results of the analyses show the following:
(2) ANOVA (relative phase mode x oscillatory frequency) conducted on mean ø
of articulations and the wrist movements shows no significant main effect of
either
phase mode or oscillatory frequency.
(3) ANOVA (relative phase mode x oscillatory frequency) conducted on SD ø
of speech and the wrist movements shows no significant main effect of phase
mode
(F(1,8)=0.99, ns), but a significant main effect of frequency (F(8,64)=9.88,
p<.01). Further tests show that the higher the oscillatory frequency, the
significantly higher
the SD ø becomes.
(4) ANOVA (relative phase mode) conducted on the ribcage movements and the wrist
movements shows that SD ø is higher when the intended relative phase
is
anti-phase mode than when it is in-phase mode (F(1,7)=6.35, p<.05). (Data
of one of the subjects were not taken into consideration as there was a problem
with an
algorithm that was meant to capture the data points.)
Discussion and Conclusion
(1a) and (1b) show that the continuous articulation becomes difficult for some
subjects as the oscillatory frequency increases. (1c), though it was only observed
with
a single subject, may possibly be what is a so-called prolonged phase transition
and it calls for further examination. (2) may have resulted from the fact that
in the present study the control of population was minimum and it calls for
further study with more population control. (3) is consistent with (1a) and
(1b). (4) shows that the relative phase between breathing movements and wrist
movements is less stable when the intended phase mode is anti-phase than when
it is in-phase. This implies that breathing movements are a crucial part of
the coordination between articulations and wrist movements/gesticulations.
One of the main results of the present study is that the relative phase between
breathing movements and wrist movements is less stable when the intended relative
phase mode between articulations and wrist movements is anti-phase than when
it is in-phase. This suggests that breathing movements are a crucial part of
the coordination between articulations and wrist movements. To fully understand
the mechanism underlying speech-gesture coordination, then, we should further
consider the coordination by taking into consideration breathing movements as
a sub-system of the coordination between articulation and gesticulation. The
other results call for further examination. Particularly interesting is the
case of prolonged phase transition that we observed with one of our subjects.
Acknowledgements
The present study is supported by the Grant-in-Aid for Scientific Research from
the Japan Society for the Promotion of Science (Contract number: 13224095).
References
Bernstein, N. (1967). The coordination and regulation of movements. London:
Pergamon.
Furuyama, N. (2000b). Gestural interaction between the instructor and the learner
in
origami instruction: In McNeill, D. (Ed.), Language and Gesture. Cambridge:
Cambridge University Press.
Kelso, J.A.S. (1984). Phase transitions and critical behavior in human bimanual
coordination.
American Journal of Physiology: Regulatory, Integrative and Comparative,
246, R1000-R1004.
Schmidt, R.C., Carello, C., & Turvey, M.T. 1990. Phase transitions and critical
fluctuations
in the visual coordination of rhythmic movements between people. Journal
of Experimental Psychology: Human Perception and Performance, 16, 227-247.
Takase, H., Furuyama, N., Mishima, H., & Haruki, Y. (in press). Interpersonal
coordination
between breathing and limb movements.
Takase, H., Mishima, H. & Haruki, Y. (1999) Coordination between breathing
and
limb movement, Poster presented at ICPA-X at Edinburgh University.
Turvey, M.T. (1990). Coordination. American Psychologist, 45, 938-953.