Skip to main content

Lund University Publications

LUND UNIVERSITY LIBRARIES

Measuring the effects of pitch accent realization and the patterning of head and eyebrow gestures on perceived multimodal prominence

Ambrazaitis, Gilbert ; Frid, Johan LU orcid and House, David (2025) 10th Conference of the International Society for Gesture Studies
Abstract
Co-speech head and eyebrow gestures often align with pitch accents in spoken language production, thereby likely instantiating what we could refer to as multimodal prominence. Indeed, when speech is perceived audio-visually, controlled experimental studies have shown that both pitch accents and head and eyebrow gestures, when independently manipulated, have sizable effects on perceived prominence. However, studies on naturally produced, co-occurring speech and gesture suggest that the acoustic realization of pitch accents tends to correlate with the presence and magnitude of gestures. Thus, according to the Cumulative-Cue Hypothesis, when a higher prominence level is to be produced, we tend to increase the effects of acoustic and gestural... (More)
Co-speech head and eyebrow gestures often align with pitch accents in spoken language production, thereby likely instantiating what we could refer to as multimodal prominence. Indeed, when speech is perceived audio-visually, controlled experimental studies have shown that both pitch accents and head and eyebrow gestures, when independently manipulated, have sizable effects on perceived prominence. However, studies on naturally produced, co-occurring speech and gesture suggest that the acoustic realization of pitch accents tends to correlate with the presence and magnitude of gestures. Thus, according to the Cumulative-Cue Hypothesis, when a higher prominence level is to be produced, we tend to increase the effects of acoustic and gestural signals in conjunction, rather than independently of each other. This raises the question as to what extent the two modalities, speech and gesture, contribute to the overall audio-visual, or multimodal, prominence percept, in naturally produced multimodal speech. To approach this question, we study the co-occurrence of pitch accents and head and eyebrow gestures, and how these affect perceived prominence, in a data set of 60 brief audio-visually displayed Swedish television news readings (1936 words in total). In this data, accents and gestures are typically combined cumulatively (accent only, accent+head, accent+head+eyebrows). We manually annotated the occurrence of gestures, pitch accents, and tonal landmarks of the accents. From the tone labels, f0 values were extracted semi-automatically. The perceived prominence of all words in the dataset was measured using a web-based crowdsourcing approach, where adult volunteers rated each word in a semi-random selection of clips using a 3-point scale (manifested in color-changing word buttons). In our analysis (in progress) we compare (a) the effect sizes of the acoustic realization of pitch accents and (b) the patterning of head and eyebrow gestures as predictors for words’ average perceived prominence levels. Results will be presented at the conference. (Less)
Please use this url to cite or link to this publication:
author
; and
organization
publishing date
type
Contribution to conference
publication status
unpublished
subject
conference name
10th Conference of the International Society for Gesture Studies
conference location
Nijmegen, Netherlands
conference dates
2025-07-09 - 2025-07-11
project
Språkbanken (Swedish Language Bank)
language
English
LU publication?
yes
id
69ca7790-ed20-4390-8b08-245796bf24fb
date added to LUP
2025-10-29 16:28:02
date last changed
2025-11-21 15:42:08
@misc{69ca7790-ed20-4390-8b08-245796bf24fb,
  abstract     = {{Co-speech head and eyebrow gestures often align with pitch accents in spoken language production, thereby likely instantiating what we could refer to as multimodal prominence. Indeed, when speech is perceived audio-visually, controlled experimental studies have shown that both pitch accents and head and eyebrow gestures, when independently manipulated, have sizable effects on perceived prominence. However, studies on naturally produced, co-occurring speech and gesture suggest that the acoustic realization of pitch accents tends to correlate with the presence and magnitude of gestures. Thus, according to the Cumulative-Cue Hypothesis, when a higher prominence level is to be produced, we tend to increase the effects of acoustic and gestural signals in conjunction, rather than independently of each other. This raises the question as to what extent the two modalities, speech and gesture, contribute to the overall audio-visual, or multimodal, prominence percept, in naturally produced multimodal speech. To approach this question, we study the co-occurrence of pitch accents and head and eyebrow gestures, and how these affect perceived prominence, in a data set of 60 brief audio-visually displayed Swedish television news readings (1936 words in total). In this data, accents and gestures are typically combined cumulatively (accent only, accent+head, accent+head+eyebrows). We manually annotated the occurrence of gestures, pitch accents, and tonal landmarks of the accents. From the tone labels, f0 values were extracted semi-automatically. The perceived prominence of all words in the dataset was measured using a web-based crowdsourcing approach, where adult volunteers rated each word in a semi-random selection of clips using a 3-point scale (manifested in color-changing word buttons). In our analysis (in progress) we compare (a) the effect sizes of the acoustic realization of pitch accents and (b) the patterning of head and eyebrow gestures as predictors for words’ average perceived prominence levels. Results will be presented at the conference.}},
  author       = {{Ambrazaitis, Gilbert and Frid, Johan and House, David}},
  language     = {{eng}},
  title        = {{Measuring the effects of pitch accent realization and the patterning of head and eyebrow gestures on perceived multimodal prominence}},
  year         = {{2025}},
}