Early or synchronized gestures facilitate speech recall — a study based on motion capture data
(2024) In Frontiers in Psychology 15.- Abstract
- Introduction: Temporal co-ordination between speech and gestures has been thoroughly studied in natural production. In most cases gesture strokes precede or coincide with the stressed syllable in words that they are semantically associated with.
Methods: To understand whether processing of speech and gestures is attuned to such temporal coordination, we investigated the effect of delaying, preposing or eliminating individual gestures on the memory for words in an experimental study in which 83 participants watched video sequences of naturalistic 3D-animated speakers generated based on motion capture data. A target word in the sequence appeared (a) with a gesture presented in its original position synchronized with speech, (b)... (More) - Introduction: Temporal co-ordination between speech and gestures has been thoroughly studied in natural production. In most cases gesture strokes precede or coincide with the stressed syllable in words that they are semantically associated with.
Methods: To understand whether processing of speech and gestures is attuned to such temporal coordination, we investigated the effect of delaying, preposing or eliminating individual gestures on the memory for words in an experimental study in which 83 participants watched video sequences of naturalistic 3D-animated speakers generated based on motion capture data. A target word in the sequence appeared (a) with a gesture presented in its original position synchronized with speech, (b) temporally shifted 500 ms before or (c) after the original position, or (d) with the gesture eliminated. Participants were asked to retell the videos in a free recall task. The strength of recall was operationalized as the inclusion of the target word in the free recall.
Results: Both eliminated and delayed gesture strokes resulted in reduced recall rates compared to synchronized strokes, whereas there was no difference between advanced (preposed) and synchronized strokes. An item-level analysis also showed that the greater the interval between the onsets of delayed strokes and stressed syllables in target words, the greater the negative effect was on recall.
Discussion: These results indicate that speech-gesture synchrony affects memory for speech, and that temporal patterns that are common in production lead to the best recall. Importantly, the study also showcases a procedure for using motion capture-based 3D-animated speakers to create an experimental paradigm for the study of speech-gesture comprehension. (Less)
Please use this url to cite or link to this publication:
https://lup.lub.lu.se/record/4dea6e57-3b2b-43c4-9acb-6aff4d464b3a
- author
- Nirme, Jens LU ; Gulz, Agneta LU ; Haake, Magnus LU and Gullberg, Marianne LU
- organization
-
- Ergonomics and Aerosol Technology
- LU Profile Area: Natural and Artificial Cognition
- LAMiNATE (Language Acquisition, Multilingualism, and Teaching) (research group)
- Communication and Cognition (research group)
- The Educational Technology Group (research group)
- Lund University Humanities Lab
- Cognitive Science
- General Linguistics
- eSSENCE: The e-Science Collaboration
- publishing date
- 2024-03-25
- type
- Contribution to journal
- publication status
- published
- subject
- keywords
- gesture, motion capture, Speech-gesture integration
- in
- Frontiers in Psychology
- volume
- 15
- article number
- 1345906
- pages
- 13 pages
- publisher
- Frontiers Media S. A.
- external identifiers
-
- scopus:85189919291
- pmid:38596333
- ISSN
- 1664-1078
- DOI
- 10.3389/fpsyg.2024.1345906
- project
- Thinking in Time: Cognition, Communication and Learning
- Embodied bilingualism (a Wallenberg Scholar project)
- language
- English
- LU publication?
- yes
- id
- 4dea6e57-3b2b-43c4-9acb-6aff4d464b3a
- date added to LUP
- 2024-03-25 19:08:33
- date last changed
- 2024-06-25 03:00:03
@article{4dea6e57-3b2b-43c4-9acb-6aff4d464b3a, abstract = {{Introduction: Temporal co-ordination between speech and gestures has been thoroughly studied in natural production. In most cases gesture strokes precede or coincide with the stressed syllable in words that they are semantically associated with.<br/>Methods: To understand whether processing of speech and gestures is attuned to such temporal coordination, we investigated the effect of delaying, preposing or eliminating individual gestures on the memory for words in an experimental study in which 83 participants watched video sequences of naturalistic 3D-animated speakers generated based on motion capture data. A target word in the sequence appeared (a) with a gesture presented in its original position synchronized with speech, (b) temporally shifted 500 ms before or (c) after the original position, or (d) with the gesture eliminated. Participants were asked to retell the videos in a free recall task. The strength of recall was operationalized as the inclusion of the target word in the free recall.<br/>Results: Both eliminated and delayed gesture strokes resulted in reduced recall rates compared to synchronized strokes, whereas there was no difference between advanced (preposed) and synchronized strokes. An item-level analysis also showed that the greater the interval between the onsets of delayed strokes and stressed syllables in target words, the greater the negative effect was on recall.<br/>Discussion: These results indicate that speech-gesture synchrony affects memory for speech, and that temporal patterns that are common in production lead to the best recall. Importantly, the study also showcases a procedure for using motion capture-based 3D-animated speakers to create an experimental paradigm for the study of speech-gesture comprehension.}}, author = {{Nirme, Jens and Gulz, Agneta and Haake, Magnus and Gullberg, Marianne}}, issn = {{1664-1078}}, keywords = {{gesture; motion capture; Speech-gesture integration}}, language = {{eng}}, month = {{03}}, publisher = {{Frontiers Media S. A.}}, series = {{Frontiers in Psychology}}, title = {{Early or synchronized gestures facilitate speech recall — a study based on motion capture data}}, url = {{http://dx.doi.org/10.3389/fpsyg.2024.1345906}}, doi = {{10.3389/fpsyg.2024.1345906}}, volume = {{15}}, year = {{2024}}, }