Translating from monosemiotic to polysemiotic narratives : A study of Finnish speech and gestures
Louhema, Karoliina ; Zlatev, Jordan LU ; Graziano, Maria LU
and van de Weijer, Joost
LU
(2019)
In Sign Systems Studies
47(3/4).
p.480-525
- Abstract
- Human communication can be either monosemiotic or polysemiotic, depending on whether it combines ensembles of representations from one or more semiotic systems such as language, gesture and depiction. Each semiotic system has its unique storytelling potentials, which makes intersemiotic translation from one system to another challenging. We investigated the influence of the source semiotic system, realised in speech and a sequence of pictures, respectively, on the way the same story was retold using speech and co-speech gestures. The story was the content of the picture book Frog, where are you? A group of Finnish speakers saw the story in pictures, and another group heard it in matched oral narration. Each participant retold the story to... (More)
- Human communication can be either monosemiotic or polysemiotic, depending on whether it combines ensembles of representations from one or more semiotic systems such as language, gesture and depiction. Each semiotic system has its unique storytelling potentials, which makes intersemiotic translation from one system to another challenging. We investigated the influence of the source semiotic system, realised in speech and a sequence of pictures, respectively, on the way the same story was retold using speech and co-speech gestures. The story was the content of the picture book Frog, where are you? A group of Finnish speakers saw the story in pictures, and another group heard it in matched oral narration. Each participant retold the story to an addressee and all narrations were video-recorded and analysed for both speech and gestures. Given the high degree of iconicity in depiction, we expected more iconic gestures (especially enactments) in the narratives translated from pictures than from speech. Conversely, we expected greater narrative coherence in the narratives translated from speech. The results showed that more iconic gestures were produced in the narratives translated from speech, but these were primarily not from the enactment subtype. As expected, iconic enactments were more frequent in the narratives translated from the story presented in pictures. The narratives produced by participants who had only heard the story did not have a greater variety of connective devices, but the type of devices differed slightly between the groups. Together with some additional differences between the groups that had not been anticipated, the results indicate that a story presented in different semiotic systems tend to be translated into different polysemiotic narratives. (Less)
Please use this url to cite or link to this publication:
https://lup.lub.lu.se/record/b142bc19-1b08-478d-a82c-b152d7ccb455
- author
- Louhema, Karoliina
; Zlatev, Jordan
LU
; Graziano, Maria
LU
and van de Weijer, Joost
LU
- organization
- publishing date
- 2019-12-31
- type
- Contribution to journal
- publication status
- published
- subject
- in
- Sign Systems Studies
- volume
- 47
- issue
- 3/4
- pages
- 480 - 525
- publisher
- Tartu University Press
- external identifiers
-
- scopus:85082936207
- ISSN
- 1406-4243
- DOI
- 10.12697/SSS.2019.47.3-4.07
- language
- English
- LU publication?
- yes
- id
- b142bc19-1b08-478d-a82c-b152d7ccb455
- date added to LUP
- 2019-12-19 07:45:37
- date last changed
- 2023-11-19 20:55:24
@article{b142bc19-1b08-478d-a82c-b152d7ccb455,
abstract = {{Human communication can be either monosemiotic or polysemiotic, depending on whether it combines ensembles of representations from one or more semiotic systems such as language, gesture and depiction. Each semiotic system has its unique storytelling potentials, which makes intersemiotic translation from one system to another challenging. We investigated the influence of the source semiotic system, realised in speech and a sequence of pictures, respectively, on the way the same story was retold using speech and co-speech gestures. The story was the content of the picture book Frog, where are you? A group of Finnish speakers saw the story in pictures, and another group heard it in matched oral narration. Each participant retold the story to an addressee and all narrations were video-recorded and analysed for both speech and gestures. Given the high degree of iconicity in depiction, we expected more iconic gestures (especially enactments) in the narratives translated from pictures than from speech. Conversely, we expected greater narrative coherence in the narratives translated from speech. The results showed that more iconic gestures were produced in the narratives translated from speech, but these were primarily not from the enactment subtype. As expected, iconic enactments were more frequent in the narratives translated from the story presented in pictures. The narratives produced by participants who had only heard the story did not have a greater variety of connective devices, but the type of devices differed slightly between the groups. Together with some additional differences between the groups that had not been anticipated, the results indicate that a story presented in different semiotic systems tend to be translated into different polysemiotic narratives.}},
author = {{Louhema, Karoliina and Zlatev, Jordan and Graziano, Maria and van de Weijer, Joost}},
issn = {{1406-4243}},
language = {{eng}},
month = {{12}},
number = {{3/4}},
pages = {{480--525}},
publisher = {{Tartu University Press}},
series = {{Sign Systems Studies}},
title = {{Translating from monosemiotic to polysemiotic narratives : A study of Finnish speech and gestures}},
url = {{http://dx.doi.org/10.12697/SSS.2019.47.3-4.07}},
doi = {{10.12697/SSS.2019.47.3-4.07}},
volume = {{47}},
year = {{2019}},
}