Advanced

I See What You’re Saying: The integration of complex speech and scenes during language comprehension

Andersson, Richard LU ; Ferreira, Fernanda and Henderson, John (2011) In Acta Psychologica 137(2). p.208-216
Abstract
The effect of language-driven eye movements in a visual scene with concurrent speech was examined using complex linguistic stimuli and complex scenes. The processing demands were manipulated using speech rate and the temporal distance between mentioned objects. This experiment differs from previous research by using complex photographic scenes, three-sentence utterances and mentioning four target objects. The main finding was that objects that are more slowly mentioned, more evenly placed and isolated in the speech stream are more likely to be fixated after having been mentioned and are fixated faster. Surprisingly, even objects mentioned in the most demanding conditions still show an effect of language-driven eye-movements. This supports... (More)
The effect of language-driven eye movements in a visual scene with concurrent speech was examined using complex linguistic stimuli and complex scenes. The processing demands were manipulated using speech rate and the temporal distance between mentioned objects. This experiment differs from previous research by using complex photographic scenes, three-sentence utterances and mentioning four target objects. The main finding was that objects that are more slowly mentioned, more evenly placed and isolated in the speech stream are more likely to be fixated after having been mentioned and are fixated faster. Surprisingly, even objects mentioned in the most demanding conditions still show an effect of language-driven eye-movements. This supports research using concurrent speech and visual scenes, and shows that the behavior of matching visual and linguistic information is likely to generalize to language situations of high information load. (Less)
Please use this url to cite or link to this publication:
author
organization
publishing date
type
Contribution to journal
publication status
published
subject
keywords
integration, vision, language, eye-movements, scene perception, complexity
in
Acta Psychologica
volume
137
issue
2
pages
208 - 216
publisher
Elsevier
external identifiers
  • pmid:21303711
  • wos:000292372600008
  • scopus:79957940673
ISSN
1873-6297
DOI
10.1016/j.actpsy.2011.01.007
project
Cognition, Communication and Learning
language
English
LU publication?
yes
id
3532016b-d530-47c6-b1a5-12a5731dd987 (old id 1759519)
date added to LUP
2011-01-27 09:38:46
date last changed
2017-07-30 03:21:37
@article{3532016b-d530-47c6-b1a5-12a5731dd987,
  abstract     = {The effect of language-driven eye movements in a visual scene with concurrent speech was examined using complex linguistic stimuli and complex scenes. The processing demands were manipulated using speech rate and the temporal distance between mentioned objects. This experiment differs from previous research by using complex photographic scenes, three-sentence utterances and mentioning four target objects. The main finding was that objects that are more slowly mentioned, more evenly placed and isolated in the speech stream are more likely to be fixated after having been mentioned and are fixated faster. Surprisingly, even objects mentioned in the most demanding conditions still show an effect of language-driven eye-movements. This supports research using concurrent speech and visual scenes, and shows that the behavior of matching visual and linguistic information is likely to generalize to language situations of high information load.},
  author       = {Andersson, Richard and Ferreira, Fernanda and Henderson, John},
  issn         = {1873-6297},
  keyword      = {integration,vision,language,eye-movements,scene perception,complexity},
  language     = {eng},
  number       = {2},
  pages        = {208--216},
  publisher    = {Elsevier},
  series       = {Acta Psychologica},
  title        = {I See What You’re Saying: The integration of complex speech and scenes during language comprehension},
  url          = {http://dx.doi.org/10.1016/j.actpsy.2011.01.007},
  volume       = {137},
  year         = {2011},
}