Skip to main content

Lund University Publications

LUND UNIVERSITY LIBRARIES

One algorithm to rule them all? : An evaluation and discussion of ten eye movement event-detection algorithms

Andersson, Richard LU ; Larsson, Linnéa LU ; Holmqvist, Kenneth LU ; Stridh, Martin LU and Nyström, Marcus LU orcid (2017) In Behavior Research Methods 49(2). p.616-637
Abstract
Almost all eye-movement researchers use algorithms to parse raw data and detect distinct types of eye movement events, such as fixations, saccades, and pursuit, and then base their results on these. Surprisingly, these algorithms are rarely evaluated. We evaluated the classifications of ten eye-movement event detection algorithms, on data from an SMI HiSpeed 1250 system, and compared them to manual ratings of two human experts. The evaluation focused on fixations, saccades, and post-saccadic oscillations. The evaluation used both event duration parameters, and sample-by-sample comparisons to rank the algorithms. The resulting event durations varied substantially as a function of what algorithm was used. This evaluation differed from... (More)
Almost all eye-movement researchers use algorithms to parse raw data and detect distinct types of eye movement events, such as fixations, saccades, and pursuit, and then base their results on these. Surprisingly, these algorithms are rarely evaluated. We evaluated the classifications of ten eye-movement event detection algorithms, on data from an SMI HiSpeed 1250 system, and compared them to manual ratings of two human experts. The evaluation focused on fixations, saccades, and post-saccadic oscillations. The evaluation used both event duration parameters, and sample-by-sample comparisons to rank the algorithms. The resulting event durations varied substantially as a function of what algorithm was used. This evaluation differed from previous evaluations by considering a relatively large set of algorithms, multiple events, and data from both static and dynamic stimuli. The main conclusion is that current detectors of only fixations and saccades work reasonably well for static stimuli, but barely better than chance for dynamic stimuli. Differing results across evaluation methods make it difficult to select one winner for fixation detection. For saccade detection, however, the algorithm by Larsson, Nyström and Stridh (IEEE Transaction on Biomedical Engineering, 60(9):2484–2493,2013) outperforms all algorithms in data from both static and dynamic stimuli. The data also show how improperly selected algorithms applied to dynamic data misestimate fixation and saccade properties. (Less)
Please use this url to cite or link to this publication:
author
; ; ; and
organization
publishing date
type
Contribution to journal
publication status
published
subject
keywords
ögonrörelsemätning, Eye-tracking, Inter-rater reliability, Parsing
in
Behavior Research Methods
volume
49
issue
2
pages
616 - 637
publisher
Springer
external identifiers
  • scopus:84969836647
  • pmid:27193160
  • wos:000400391800016
ISSN
1554-3528
DOI
10.3758/s13428-016-0738-9
language
English
LU publication?
yes
id
784a92cf-a4aa-452f-9acd-faafd94366cd
date added to LUP
2016-04-20 14:21:57
date last changed
2023-02-07 18:50:59
@article{784a92cf-a4aa-452f-9acd-faafd94366cd,
  abstract     = {{Almost all eye-movement researchers use algorithms to parse raw data and detect distinct types of eye movement events, such as fixations, saccades, and pursuit, and then base their results on these. Surprisingly, these algorithms are rarely evaluated. We evaluated the classifications of ten eye-movement event detection algorithms, on data from an SMI HiSpeed 1250 system, and compared them to manual ratings of two human experts. The evaluation focused on fixations, saccades, and post-saccadic oscillations. The evaluation used both event duration parameters, and sample-by-sample comparisons to rank the algorithms. The resulting event durations varied substantially as a function of what algorithm was used. This evaluation differed from previous evaluations by considering a relatively large set of algorithms, multiple events, and data from both static and dynamic stimuli. The main conclusion is that current detectors of only fixations and saccades work reasonably well for static stimuli, but barely better than chance for dynamic stimuli. Differing results across evaluation methods make it difficult to select one winner for fixation detection. For saccade detection, however, the algorithm by Larsson, Nyström and Stridh (IEEE Transaction on Biomedical Engineering, 60(9):2484–2493,2013) outperforms all algorithms in data from both static and dynamic stimuli. The data also show how improperly selected algorithms applied to dynamic data misestimate fixation and saccade properties.}},
  author       = {{Andersson, Richard and Larsson, Linnéa and Holmqvist, Kenneth and Stridh, Martin and Nyström, Marcus}},
  issn         = {{1554-3528}},
  keywords     = {{ögonrörelsemätning; Eye-tracking; Inter-rater reliability; Parsing}},
  language     = {{eng}},
  number       = {{2}},
  pages        = {{616--637}},
  publisher    = {{Springer}},
  series       = {{Behavior Research Methods}},
  title        = {{One algorithm to rule them all? : An evaluation and discussion of ten eye movement event-detection algorithms}},
  url          = {{https://lup.lub.lu.se/search/files/16640393/2016_Andersson_Algorithms_eval_accepted_version.pdf}},
  doi          = {{10.3758/s13428-016-0738-9}},
  volume       = {{49}},
  year         = {{2017}},
}