Skip to main content

Lund University Publications

LUND UNIVERSITY LIBRARIES

Eye Tracking-Based Speech Label Estimation for Auditory Attention Decoding with Portable EEG

Wilroth, Johanna ; Keding, Oskar LU ; Skoglund, Martin A. ; Alickovic, Emina and Enqvist, Martin (2025) 28th International Conference on Information Fusion, FUSION 2025
Abstract

In this study, we investigate integrating eye tracking with auditory attention decoding (AAD) using portable EEG devices, specifically a mobile EEG cap and cEEGrid, in a preliminary analysis with a single participant. A novel audiovisual dataset was collected using a mobile EEG system designed to simulate real-life listening environments. Our study has two main objectives: (1) to use eye tracking data to automatically infer the labels of attended and unattended speech streams, and (2) to train an AAD model using these estimated labels, evaluating its performance through speech reconstruction accuracy. The results demonstrate the feasibility of using eye tracking data to estimate attended speech labels, which were then used to train... (More)

In this study, we investigate integrating eye tracking with auditory attention decoding (AAD) using portable EEG devices, specifically a mobile EEG cap and cEEGrid, in a preliminary analysis with a single participant. A novel audiovisual dataset was collected using a mobile EEG system designed to simulate real-life listening environments. Our study has two main objectives: (1) to use eye tracking data to automatically infer the labels of attended and unattended speech streams, and (2) to train an AAD model using these estimated labels, evaluating its performance through speech reconstruction accuracy. The results demonstrate the feasibility of using eye tracking data to estimate attended speech labels, which were then used to train speech reconstruction models. We validated our models with varying amounts of training data and a second dataset from the same participant to assess generalization. Additionally, we examined the impact of mislabeling on AAD accuracy. These findings provide preliminary evidence that eye tracking can be used to infer speech labels, offering a potential pathway for brain-controlled hearing aids, where true labels are unknown.

(Less)
Please use this url to cite or link to this publication:
author
; ; ; and
organization
publishing date
type
Chapter in Book/Report/Conference proceeding
publication status
published
subject
keywords
audiovisual stimulus, auditory attention decoding, EEG, eye tracking, speech label, stimulus reconstruction
host publication
Proceedings of the 2025 28th International Conference on Information Fusion, FUSION 2025
publisher
IEEE - Institute of Electrical and Electronics Engineers Inc.
conference name
28th International Conference on Information Fusion, FUSION 2025
conference location
Rio de Janiero, Brazil
conference dates
2025-07-07 - 2025-07-11
external identifiers
  • scopus:105015862747
ISBN
9781037056239
DOI
10.23919/FUSION65864.2025.11123918
language
English
LU publication?
yes
id
966d59a1-b7ed-4c35-b7da-b8aceb1afcbd
date added to LUP
2025-11-12 12:26:18
date last changed
2025-11-12 12:26:53
@inproceedings{966d59a1-b7ed-4c35-b7da-b8aceb1afcbd,
  abstract     = {{<p>In this study, we investigate integrating eye tracking with auditory attention decoding (AAD) using portable EEG devices, specifically a mobile EEG cap and cEEGrid, in a preliminary analysis with a single participant. A novel audiovisual dataset was collected using a mobile EEG system designed to simulate real-life listening environments. Our study has two main objectives: (1) to use eye tracking data to automatically infer the labels of attended and unattended speech streams, and (2) to train an AAD model using these estimated labels, evaluating its performance through speech reconstruction accuracy. The results demonstrate the feasibility of using eye tracking data to estimate attended speech labels, which were then used to train speech reconstruction models. We validated our models with varying amounts of training data and a second dataset from the same participant to assess generalization. Additionally, we examined the impact of mislabeling on AAD accuracy. These findings provide preliminary evidence that eye tracking can be used to infer speech labels, offering a potential pathway for brain-controlled hearing aids, where true labels are unknown.</p>}},
  author       = {{Wilroth, Johanna and Keding, Oskar and Skoglund, Martin A. and Alickovic, Emina and Enqvist, Martin}},
  booktitle    = {{Proceedings of the 2025 28th International Conference on Information Fusion, FUSION 2025}},
  isbn         = {{9781037056239}},
  keywords     = {{audiovisual stimulus; auditory attention decoding; EEG; eye tracking; speech label; stimulus reconstruction}},
  language     = {{eng}},
  publisher    = {{IEEE - Institute of Electrical and Electronics Engineers Inc.}},
  title        = {{Eye Tracking-Based Speech Label Estimation for Auditory Attention Decoding with Portable EEG}},
  url          = {{http://dx.doi.org/10.23919/FUSION65864.2025.11123918}},
  doi          = {{10.23919/FUSION65864.2025.11123918}},
  year         = {{2025}},
}