Eye Tracking-Based Speech Label Estimation for Auditory Attention Decoding with Portable EEG - Wilroth, Johanna; Keding, Oskar; Skoglund, Martin A.; Alickovic, Emina; Enqvist, Martin (28th International Conference on Information Fusion, FUSION 2025, Rio de Janiero, Brazil, 2025-07-07 - 2025-07-11, 2025)
In this study, we investigate integrating eye tracking with auditory attention decoding (AAD) using portable EEG devices, specifically a mobile EEG cap and cEEGrid, in a preliminary analysis with a single participant. A novel audiovisual dataset was collected using a mobile EEG system designed to simulate real-life listening environments. Our study has two main objectives: (1) to use eye tracking data to automatically infer the labels of attended and unattended speech streams, and (2) to train an AAD model using these estimated labels, evaluating its performance through speech reconstruction accuracy. The results demonstrate the feasibility of using eye tracking data to estimate attended speech labels, which were then used to train speech reconstruction models. We validated our models with varying amounts of training data and a second dataset from the same participant to assess generalization. Additionally, we examined the impact of mislabeling on AAD accuracy. These findings provide preliminary evidence that eye tracking can be used to infer speech labels, offering a potential pathway for brain-controlled hearing aids, where true labels are unknown.
Keywords: audiovisual stimulus; auditory attention decoding; EEG; eye tracking; speech label; stimulus reconstruction