Skip to main content

Lund University Publications

LUND UNIVERSITY LIBRARIES

Real-time voice-based LLM integration for XR tutoring : a prototype implementation

Geris, Ali LU orcid and Alce, Günter LU (2026) 8th IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality, AIxVR 2026 In Proceedings - 2026 IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality, AIxVR 2026 p.285-289
Abstract

This paper presents a lightweight architecture for AI-driven tutoring in extended reality (XR) environments, integrating OpenAI's GPT-4o real-time API directly into Unitybased virtual reality (VR) to enable seamless voice-native interaction without external speech modules. Designed for taskbased learning, the tutor provides immediate, context-sensitive verbal feedback with adaptive brevity and low latency. A blockbased programming task demonstrates real-time evaluation, where user-assembled logic structures are analyzed and corrected through spoken responses. Across ten structured sessions and forty voice interactions, the prototype achieved consistent sub500 ms latency, generated under four minutes of total speech, and averaged 0.005... (More)

This paper presents a lightweight architecture for AI-driven tutoring in extended reality (XR) environments, integrating OpenAI's GPT-4o real-time API directly into Unitybased virtual reality (VR) to enable seamless voice-native interaction without external speech modules. Designed for taskbased learning, the tutor provides immediate, context-sensitive verbal feedback with adaptive brevity and low latency. A blockbased programming task demonstrates real-time evaluation, where user-assembled logic structures are analyzed and corrected through spoken responses. Across ten structured sessions and forty voice interactions, the prototype achieved consistent sub500 ms latency, generated under four minutes of total speech, and averaged 0.005 per turn. These results demonstrate the practical feasibility of low-overhead, real-time AI tutoring in XR and provide a foundation for future multimodal, context-aware learning environments.

(Less)
Please use this url to cite or link to this publication:
author
and
organization
publishing date
type
Chapter in Book/Report/Conference proceeding
publication status
published
subject
keywords
Real-time LLM, Voice interaction, XR tutoring
host publication
Proceedings - 2026 IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality, AIxVR 2026
series title
Proceedings - 2026 IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality, AIxVR 2026
pages
285 - 289
publisher
IEEE - Institute of Electrical and Electronics Engineers Inc.
conference name
8th IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality, AIxVR 2026
conference location
Osaka, Japan
conference dates
2026-01-26 - 2026-01-28
external identifiers
  • scopus:105037732289
ISBN
979-8-3315-4967-1
DOI
10.1109/AIxVR67263.2026.00050
language
English
LU publication?
yes
id
e0a7f305-afbf-442c-ba3e-7e34895988c1
date added to LUP
2026-05-15 07:47:32
date last changed
2026-06-15 16:29:24
@inproceedings{e0a7f305-afbf-442c-ba3e-7e34895988c1,
  abstract     = {{<p>This paper presents a lightweight architecture for AI-driven tutoring in extended reality (XR) environments, integrating OpenAI's GPT-4o real-time API directly into Unitybased virtual reality (VR) to enable seamless voice-native interaction without external speech modules. Designed for taskbased learning, the tutor provides immediate, context-sensitive verbal feedback with adaptive brevity and low latency. A blockbased programming task demonstrates real-time evaluation, where user-assembled logic structures are analyzed and corrected through spoken responses. Across ten structured sessions and forty voice interactions, the prototype achieved consistent sub500 ms latency, generated under four minutes of total speech, and averaged 0.005 per turn. These results demonstrate the practical feasibility of low-overhead, real-time AI tutoring in XR and provide a foundation for future multimodal, context-aware learning environments.</p>}},
  author       = {{Geris, Ali and Alce, Günter}},
  booktitle    = {{Proceedings - 2026 IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality, AIxVR 2026}},
  isbn         = {{979-8-3315-4967-1}},
  keywords     = {{Real-time LLM; Voice interaction; XR tutoring}},
  language     = {{eng}},
  pages        = {{285--289}},
  publisher    = {{IEEE - Institute of Electrical and Electronics Engineers Inc.}},
  series       = {{Proceedings - 2026 IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality, AIxVR 2026}},
  title        = {{Real-time voice-based LLM integration for XR tutoring : a prototype implementation}},
  url          = {{http://dx.doi.org/10.1109/AIxVR67263.2026.00050}},
  doi          = {{10.1109/AIxVR67263.2026.00050}},
  year         = {{2026}},
}