Addressing Failures in Robotics Using Vision-Based Language Models (VLMs) and Behavior Trees (BT)
Ahmad, Faseeh LU ; Styrud, Jonathan and Krueger, Volker LU
(2025)
16th European Robotics Forum, ERF 2025
In Springer Proceedings in Advanced Robotics
36 SPAR.
p.281-287
- Abstract
In this paper, we propose an approach that combines Vision Language Models (VLMs) and Behavior Trees (BTs) to address failures in robotics. Current robotic systems can handle known failures with pre-existing recovery strategies, but they are often ill-equipped to manage unknown failures or anomalies. We introduce VLMs as a monitoring tool to detect and identify failures during task execution. Additionally, VLMs generate missing conditions or skill templates that are then incorporated into the BT, ensuring the system can autonomously address similar failures in future tasks. We validate our approach through simulations in several failure scenarios.
Please use this url to cite or link to this publication:
https://lup.lub.lu.se/record/61418efe-e883-4b24-bc56-595768c0dd65
- author
- Ahmad, Faseeh
LU
; Styrud, Jonathan
and Krueger, Volker
LU
- organization
- publishing date
- 2025
- type
- Chapter in Book/Report/Conference proceeding
- publication status
- published
- subject
- keywords
- Behavior Trees, Failure Detection, Recovery Behaviors, Robotics, Vision Language Models
- host publication
- European Robotics Forum 2025 - Boosting the Synergies between Robotics and AI for a Stronger Europe
- series title
- Springer Proceedings in Advanced Robotics
- editor
- Huber, Marco ; Verl, Alexander and Kraus, Werner
- volume
- 36 SPAR
- pages
- 7 pages
- publisher
- Springer Nature
- conference name
- 16th European Robotics Forum, ERF 2025
- conference location
- Stuttgart, Germany
- conference dates
- 2025-03-25 - 2025-03-27
- external identifiers
-
- scopus:105006603744
- ISSN
- 2511-1264
- 2511-1256
- ISBN
- 9783031894701
- DOI
- 10.1007/978-3-031-89471-8_43
- language
- English
- LU publication?
- yes
- additional info
- Publisher Copyright: © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.
- id
- 61418efe-e883-4b24-bc56-595768c0dd65
- date added to LUP
- 2025-08-15 13:11:55
- date last changed
- 2025-08-16 03:21:43
@inproceedings{61418efe-e883-4b24-bc56-595768c0dd65,
abstract = {{<p>In this paper, we propose an approach that combines Vision Language Models (VLMs) and Behavior Trees (BTs) to address failures in robotics. Current robotic systems can handle known failures with pre-existing recovery strategies, but they are often ill-equipped to manage unknown failures or anomalies. We introduce VLMs as a monitoring tool to detect and identify failures during task execution. Additionally, VLMs generate missing conditions or skill templates that are then incorporated into the BT, ensuring the system can autonomously address similar failures in future tasks. We validate our approach through simulations in several failure scenarios.</p>}},
author = {{Ahmad, Faseeh and Styrud, Jonathan and Krueger, Volker}},
booktitle = {{European Robotics Forum 2025 - Boosting the Synergies between Robotics and AI for a Stronger Europe}},
editor = {{Huber, Marco and Verl, Alexander and Kraus, Werner}},
isbn = {{9783031894701}},
issn = {{2511-1264}},
keywords = {{Behavior Trees; Failure Detection; Recovery Behaviors; Robotics; Vision Language Models}},
language = {{eng}},
pages = {{281--287}},
publisher = {{Springer Nature}},
series = {{Springer Proceedings in Advanced Robotics}},
title = {{Addressing Failures in Robotics Using Vision-Based Language Models (VLMs) and Behavior Trees (BT)}},
url = {{http://dx.doi.org/10.1007/978-3-031-89471-8_43}},
doi = {{10.1007/978-3-031-89471-8_43}},
volume = {{36 SPAR}},
year = {{2025}},
}