Skip to main content

Lund University Publications

LUND UNIVERSITY LIBRARIES

A unified framework for real-time failure handling in robotics using vision-language models, reactive planner and behavior trees

Ahmad, Faseeh LU ; Ismail, Hashim LU orcid ; Styrud, Jonathan ; Stenmark, Maj LU and Krueger, Volker LU orcid (2025)
Abstract
Robotic systems often face execution failures due to unexpected obstacles, sensor errors, or environmental changes. Traditional failure recovery methods rely on predefined strategies or human intervention, making them less adaptable. This paper presents a unified failure recovery framework that combines Vision-Language Models (VLMs), a reactive planner, and Behavior Trees (BTs) to enable real-time failure handling. Our approach includes pre-execution verification, which checks for potential failures before execution, and reactive failure handling, which detects and corrects failures during execution by verifying existing BT conditions, adding missing preconditions and, when necessary, generating new skills. The framework uses a scene graph... (More)
Robotic systems often face execution failures due to unexpected obstacles, sensor errors, or environmental changes. Traditional failure recovery methods rely on predefined strategies or human intervention, making them less adaptable. This paper presents a unified failure recovery framework that combines Vision-Language Models (VLMs), a reactive planner, and Behavior Trees (BTs) to enable real-time failure handling. Our approach includes pre-execution verification, which checks for potential failures before execution, and reactive failure handling, which detects and corrects failures during execution by verifying existing BT conditions, adding missing preconditions and, when necessary, generating new skills. The framework uses a scene graph for structured environmental perception and an execution history for continuous monitoring, enabling context-aware and adaptive failure handling. We evaluate our framework through real-world experiments with an ABB YuMi robot on tasks like peg insertion, object sorting, and drawer placement, as well as in AI2-THOR simulator. Compared to using pre-execution and reactive methods separately, our approach achieves higher task success rates and greater adaptability. Ablation studies highlight the importance of VLM-based reasoning, structured scene representation, and execution history tracking for effective failure recovery in robotics. (Less)
Please use this url to cite or link to this publication:
author
; ; ; and
organization
publishing date
type
Book/Report
publication status
submitted
subject
pages
8 pages
DOI
10.48550/arXiv.2503.15202
project
Efficient Transfer Learning for Robot Skills
language
English
LU publication?
yes
id
2f5aa6f1-b779-4e64-97bf-ae4d3302d1ed
date added to LUP
2025-05-05 11:11:51
date last changed
2025-05-06 15:28:06
@proceedings{2f5aa6f1-b779-4e64-97bf-ae4d3302d1ed,
  abstract     = {{Robotic systems often face execution failures due to unexpected obstacles, sensor errors, or environmental changes. Traditional failure recovery methods rely on predefined strategies or human intervention, making them less adaptable. This paper presents a unified failure recovery framework that combines Vision-Language Models (VLMs), a reactive planner, and Behavior Trees (BTs) to enable real-time failure handling. Our approach includes pre-execution verification, which checks for potential failures before execution, and reactive failure handling, which detects and corrects failures during execution by verifying existing BT conditions, adding missing preconditions and, when necessary, generating new skills. The framework uses a scene graph for structured environmental perception and an execution history for continuous monitoring, enabling context-aware and adaptive failure handling. We evaluate our framework through real-world experiments with an ABB YuMi robot on tasks like peg insertion, object sorting, and drawer placement, as well as in AI2-THOR simulator. Compared to using pre-execution and reactive methods separately, our approach achieves higher task success rates and greater adaptability. Ablation studies highlight the importance of VLM-based reasoning, structured scene representation, and execution history tracking for effective failure recovery in robotics.}},
  author       = {{Ahmad, Faseeh and Ismail, Hashim and Styrud, Jonathan and Stenmark, Maj and Krueger, Volker}},
  language     = {{eng}},
  month        = {{03}},
  note         = {{Conference Editor}},
  title        = {{A unified framework for real-time failure handling in robotics using vision-language models, reactive planner and behavior trees}},
  url          = {{http://dx.doi.org/10.48550/arXiv.2503.15202}},
  doi          = {{10.48550/arXiv.2503.15202}},
  year         = {{2025}},
}