On-Demand Human Assistance for Task Continuation under Physical Action Failures in LLM-based Planning
Abstract
While robot action planning based on large language models (LLMs) has advanced remarkably, continuing a task after a physical action failure during execution remains a key challenge. For example, after failing to grasp an object, the robot may not notice that it has fallen off the table and keep trying to detect and pick it up, stalling the task. Existing approaches either rely on closed-loop autonomous re-planning, which fails when errors exceed the robot's capabilities, or on human intervention without updating the LLM's plan, hindering subsequent planning. We present a system for LLM-based robot action planning that integrates remote human intervention with feedback-based replanning. When the LLM detects an action failure, the robot requests remote assistance; an operator resolves the failure through teleoperation and reports the outcome in natural language, which is fed back to the LLM to update its plan so that the task can continue. We evaluated the proposed system on a real-world trash-collection task, comparing it with a fully autonomous method and a teleoperation-only baseline. The proposed system improves task progress over the autonomous method in this case study. For easy-to-retrieve trash, the gap from the teleoperation reference was small, while a larger gap remained for harder-to-retrieve trash. These results suggest that, in this case study, combining remote human intervention with feedback-based replanning was associated with continued task execution even in the face of otherwise unrecoverable failures.
An overview of our study. When a robot drops an object and subsequently fails to detect it, it falls into a loop state. In such cases, the robot requests assistance from an operator on demand, and the issue is resolved through remote error resolution.
An overview of the proposed system. The LLM takes a user's instruction, decomposes it based on robot capabilities and rules, and assigns tasks to each robot. After that, LLMs generate executable actions, which the robots perform. When failures occur, a robot requests operator assistance. The operator resolves the issue remotely and provides feedback, allowing the LLM to update its understanding and resume execution.
Video Demonstration (Full)
BibTeX
@inproceedings{hasegawa2026repair,
title={{On-Demand Human Assistance for Task Continuation under Physical Action Failures in LLM-based Planning}},
author={Hasegawa, Shoichi and Taniguchi, Akira and El Hafi, Lotfi and Garcia Ricardez, Gustavo Alfonso and Taniguchi, Tadahiro},
booktitle={{Under Review}},
year={2026, under review}
}