Closing the Loop on Task State: Robust Multi-Stage Robot Manipulation

In multi-stage robot manipulation, local deviations can propagate to later stages. Policies learned from successful demonstrations can be data-efficient, but deployment disturbances can separate the controller's assumed task state from the physical task state. Anomaly monitors detect unreliable execution, but detection signals alone do not specify how to revise task state. Supervised failure handling requires additional data from failed rollouts or corrective supervision, with effectiveness depending on coverage of deployment-time failures. The challenge is to use runtime evidence to guide recovery without collecting additional failure data. We introduce TSF (Task-State Feedback), which models valid task evolution from successful demonstrations alone. The TSF representation combines object-relative motion with interaction relations. At deployment, the inferred relations and progress regulate advancement under benign timing variations and guide recovery to a model-supported state after invalid transitions. On held-out failure types in RLBench, TSF achieves 2.44 times the average detection F1 of the failure-supervised GRU. With TSF monitoring fixed, full recovery achieves 2.46 times the intention-to-treat success rate of generic skill retry. TSF improves perturbed-task success by 65% over the same action policy without task-state feedback while satisfying aggregate nominal non-inferiority.