- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces LIBERO-Recover, a benchmark designed to evaluate failure recovery in robotic manipulation beyond conventional task success rates. The benchmark collects 2,178 failure scenarios from the execution of six embodied models and categorizes them into four difficulty levels, ranging from simple action retries to complex environmental recovery. It also provides 3,184 human-demonstrated recovery trajectories for training. Experiments on six VLA and world-action models reveal substantial performance degradation when encountering failure states, particularly those requiring object-state restoration or environmental reasoning. The authors further investigate the effects of action chunk size, temporal context, and recovery-oriented training, offering insights into the limitations of current manipulation policies.
Critical strengths
-
Important and practically relevant problem. The paper addresses a meaningful limitation of existing robotic manipulation benchmarks, where high task success rates do not necessarily translate into robust behavior after execution failures. Evaluating recovery capability is particularly relevant for deploying embodied agents in less controlled environments.
-
Systematic benchmark construction. The proposed four-level taxonomy provides a useful way to organize different types of manipulation failures according to their recovery requirements. Collecting failure states from actual model executions, rather than relying entirely on manually designed perturbations, is also a reasonable design choice.
-
Comprehensive empirical evaluation. The paper evaluates six representative VLA and world-action models across multiple LIBERO task suites. The results reveal substantial differences between standard task performance and recovery performance, demonstrating the practical value of the proposed benchmark.
Critical weaknesses
-
Limited novelty beyond benchmark construction. While failure recovery is an important problem, the paper primarily contributes a new evaluation dataset and taxonomy built on the existing LIBERO environment. The underlying methodology is relatively straightforward, and the proposed evaluation largely relies on task completion from selected failure states. The paper would benefit from a clearer explanation of what fundamentally distinguishes its evaluation protocol from existing robustness benchmarks beyond the source and categorization of failure states.
-
The evaluation does not fully capture online failure recovery. A central motivation of the paper is that robots should recognize failures, understand their consequences, and adapt their execution accordingly. However, the main evaluation initializes policies directly from predefined failure states. This primarily measures whether a policy can complete a task from an abnormal state, rather than whether it can detect a failure during its own execution and autonomously initiate recovery. These capabilities are closely related but not equivalent, and the current evaluation does not sufficiently distinguish them.
-
Potential biases in failure collection and categorization. The benchmark is constructed using failures generated by a limited set of existing policies, several of which are also evaluated as baselines. This raises concerns about whether the collected failure distribution adequately represents failures from unseen policies or different control architectures. In addition, although the appendix describes human verification of model-generated annotations, quantitative agreement statistics or annotation reliability measurements are not reported. Such analyses would help establish the validity of the four-level taxonomy.
Questions for authors
-
How does the proposed benchmark distinguish genuine failure recovery from generalization to unfamiliar initial states? Have the authors considered evaluating models on matched, non-failure states with comparable state deviations to isolate recovery-specific capabilities?
-
Can the authors clarify why the evaluation starts directly from failure states rather than requiring policies to identify failures during continuous execution? How much of the intended failure recognition and recovery planning capability is actually captured by the current protocol?
-
Since the failure scenarios are collected from a specific set of embodied models, how well does the benchmark generalize to failures produced by previously unseen policies? Are there any experiments that evaluate cross-policy failure distributions?
3 comments
Response to Reviewer va1o (2/2)
Thank you for the constructive comments. We address each concern in the review's order below. The revised PDF adds Appendix B, C and D;Major changes are in blue. And repetitive expressions in the appendix are streamlined.
Q1. Distinguishing Failure Recovery from Generalization to Unfamiliar States
We evaluate FastWAM and VLA-JEPA because neither model was used to collect the failure scenes, reducing potential bias associated with using the same models for collection and evaluation. Across all four suites, we compare failure states with paired pose-restored controls. Each control preserves the target object’s horizontal position and the states of the robotand other objects, while restoring the target object to a normally supported height and orientation on the same supporting surface. Both conditions retain the original task and use identical interaction budgets. Both conditions use the same task and interaction budget. Mean success rises from 15.74% to 39.96% for FastWAM and from 23.91% to 54.04% for VLA-JEPA, with improvements across all four suites. We also report LIBERO-Plus Layout results as an external reference. The performance differences in the table below highlight the fundamental distinction between our benchmark and robustness benchmarks based on object-position perturbations.
| Model | Setting | Spatial | Object | Goal | 100 | Avg. |
|---|---|---|---|---|---|---|
| FastWAM | Failure | 13.28 | 14.75 | 23.83 | 11.10 | 15.74 |
| FastWAM | Pose-restored | 38.25 | 35.58 | 57.08 | 28.91 | 39.96 |
| FastWAM | LIBERO-Plus Layout | — | — | — | — | 60.70 |
| VLA-JEPA | Failure | 19.75 | 34.78 | 22.93 | 18.18 | 23.91 |
| VLA-JEPA | Pose-restored | 53.33 | 72.16 | 54.00 | 36.67 | 54.04 |
| VLA-JEPA | LIBERO-Plus Layout | — | — | — | — | 79.50 |
Q2. Evaluation of Online Failure Detection and Recovery
As discussed in our response to Weakness 2, our protocol already evaluates online, closed-loop recovery. For single-frame policies without persistent memory, each action decision depends only on the current inputs. If a model cannot recover from a given failure state, replaying the preceding trajectory to reach that same state will not change its recovery capability. For history-dependent policies, we preserve the action and sensor observation history preceding the failure state. The DM0.5 and MemoryVLA experiments in Weakness 2 explicitly evaluate recovery with this additional context.
Our benchmark therefore supports online, closed-loop recovery. Whether a model can perform such recovery depends on its capabilities, rather than the benchmark. We believe our benchmark adequately covers the recovery capabilities we aim to evaluate, as its four levels encompass most failure states.
Q3. Generalization to Failure Distributions of Unseen Policies
Our findings generalize to models that were not used for failure collection. We evaluate four such models on our benchmark, with results shown below. The key finding remains consistent: these models also show substantial performance drops compared with their performance on the original LIBERO benchmark. Further experiments and detailed analyses are provided in Appendix C. RSR (%); each cell lists L1/L2/L3/L4.
| Model | Spatial | Object | Goal | 100 |
|---|---|---|---|---|
| FastWAM | 18/17.7/13.7/3.7 | 39.7/12.7/6.3/0.3 | 57.3/31.3/5/1.7 | 25.7/9.7/9/0 |
| DM0.5 | 72/45/23.3/8.7 | 80.3/73/60.7/19 | 81.7/46/21.7/7 | 55.7/29.7/13/0.3 |
| VLA-JEPA | 46/25/6.7/1.3 | 49.7/42.7/39/7.7 | 57/29/4.7/1 | 38.7/24.3/9.7/0 |
| G0.5 | 70/30/15/3 | 85.3/75.3/67.3/18.7 | 94/52.7/27.7/7.3 | 54/24.7/17.3/0 |
Thank You for Your Positive Assessment of Our Work
Dear Reviewer va1o,
Thank you very much for your positive assessment of our work. As the response period draws to a close, we would appreciate any feedback on whether our responses have addressed your concerns. If any questions remain, we would be happy to provide further clarification.
Thank you again for your time and thoughtful feedback.
Response to Reviewer va1o (1/2)
Thank you for the constructive comments. We address each concern in the review's order below. The revised PDF adds Appendix B, C and D;Major changes are in blue. And repetitive expressions in the appendix are streamlined.
W1. Novelty and Distinction from Existing Robustness Benchmarks
Beyond failure-state provenance and categorization, our evaluation protocol has three key distinctions.
First, we evaluate whether policies can handle abnormal object poses and environmental obstructions caused by execution failures, take corrective actions, and complete the original task. In the LIBERO-Plus layout examples in Fig.11, a relocated mug remains upright. A failed interaction may instead overturn it, changing feasible grasps and contact relations. The policy must select appropriate actions and, when necessary, restore object states or remove obstructions.
Second, the arm continues from its recorded intermediate pose at failure, without resetting to the task’s standard initial pose. Policies must handle the arm and object configurations left by failure.
Third, each failure scenario preserves the preceding actions and sensor observations of the policy that generated it, so recovery can use prior execution context. Policies supporting temporal inputs can combine current observations with execution history to interpret the current configuration and select subsequent actions.
We provide a more detailed comparison highlighting the differences between LIBERO-Recover and robustness benchmarks such as LIBERO-Plus and LIBERO-Pro in Appendix B and Fig. 11.
W2. Evaluation of Online Failure Detection and Recovery
Our current evaluation already involves online, closed-loop recovery: policies continuously receive new observations and autonomously select and execute corrective actions to complete the original task. The evaluator only specifies the starting state, without recovery guidance. Our benchmark therefore supports online, closed-loop recovery. Whether a model can perform such recovery depends on its capabilities, rather than the benchmark.
First, for single-frame policies (π0,Gr00TN1.5,OpenVLA) without persistent memory, recovery decisions depend on current inputs rather than the preceding execution history. Under matched physical states, policy inputs and control conditions, continuing from a restored failure state provides the same decision information as reaching that state through uninterrupted execution. Replaying the preceding trajectory therefore provides no additional recovery information. A policy struggling to select effective corrective actions from that state would face the same difficulty when encountering it during continuous execution.
Second, we evaluate history-conditioned online recovery with DM0.5 and MemoryVLA. With 300 trials per suite–level pair, history increases mean RSR across the 16 pairs from 39.8% to 41.1% and 21.7% to 22.8%, respectively. These gains are modest: LIBERO-100 L4 rates remain 0.3% and 0.0%. Thus, our protocol supports history-dependent policies, while difficult recovery cases remain challenging even with temporal context. RSR (%); each cell lists L1/L2/L3/L4. +H: history.
W3. Bias in Failure Collection and Annotation Reliability
First, VLA-JEPA, DM0.5, FastWAM, and G0.5 were excluded from failure-scenario construction and evaluated. All four show consistently weaker recovery on L3/L4 than on L1/L2, confirming our findings generalize beyond the failure-generating policies. RSR (%); each cell lists L1/L2/L3/L4.
Second, a 200-scene manual audit provides quantitative evidence of annotation reliability. We examined 50 scenes from each LIBERO suite, obtaining 100% agreement with manual verification for both failure-stage and recovery-level annotations. These results support the reliability of stage annotations and four-level recovery assignments within the audited sample.