acceptodds
Under review as a conference paper at ICLR 2027

LIBERO-RECOVER: Beyond Task Success Towards Failure Recovery in Robotic Manipulation Models

10 views

Abstract

Vision-Language-Action (VLA) or World Action (WAM) models have recently demonstrated remarkable performance in robotic manipulation. On LIBERO, SOTA method have achieved nearly 100% success rates, seemingly suggesting that the models are ready for deployment in real world. However, near perfect performance on existing benchmarks can be misleading: success under ideal conditions does not imply real world robustness. Existing benchmarks primarily evaluate task completion from predefined initial states, while real world interactions inevitably involve failures such as failed grasps, collisions, and unintended object movements. A robot must therefore not only execute tasks successfully, but also recognize and recover from failures to continue the task. Yet this capability remains largely unmeasured, revealing a critical gap between benchmark performance and real world reliability. To address this gap, we introduce LIBERO-Recover Benchmark, a large scale benchmark for failure recovery in robotic manipulation. Built upon LIBERO, we collect real execution failures from SOTA embodied models and construct 2,000+ scenarios across four recovery levels: (1) Action Retry, (2) Action Adaptation, (3) Object State Recovery, and (4) Environmental Recovery. We evaluate four core capabilities: spatial understanding, object structure reasoning, interaction understanding, and topological reasoning. As the first large-scale benchmark for embodied failure recovery, LIBERO-Recover shifts evaluation from ”Can the robot succeed?” to ”Can the robot recover after failure?”, promoting robust and generalizable embodied agents. The project will be available.

Primary area:
datasets and benchmarks
Keywords:
Robotic Manipulation, Failure Recovery, Embodied AI, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_va1oOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LIBERO-Recover, a benchmark designed to evaluate failure recovery in robotic manipulation beyond conventional task success rates. The benchmark collects 2,178 failure scenarios from the execution of six embodied models and categorizes them into four difficulty levels, ranging from simple action retries to complex environmental recovery. It also provides 3,184 human-demonstrated recovery trajectories for training. Experiments on six VLA and world-action models reveal substantial performance degradation when encountering failure states, particularly those requiring object-state restoration or environmental reasoning. The authors further investigate the effects of action chunk size, temporal context, and recovery-oriented training, offering insights into the limitations of current manipulation policies.

Critical strengths
  1. Important and practically relevant problem. The paper addresses a meaningful limitation of existing robotic manipulation benchmarks, where high task success rates do not necessarily translate into robust behavior after execution failures. Evaluating recovery capability is particularly relevant for deploying embodied agents in less controlled environments.

  2. Systematic benchmark construction. The proposed four-level taxonomy provides a useful way to organize different types of manipulation failures according to their recovery requirements. Collecting failure states from actual model executions, rather than relying entirely on manually designed perturbations, is also a reasonable design choice.

  3. Comprehensive empirical evaluation. The paper evaluates six representative VLA and world-action models across multiple LIBERO task suites. The results reveal substantial differences between standard task performance and recovery performance, demonstrating the practical value of the proposed benchmark.

Critical weaknesses
  1. Limited novelty beyond benchmark construction. While failure recovery is an important problem, the paper primarily contributes a new evaluation dataset and taxonomy built on the existing LIBERO environment. The underlying methodology is relatively straightforward, and the proposed evaluation largely relies on task completion from selected failure states. The paper would benefit from a clearer explanation of what fundamentally distinguishes its evaluation protocol from existing robustness benchmarks beyond the source and categorization of failure states.

  2. The evaluation does not fully capture online failure recovery. A central motivation of the paper is that robots should recognize failures, understand their consequences, and adapt their execution accordingly. However, the main evaluation initializes policies directly from predefined failure states. This primarily measures whether a policy can complete a task from an abnormal state, rather than whether it can detect a failure during its own execution and autonomously initiate recovery. These capabilities are closely related but not equivalent, and the current evaluation does not sufficiently distinguish them.

  3. Potential biases in failure collection and categorization. The benchmark is constructed using failures generated by a limited set of existing policies, several of which are also evaluated as baselines. This raises concerns about whether the collected failure distribution adequately represents failures from unseen policies or different control architectures. In addition, although the appendix describes human verification of model-generated annotations, quantitative agreement statistics or annotation reliability measurements are not reported. Such analyses would help establish the validity of the four-level taxonomy.

Questions for authors
  1. How does the proposed benchmark distinguish genuine failure recovery from generalization to unfamiliar initial states? Have the authors considered evaluating models on matched, non-failure states with comparable state deviations to isolate recovery-specific capabilities?

  2. Can the authors clarify why the evaluation starts directly from failure states rather than requiring policies to identify failures during continuous execution? How much of the intended failure recognition and recovery planning capability is actually captured by the current protocol?

  3. Since the failure scenarios are collected from a specific set of embodied models, how well does the benchmark generalize to failures produced by previously unseen policies? Are there any experiments that evaluate cross-policy failure distributions?

3 comments
AuthorsOpenReview ↗

Response to Reviewer va1o (1/2)

Thank you for the constructive comments. We address each concern in the review's order below. The revised PDF adds Appendix B, C and D;Major changes are in blue. And repetitive expressions in the appendix are streamlined.

W1. Novelty and Distinction from Existing Robustness Benchmarks

Beyond failure-state provenance and categorization, our evaluation protocol has three key distinctions.

First, we evaluate whether policies can handle abnormal object poses and environmental obstructions caused by execution failures, take corrective actions, and complete the original task. In the LIBERO-Plus layout examples in Fig.11, a relocated mug remains upright. A failed interaction may instead overturn it, changing feasible grasps and contact relations. The policy must select appropriate actions and, when necessary, restore object states or remove obstructions.

Second, the arm continues from its recorded intermediate pose at failure, without resetting to the task’s standard initial pose. Policies must handle the arm and object configurations left by failure.

Third, each failure scenario preserves the preceding actions and sensor observations of the policy that generated it, so recovery can use prior execution context. Policies supporting temporal inputs can combine current observations with execution history to interpret the current configuration and select subsequent actions.

We provide a more detailed comparison highlighting the differences between LIBERO-Recover and robustness benchmarks such as LIBERO-Plus and LIBERO-Pro in Appendix B and Fig. 11.

W2. Evaluation of Online Failure Detection and Recovery

Our current evaluation already involves online, closed-loop recovery: policies continuously receive new observations and autonomously select and execute corrective actions to complete the original task. The evaluator only specifies the starting state, without recovery guidance. Our benchmark therefore supports online, closed-loop recovery. Whether a model can perform such recovery depends on its capabilities, rather than the benchmark.

First, for single-frame policies (π0,Gr00TN1.5,OpenVLA\pi_{0}, Gr00T N1.5, OpenVLA) without persistent memory, recovery decisions depend on current inputs rather than the preceding execution history. Under matched physical states, policy inputs and control conditions, continuing from a restored failure state provides the same decision information as reaching that state through uninterrupted execution. Replaying the preceding trajectory therefore provides no additional recovery information. A policy struggling to select effective corrective actions from that state would face the same difficulty when encountering it during continuous execution.

Second, we evaluate history-conditioned online recovery with DM0.5 and MemoryVLA. With 300 trials per suite–level pair, history increases mean RSR across the 16 pairs from 39.8% to 41.1% and 21.7% to 22.8%, respectively. These gains are modest: LIBERO-100 L4 rates remain 0.3% and 0.0%. Thus, our protocol supports history-dependent policies, while difficult recovery cases remain challenging even with temporal context. RSR (%); each cell lists L1/L2/L3/L4. +H: history.

ModelSpatialObjectGoal100
DM0.572/45/23.3/8.780.3/73/60.7/1981.7/46/21.7/755.7/29.7/13/0.3
DM0.5+H73/48/25.7/1083.3/73/61.3/20.782/47.3/23/8.357/30.7/13.3/0.3
MemoryVLA36.7/22.3/13.7/6.745/34.3/18.7/956.3/30/11/5.331.3/19/8.3/0
MemoryVLA+H38.3/24.3/16/8.344.7/36/20/8.758.3/31.7/11.3/632.7/18.7/9/0

W3. Bias in Failure Collection and Annotation Reliability
First, VLA-JEPA, DM0.5, FastWAM, and G0.5 were excluded from failure-scenario construction and evaluated. All four show consistently weaker recovery on L3/L4 than on L1/L2, confirming our findings generalize beyond the failure-generating policies. RSR (%); each cell lists L1/L2/L3/L4.

ModelSpatialObjectGoal100
FastWAM18/17.7/13.7/3.739.7/12.7/6.3/0.357.3/31.3/5/1.725.7/9.7/9/0
DM0.572/45/23.3/8.780.3/73/60.7/1981.7/46/21.7/755.7/29.7/13/0.3
VLA-JEPA46/25/6.7/1.349.7/42.7/39/7.757/29/4.7/138.7/24.3/9.7/0
G0.570/30/15/385.3/75.3/67.3/18.794/52.7/27.7/7.354/24.7/17.3/0

Second, a 200-scene manual audit provides quantitative evidence of annotation reliability. We examined 50 scenes from each LIBERO suite, obtaining 100% agreement with manual verification for both failure-stage and recovery-level annotations. These results support the reliability of stage annotations and four-level recovery assignments within the audited sample.

LIBERO Suite#ScenesStage (%)Level (%)
Spatial50100100
Object50100100
Goal50100100
LIBERO-10050100100
Total200100100
AuthorsOpenReview ↗

Response to Reviewer va1o (2/2)

Thank you for the constructive comments. We address each concern in the review's order below. The revised PDF adds Appendix B, C and D;Major changes are in blue. And repetitive expressions in the appendix are streamlined.

Q1. Distinguishing Failure Recovery from Generalization to Unfamiliar States
We evaluate FastWAM and VLA-JEPA because neither model was used to collect the failure scenes, reducing potential bias associated with using the same models for collection and evaluation. Across all four suites, we compare failure states with paired pose-restored controls. Each control preserves the target object’s horizontal position and the states of the robotand other objects, while restoring the target object to a normally supported height and orientation on the same supporting surface. Both conditions retain the original task and use identical interaction budgets. Both conditions use the same task and interaction budget. Mean success rises from 15.74% to 39.96% for FastWAM and from 23.91% to 54.04% for VLA-JEPA, with improvements across all four suites. We also report LIBERO-Plus Layout results as an external reference. The performance differences in the table below highlight the fundamental distinction between our benchmark and robustness benchmarks based on object-position perturbations.

ModelSettingSpatialObjectGoal100Avg.
FastWAMFailure13.2814.7523.8311.1015.74
FastWAMPose-restored38.2535.5857.0828.9139.96
FastWAMLIBERO-Plus Layout————60.70
VLA-JEPAFailure19.7534.7822.9318.1823.91
VLA-JEPAPose-restored53.3372.1654.0036.6754.04
VLA-JEPALIBERO-Plus Layout————79.50

Q2. Evaluation of Online Failure Detection and Recovery

As discussed in our response to Weakness 2, our protocol already evaluates online, closed-loop recovery. For single-frame policies without persistent memory, each action decision depends only on the current inputs. If a model cannot recover from a given failure state, replaying the preceding trajectory to reach that same state will not change its recovery capability. For history-dependent policies, we preserve the action and sensor observation history preceding the failure state. The DM0.5 and MemoryVLA experiments in Weakness 2 explicitly evaluate recovery with this additional context.

Our benchmark therefore supports online, closed-loop recovery. Whether a model can perform such recovery depends on its capabilities, rather than the benchmark. We believe our benchmark adequately covers the recovery capabilities we aim to evaluate, as its four levels encompass most failure states.

Q3. Generalization to Failure Distributions of Unseen Policies

Our findings generalize to models that were not used for failure collection. We evaluate four such models on our benchmark, with results shown below. The key finding remains consistent: these models also show substantial performance drops compared with their performance on the original LIBERO benchmark. Further experiments and detailed analyses are provided in Appendix C. RSR (%); each cell lists L1/L2/L3/L4.

ModelSpatialObjectGoal100
FastWAM18/17.7/13.7/3.739.7/12.7/6.3/0.357.3/31.3/5/1.725.7/9.7/9/0
DM0.572/45/23.3/8.780.3/73/60.7/1981.7/46/21.7/755.7/29.7/13/0.3
VLA-JEPA46/25/6.7/1.349.7/42.7/39/7.757/29/4.7/138.7/24.3/9.7/0
G0.570/30/15/385.3/75.3/67.3/18.794/52.7/27.7/7.354/24.7/17.3/0
AuthorsOpenReview ↗

Thank You for Your Positive Assessment of Our Work

Dear Reviewer va1o,

Thank you very much for your positive assessment of our work. As the response period draws to a close, we would appreciate any feedback on whether our responses have addressed your concerns. If any questions remain, we would be happy to provide further clarification.

Thank you again for your time and thoughtful feedback.

Reviewer_o67NOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

LIBERO-Recover is a benchmark that collects naturally occurring failures from six deployed embodied models (π0, π0-Fast, GR00T N1.5, OpenVLA-OFT, Wan2-Policy, Cosmos-Predict2-Policy) on top of LIBERO, rather than manually perturbing object poses. It yields 2,178 recovery scenarios over 130 subtasks, four recovery levels (L1 Action Retry, L2 Action Adaptation, L3 Object State Recovery, L4 Environmental Recovery), and 16 evaluation dimensions; four teleoperators collected 3,184 recovery trajectories across 413 scenarios. Eight Key Findings contrast nominal success with recovery, e.g., >50% performance drop on natural failures, L1/L2 >> L3/L4, compounding post-failure degradation, WAM policies' higher Recovery Consistency, smaller action chunks helping, and temporal context aiding recovery.

Critical strengths
  1. Instead of injecting scripted perturbations, failures arise from real policy executions; Qwen3.5-27B assists localization/categorization followed by human review (Stage 2–3), and recovery demonstrations are teleoperated (§3.1). This fills a recognized gap between one-shot LIBERO success and post-failure recovery.
  2. RSR per level, Recovery Consistency (RC), Recovery Degradation (RD) over pre/during/post failure, chunk-size and temporal-prompt ablations, and a mixed-training experiment (Key Finding 6) give the community more than a leaderboard.
Critical weaknesses
  1. Tables/figures report point RSR without standard deviation, confidence intervals, or significance tests
  2. L3/L4 scenarios are far fewer (Fig. 4), and this empirical imbalance is preserved; RSR on sparse cells has high variance, yet is used to claim a "wall" between L2 and L3.
Questions for authors

Add a recovery-trained/fine-tuned VLA and at least one more WAM, and state whether high-level cells can be supplemented rather than left imbalanced.

2 comments
AuthorsOpenReview ↗

Response to Reviewer o67N

Thank you for the constructive comments. We address each concern in the review's order below. The revised PDF adds Appendix B, C and D; Major changes are in blue. And repetitive expressions in the appendix are streamlined.

W1. Statistical reliability
We evaluated each model using five random seeds, with 300 trials per recovery level per seed (1,500 trials per level). The inter-seed SD was at most 1.11 percentage points across all settings, supporting the reliability of our key findings. For example, PI achieved 40.0% ± 0.85% on LIBERO-Spatial L1 (95% CI: [37.5%, 42.5%]). Detailed SDs, 95% Wilson CIs, and per-level sample counts are provided in Appendix C.2.

W2. L3/L4 Sample Imbalance and Reliability of Recovery Findings
We retain the naturally imbalanced distribution because L3/L4 failures occur less frequently in collected model executions, reflecting the empirical distribution of observed failures. To assess whether the limited number of challenging scenarios affects our findings, we collected additional L3/L4 scenes. The expanded scene counts are reported in Appendix C (Tab.4). We further evaluated representative models on these augmented sets, with results shown in the table below. The performance trends on L3 and L4 remain consistent after scenario expansion, indicating that the observed recovery gap is not merely an artifact of the limited number of L3/L4 scenarios. Further details are provided in Appendix C.3.

MethodSpatial-L3Spatial-L4Object-L3Object-L4Goal-L3Goal-L4LIBERO-100-L3LIBERO-100-L4
PI13.26.624.08.85.92.710.20.0
GR00T6.93.839.81.85.60.017.60.0
Cos.14.60.08.40.06.30.011.00.0

Q1. Additional Model Validation and L3/L4 Scenario Expansion
We further evaluated a recovery-data fine-tuned VLA (VLA-JEPA) and an additional WAM (FastWAM). As shown in the table below, fine-tuning with expert recovery data improves their average RSR from 23.9% to 25.8% and from 15.8% to 18.5%, respectively. Additional L3/L4 scenarios can be collected by following the construction pipeline in Fig. 3. We collected additional L3/L4 scenarios and Appendix Tabs 4 and 8 report the expanded sample counts and evaluation results.

EvaluationMethodSpatialObjectGoalLIBERO-100
LIBEROFastWAM0.9760.9900.9500.940
FastWAM (Combined)0.9800.9940.9540.944
VLA-JEPA0.9620.9960.9720.958
VLA-JEPA (Combined)0.9580.9920.9820.962
LIBERO-RecoverFastWAM0.1330.1480.2380.111
FastWAM (Combined)0.1580.1830.2670.133
VLA-JEPA0.1980.3480.2290.182
VLA-JEPA (Combined)0.2080.3830.2500.191
AuthorsOpenReview ↗

Thank You for Your Positive Assessment of Our Work

Dear Reviewer o67N,

Thank you very much for your positive assessment of our work. As the response period draws to a close, we would appreciate any feedback on whether our responses have addressed your concerns. If any questions remain, we would be happy to provide further clarification.

Thank you again for your time and thoughtful feedback.

Reviewer_SDvWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents LIBERO-Recover, a benchmark for evaluating failure recovery in robotic manipulation. The authors categorize failures into four recovery levels (Action Retry, Action Adaptation, Object State Recovery, and Environmental Recovery). They propose three metrics (RSR, RD, and RC), evaluate six models, and provide several key findings for the community.

Critical strengths

Strengths  The problem is valuable. As LIBERO has become saturated, the paper shifts the focus to evaluating failure recovery.  Failure states are produced by actual executions of multiple models, which is closer to the real failure distribution.  The recovery levels are clearly defined, and the results show a clear difficulty gradient across levels.  The analysis covers a relatively rich set of dimensions, which provides several key findings for the community.

Critical weaknesses

Weaknesses  The writing is rough and needs further polishing.  A benchmark requires high data quality, yet the annotation quality in this work is not quantified.  The benchmark is built entirely in simulation, so it is unclear whether the failure modes match those on real robots.  Some of the key findings may require further validation.

Questions for authors

Questions  The evaluation results appear to be noisy, which may affect the reliability of the key findings. Could the authors provide results over multiple random seeds, confidence intervals, and the number of samples per recovery level?  The failure generators overlap with the evaluated models. Could this introduce selection bias?  Please provide further explanation and analysis of the proposed metrics.

3 comments
AuthorsOpenReview ↗

Response to Reviewer SDvW (1/2)

Thank you for the constructive comments. We address each concern in the review's order below. The revised PDF adds Appendix B, C and D; Major changes are in blue. And repetitive expressions in the appendix are streamlined.

W1. Writing Quality and Clarity
We have revised the text to improve clarity and corrected all grammatical and spelling errors throughout the paper, especially the Appendix.

W2. Annotation Quality
We manually audited 200 annotated scenes (50 per LIBERO suite). Both failure-stage and recovery-level annotations achieved 100% agreement with our manual verification, as summarized in the table below:

LIBERO Suite#ScenesStage (%)Level (%)
Spatial50100100
Object50100100
Goal50100100
LIBERO-10050100100
Total200100100

W3. Real-world failure correspondence
We deployed π0.5\pi_{0.5} on a physical robot to execute tasks analogous to LIBERO. Appendix D (Fig 15) presents three simulated–real-world failure comparisons, showing similar failure modes, including toppled objects and failed placements. These results provide qualitative evidence that LIBERO-Recover captures failure modes relevant to real-world robotic execution.

W4. Validation of key findings
We further validate our key findings using four additional models: VLA-JEPA, DM0.5, FastWAM, and G0.5. Due to evaluation time constraints, we randomly sampled 30 scenes for each recovery level within each suite (We further supplemented the number of L3 and L4 scenarios, with the updated counts detailed in Tab.4 of the Appendix). The table below presents representative results, with further experiments and analyses provided in the revised Appendix C.1.

MethodSpatialObjectGoal100
L1L2L3L4L1L2L3L4L1L2L3L4L1L2L3L4
FastWAM18.017.713.73.739.712.76.30.357.331.35.01.725.79.79.00.0
DM$_{0.5}$72.045.023.38.780.373.060.719.081.746.021.77.055.729.713.00.3
VLA-JEPA46.025.06.71.349.742.739.07.757.029.04.71.038.724.39.70.0
G$_{0.5}$70.030.015.03.085.375.367.318.794.052.727.77.354.024.717.30.0
AuthorsOpenReview ↗

Response to Reviewer SDvW (2/2)

Thank you for the constructive comments. We address each concern in the review's order below. The revised PDF adds Appendix B, C and D; Major changes are in blue. And repetitive expressions in the appendix are streamlined.

Q1. Statistical reliability
We evaluated all models using five random seeds with fixed evaluation scenarios. The evaluation settings follow those in Weakness 3. The results, including confidence intervals and per-level sample counts, are summarized in the table below, with further analysis provided in Appendix C.2. Despite variations across models, the overall trends and key findings remain consistent across seeds.

SuiteModelL1 (RSR±SD)95% CIL2 (RSR±SD)95% CIL3 (RSR±SD)95% CIL4 (RSR±SD)95% CI
SpatialPI40.0±0.85[37.5,42.5]12.3±0.67[10.8,14.1]14.0±0.71[12.3,15.8]6.7±0.47[5.5,8.0]
FAST30.0±0.85[27.7,32.4]12.0±0.62[10.5,13.7]7.0±0.53[5.8,8.4]2.7±0.41[2.0,3.6]
GR00T7.3±0.58[6.1,8.8]14.3±0.71[12.7,16.2]6.3±0.47[5.2,7.7]3.7±0.41[2.8,4.7]
OFT41.3±1.11[38.9,43.8]13.0±0.78[11.4,14.8]13.3±0.71[11.7,15.1]8.7±0.62[7.3,10.2]
Wan.17.3±0.62[15.5,19.3]2.7±0.33[2.0,3.6]11.3±0.71[9.8,13.0]12.0±0.53[10.5,13.7]
Cos.8.0±0.58[6.7,9.5]4.7±0.41[3.7,5.9]14.0±0.67[12.3,15.8]0.0±0.00[0.0,0.3]
ObjectPI55.3±1.05[52.8,57.8]42.0±0.85[39.5,44.5]24.7±0.75[22.6,26.9]8.0±0.67[6.7,9.5]
FAST50.3±1.03[47.8,52.9]41.0±0.88[38.5,43.5]35.3±1.05[33.0,37.8]12.3±0.58[10.8,14.1]
GR00T47.3±1.03[44.8,49.9]26.3±0.97[24.2,28.6]30.3±1.05[28.1,32.7]0.7±0.24[0.4,1.2]
OFT55.3±0.82[52.8,57.8]43.0±0.97[40.5,45.5]25.0±0.75[22.9,27.3]9.0±0.47[7.7,10.6]
Wan.8.3±0.67[7.0,9.8]32.7±1.11[30.3,35.1]20.0±0.94[18.1,22.1]0.0±0.00[0.0,0.3]
Cos.10.0±0.67[8.6,11.6]25.7±0.97[23.5,27.9]9.3±0.71[8.0,10.9]0.0±0.00[0.0,0.3]
GoalPI49.7±1.11[47.1,52.2]23.0±0.67[20.9,25.2]5.0±0.47[4.0,6.2]2.0±0.24[1.4,2.8]
FAST56.0±0.91[53.5,58.5]26.7±0.88[24.5,29.0]5.3±0.41[4.3,6.6]0.0±0.00[0.0,0.3]
GR00T52.3±1.11[49.8,54.9]42.0±0.94[39.5,44.5]5.3±0.41[4.3,6.6]0.0±0.00[0.0,0.3]
OFT57.0±0.82[54.5,59.5]30.3±1.00[28.1,32.7]9.0±0.62[7.7,10.6]3.3±0.41[2.5,4.4]
Wan.34.7±0.91[32.3,37.1]21.3±0.85[19.3,23.5]3.3±0.33[2.5,4.4]2.0±0.33[1.4,2.8]
Cos.31.3±0.94[29.0,33.7]20.0±0.91[18.1,22.1]6.7±0.58[5.5,8.0]0.0±0.00[0.0,0.3]
LIBERO-100PI40.0±0.78[37.5,42.5]20.3±0.97[18.4,22.4]9.7±0.58[8.3,11.3]0.0±0.00[0.0,0.3]
FAST24.7±0.82[22.6,26.9]11.0±0.53[9.5,12.7]17.3±0.91[15.5,19.3]0.0±0.00[0.0,0.3]
GR00T15.3±0.62[13.6,17.2]20.0±0.94[18.1,22.1]18.3±0.75[16.5,20.4]0.0±0.00[0.0,0.3]
OFT38.7±0.94[36.2,41.2]20.3±0.88[18.4,22.4]9.0±0.53[7.7,10.6]0.0±0.00[0.0,0.3]
Wan.9.7±0.58[8.3,11.3]8.7±0.62[7.3,10.2]15.0±0.78[13.3,16.9]0.0±0.00[0.0,0.3]
Cos.12.7±0.82[11.1,14.4]10.7±0.53[9.2,12.3]11.0±0.62[9.5,12.7]0.0±0.00[0.0,0.3]

Q2. Potential bias from overlapping failure generators and evaluation models
Our benchmark pools failure scenarios generated by multiple models. Each evaluated model is tested on both its own generated failures and those generated by other models, reducing dependence on model-specific failure distributions. Furthermore, as shown in our response to Weakness 4, we evaluated additional models not involved in failure generation, and our key findings remained consistent. These results suggest that our findings are not driven solely by the overlap between failure generators and evaluated models.

Q3. Clarification and Analysis of Evaluation Metrics
Thanks. Appendix G.5 provides detailed definitions and interpretations of all three metrics. Specifically, RSR measures task completion from failure states, RD quantifies relative performance degradation after failure, and RC captures recovery consistency across tasks. We also detail the aggregation procedures at the episode, scenario, and task levels, and discuss the low-baseline limitation of RD.

AuthorsOpenReview ↗

Thank You for Your Positive Assessment of Our Work

Dear Reviewer SDvW,

Thank you very much for your positive assessment of our work. As the response period draws to a close, we would appreciate any feedback on whether our responses have addressed your concerns. If any questions remain, we would be happy to provide further clarification.

Thank you again for your time and thoughtful feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

LIBERO-Recover: Key Contributions and Rebuttal Updates

Dear Area Chair, 

Thank you for overseeing the review process. We briefly summarize our contributions and rebuttal updates.  Core contribution. LIBERO-Recover introduces a controlled protocol for evaluating whether policies can recover from execution-induced failures and complete the original task. Unlike predefined perturbation tests, it includes disrupted object states and environmental obstructions that require corrective actions. Shared failure states enable consistent comparisons, while four recovery levels distinguish action correction from state restoration.  Additional evidence during rebuttal: 

  • Broader validation: Added FastWAM, DM0.5, VLA-JEPA, and G0.5, confirming that high LIBERO success does not imply strong recovery.
  • Stronger statistical support: Expanded L3/L4 scenarios and added five-seed evaluations with standard deviations and 95% confidence intervals.
  • Recovery analysis: Added temporal-context and mixed-data training experiments, showing modest improvements.
  • Data quality and practical relevance: Rechecked 200 scenarios with full annotation agreement and added real-robot examples of corresponding failure modes.  We also clarified the protocol’s scope, its distinction from existing robustness benchmarks, and implementation details. These additions reinforce the central finding that post-failure recovery remains a substantial challenge despite strong standard benchmark performance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.