acceptodds
Under review as a conference paper at ICLR 2027

LIBERO-Pro: A Benchmark for Evaluating the Robustness of Embodied Foundation Models

Abstract

LIBERO has become a de facto standard benchmark for evaluating embodied intelligence. However, high success rates on LIBERO do not necessarily reflect robust embodied capabilities, as models may achieve strong performance by exploiting benchmark-specific patterns. In this work, we introduce LIBERO-Pro, a comprehensive benchmark for systematically evaluating the robustness of embodied foundation models. LIBERO-Pro defines 22 static perturbations and 20 runtime interventions, resulting in 1,648 validated perturbation variants. These perturbations cover changes in scene configurations, object properties, visual observations, robot states, temporal conditions, task instructions, and runtime execution states. Each perturbation is carefully designed and undergoes six rounds of human verification to ensure physical plausibility and task validity. We evaluate 11 representative robot policies spanning vision-language-action models, world-action models, and robustness-oriented policies. Our results reveal substantial robustness gaps across all evaluated paradigms: under runtime distribution shifts, the evaluated models suffer an average performance drop of 25.9 percentage points, with individual drops reaching 32.5 points. These findings demonstrate that strong nominal benchmark performance does not necessarily translate into robustness under distribution shifts, highlighting the need for robustness-oriented evaluation of embodied foundation models.

Primary area:
datasets and benchmarks
Keywords:
Embodied AI, Sim2Real Evaluation, Robot Manipulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8wkqOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LIBERO-Pro, a robustness benchmark for embodied foundation models built upon LIBERO. It extends standard evaluation with 22 static distribution shifts and 20 runtime interventions, covering changes in environment, observations, robot states, execution conditions, and language. The authors evaluate 11 representative policies and further study feedback utilization, replanning frequency, recovery, compound perturbations, and robustness transfer. The paper shows that high nominal LIBERO performance does not necessarily imply robustness under distribution shifts, and that different models exhibit substantially different robustness profiles.

Critical strengths
  1. Large-scale and carefully constructed evaluation. The benchmark covers a broad range of perturbations and evaluates 11 representative policies. The authors also conduct extensive manual verification of the perturbed task instances.

  2. Runtime interventions are potentially useful. Disturbances introduced during task execution, such as object relocation, sensing failures, latency, and action-execution faults, test capabilities that are not fully captured by standard static OOD evaluation.

Critical weaknesses
  1. Limited conceptual novelty over existing robustness benchmarks. A substantial part of the benchmark (including changes to objects, scenes, cameras, robot states, observations, and language) overlaps with prior robustness evaluation, most directly LIBERO-Plus (CVPR 2026), which has already demonstrated that strong nominal LIBERO performance can degrade significantly under systematic distribution shifts. The paper itself identifies runtime interventions as the main distinction from LIBERO-Plus. Therefore, the key novelty largely reduces to introducing perturbations during execution. While useful, I am not convinced that this alone constitutes a sufficiently strong conceptual advance for a new benchmark paper, particularly because many of the main conclusions like nominal success does not imply robustness and robustness varies substantially across perturbation categories are already expected from prior robustness studies.

  2. The experiments do not convincingly establish that runtime robustness measures a fundamentally distinct capability. The paper interprets differences between matched static and runtime perturbations as evidence for online adaptation. However, static and runtime settings are not clearly controlled for perturbation duration, trigger time, remaining task horizon, or effective difficulty. Thus, the current results show that static and runtime evaluation behave differently, but do not cleanly establish that the difference specifically reflects closed-loop adaptation.

  3. The evaluation diversity is limited despite the large number of perturbation instances. The main evaluation uses only eight held-out base tasks, two from each LIBERO suite, selected using one task split. Since many perturbation instances share the same underlying tasks, it is unclear how stable the reported rankings and conclusions are to task selection. Multiple splits or task-level uncertainty estimates would substantially strengthen the benchmark.

Questions for authors
  1. Can the authors more clearly articulate what new scientific capability is measured by runtime interventions that cannot be captured by existing static robustness benchmarks?

  2. Can the authors provide a matched static/runtime experiment controlling perturbation magnitude, duration, trigger point, and remaining task horizon?

  3. How stable are the main model rankings across different choices of held-out LIBERO tasks? Are multiple splits or confidence intervals available?

  4. In RQ3, the evaluation appears to hold out tasks while using semantically matched perturbation families during training. Would “cross-task robustness transfer” be a more accurate description than generalization to unseen perturbations?

6 comments
AuthorsOpenReview ↗

Author Response to Q1–Q2

Response to Reviewer 8wkq

We sincerely thank you for your valuable suggestions. Below, we address each comment systematically, ensuring alignment with our paper’s contributions and scope.

Q1. LIBERO-Plus Comparison and the Value of Runtime Evaluation

A1.

We thank the reviewer for this suggestion. The added value of LIBERO-Pro can be explained from four aspects.

1. Necessity of Runtime Evaluation.

Runtime perturbations are important reliability risks in real-world robot deployment. Previous real-robot studies provide direct evidence:

  • CoRe (Zhang et al., 2026) reported that moving objects during execution reduced the grasp success of π0.5 from 19/20 to 4/20.
  • RTC (Black et al., 2025) showed that adding 100 or 200 ms of inference delay caused severe oscillations in two temporal ensembling baselines and triggered robot safety stops.

These results show that runtime perturbations are practical reliability risks that should be evaluated, rather than artificial challenges introduced by benchmarks.

2. Evaluation of Additional Capabilities.

Static evaluation mainly measures whether a model can complete tasks under changed conditions. Runtime evaluation further examines how a model responds to unexpected changes and failures during execution. In particular, we evaluate two additional capabilities:

  • Action-level Adaptation: Whether a model can adjust its actions and continue the task when the target position or motion path changes during execution.
  • State-level Recovery: Whether a model can recover from failures, such as a failed grasp or a dropped object, and then complete the task.

3. Differences from LIBERO-Plus.

Both LIBERO-Pro and LIBERO-Plus evaluate robustness under distribution shifts, but they differ in their evaluation focus and perturbation coverage.

  • Runtime interventions: We introduce 20 types of runtime interventions, including target relocation, dynamic occlusion, execution errors, and grasp failures. These interventions allow us to evaluate adaptation and recovery capabilities beyond static robustness.
  • Broader static coverage: We also introduce additional static perturbations. For example, the average success rates under S03, S05, and S20 are only 23.5%, 21.9%, and 15.6%, respectively, revealing robust weaknesses not fully covered by existing benchmarks.
  • Complementary model rankings: The two benchmarks produce different robustness rankings, showing that they capture different aspects of model robustness.

4. Scientific Value Beyond Success Rates.

While both benchmarks reveal performance drops under perturbations, they examine different capability gaps. LIBERO-Plus mainly focuses on generalization under distribution shifts, while our runtime evaluation additionally examines action adaptation and failure recovery during execution.

Furthermore, our diagnostic experiments on feedback utilization, action update frequency, and recovery interventions help identify factors behind runtime failures and provide insights into recovery mechanisms beyond the overall success rate.

We will further clarify these differences and the added evaluation value of LIBERO-Pro in the revised manuscript.

Q2. Static–Runtime Comparison under Matched Perturbation Settings

A2.

Thank you for the suggestion. Static and runtime perturbations are inherently different: static perturbations occur before execution, whereas runtime perturbations occur during execution. Therefore, their durations, trigger points, and remaining task horizons cannot be fully matched.

Perturbation magnitude is a parameter we can directly match. To address the reviewer's concern, we conducted additional experiments with matched perturbation settings, covering seven perturbation pairs, three models, and eight tasks, with 6,598 valid evaluation episodes. For example, S01/D01 used identical target translations (8.28–12.81 cm) and rotations (16–18.3°), while S02/D02 used identical receiver translations (8.75–15 cm) and rotations (13.8–17.2°). The complete configurations will be provided in revised manuscript.

The table reports Static → Runtime success rates (%), averaged across applicable tasks and suites.

Perturbation Pairπ0.5FastWAMxVLA
S01 / D0163.13 → 41.8839.38 → 19.3815.00 → 11.25
S02 / D0265.63 → 78.7515.00 → 13.750.63 → 8.13
S05 / D059.38 → 13.135.63 → 6.2514.38 → 18.75
S12 / D0477.50 → 90.0085.63 → 90.6353.75 → 95.63
S15 / D070.63 → 10.000.63 → 7.5020.63 → 24.38
S17 / D1980.00 → 50.9580.63 → 57.5093.13 → 98.75
S21 / D2090.63 → 91.2589.38 → 92.5093.13 → 97.50

This further reinforces our argument in A1: runtime evaluation additionally probes execution-time capabilities, including Action-level Adaptation to unexpected changes and State-level Recovery from execution failures.

We will clarify this distinction in the revised manuscript, without attributing the observed differences solely to closed-loop adaptation.

AuthorsOpenReview ↗

Author Response to Q3-Q4

Q3. Task Coverage and Ranking Stability

A3.

Our original evaluation uses a stratified task selection strategy. We select two held-out tasks from each of the four LIBERO suites (Spatial, Object, Goal, and Long) and evaluate all 42 perturbation types on each task. This design balances evaluation cost (8 tasks × 42 perturbations × 30 rollouts) and ranking stability.

To further address the reviewer's concern, we conducted additional experiments with 5 models on the remaining 32 LIBERO tasks, covering all 42 perturbation types and 1,317 valid task–perturbation combinations.

The following table compares the results of four representative models on the original 8 tasks and the additional 32 tasks.

ModelOriginal 8 Tasks SRRankAdditional 32 Tasks SRRank
π0.570.14%170.87%1
FastWAM62.72%258.53%2
UniVLA56.76%356.69%3
OpenVLA35.11%437.65%4

The model rankings are identical across the two task sets, with Spearman's ρ = 1.0 and Kendall's τ = 1.0.

We further examined ranking stability across different task selections. From the additional 32 tasks, we exhaustively evaluated 614,656 possible 8-task subsets, each containing two tasks from each LIBERO suite. The results show:

  • Mean Spearman's ρ = 0.953.
  • Pairwise Ranking Agreement = 96.06%.
  • Leave-One-Task-Out: The complete ranking remains unchanged in all 32 cases (32/32).

These results provide direct quantitative evidence that our model rankings are highly stable across different task selections, supporting the reliability of the original evaluation.

Q4. Scope of RQ3: Cross-Task Robustness Transfer

A4.

We thank the reviewer for pointing out this inconsistency. We agree that “cross-task robustness transfer” more accurately describes the objective of RQ3. The phrase “generalization to unseen perturbations” in Section 4.1 was an editorial error, not a discrepancy in our experimental design.

As explicitly stated in Section 4.5, “We ask whether robustness can be learned from limited demonstrations and transferred to unseen tasks.” Accordingly, our experiments train policies on a limited set of training tasks and evaluate them on disjoint held-out tasks, while retaining semantically matched perturbation families (e.g., object pose, receiver pose, path obstacles, camera calibration, and robot-state variations) across training and evaluation.

Thus, our experimental design and analysis consistently investigate whether robustness learned from limited demonstrations transfers to unseen tasks. The results in Table 6 demonstrate that such cross-task robustness transfer is achievable, with our fine-tuned policy attaining 70.1% average success rate, compared with 47.9% for the original π0.5, an improvement of 22.2 percentage points.

We will correct the RQ3 description in Section 4.1 to “Can robustness be learned and transferred to unseen tasks?” and consistently use cross-task robustness transfer throughout the revised manuscript. This correction aligns with the existing experimental protocol and does not affect any reported results or conclusions.

AuthorsOpenReview ↗

A Friendly Follow-up on Our Rebuttal Responses

Thank you again for your valuable time and consideration. As the rebuttal period is approaching its deadline, we would greatly appreciate knowing whether our responses have adequately addressed your concerns. Any further feedback or suggestions from you would be invaluable in helping us improve our paper. We sincerely look forward to hearing from you.

Best regards,

Reviewer_nvGTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces LIBERO-Pro, a benchmark extending LIBERO with 22 static and 20 runtime perturbations to evaluate the robustness of embodied foundation models. Experiments on 11 robot policies reveal substantial performance degradation under distribution shifts. Further analyses investigate runtime adaptation, recovery, compositional robustness, and the generalizability of learned robustness.

Critical strengths
  1. The benchmark systematically introduces a comprehensive set of static and runtime perturbations, covering multiple dimensions of robustness evaluation.

  2. Extensive evaluations of 11 state-of-the-art embodied models, spanning VLA, WAM, and robustness-oriented policies, provide comprehensive empirical results.

  3. The dataset of 39390 training trajectories, collected from human-validated perturbation scenarios, provides a resource for future robustness research.

Critical weaknesses
  1. The lack of real robot experiments limits the validity of the findings in real deployment scenarios.

  2. The perturbation categorization is inconsistent. For example, D13 modifies the timing of arm and gripper commands but is classified as Observation, while S15 and D07 both involve camera pose changes but are assigned to different categories. These inconsistencies undermine the validity of comparisons across categories.

  3. RQ1-C lacks detailed implementation protocols for Hold, Retry, and Reset. In particular, the duration and termination conditions of Hold, as well as the trigger timing of Retry and Reset, are not clearly specified. These details are crucial for ensuring the validity and soundness of the experimental results.

  4. In RQ1-A, freezing observations before the perturbation severely compromises task feasibility, making the comparison with live feedback neither reasonable nor fair and the resulting conclusions unreliable.

  5. RQ3 in Section 4.1 overclaims generalization to unseen perturbations, whereas Section 4.5 only evaluates generalization to unseen tasks under previously encountered perturbation types.

Questions for authors
  1. In RQ1-C, how is the duration of Hold determined for perturbations that cause persistent state changes, such as D01, D02, and D18? How does Hold differ from simply bypassing the perturbation? If the perturbation is effectively avoided, isn't the improved success rate an expected outcome rather than evidence of recovery?

  2. In RQ1-C, when are Retry and Reset triggered, and how is Reset implemented? Reset substantially improves performance under perturbations affecting robot states, such as D19 (+18.0 pp for π0.5 and +21.2 pp for Anchor-Align). Averaging these results with unrelated perturbations obscures its effectiveness, making the overall average uninformative.

  3. In RQ1-B, could the authors provide nominal success rates under different (K/H) settings? Changing the execution horizon may affect normal task performance, making it difficult to distinguish changes in general control performance from robustness to perturbations.

8 comments
AuthorsOpenReview ↗

Author Response to Q1–Q2

Response to Reviewer nvGT

We sincerely thank you for your valuable suggestions. Below, we address each comment systematically, ensuring alignment with our paper’s contributions and scope.

Q1. Real-World Applicability and Simulation Limitations

A1.

We agree that this is an important question. A large-scale physical robot evaluation is not feasible within the rebuttal period, and we leave it as future work.

Like LIBERO and LIBERO-Plus, LIBERO-Pro focuses on simulation-based evaluation. Prior work, such as SIMPLER (Li et al., 2024), has shown that policy rankings in carefully designed simulations can correlate with real-world rankings.

ALL perturbations in LIBERO-Pro can be reproduced on physical robots, such as moving objects by hand, changing camera poses, switching lights, or placing obstacles in the robot's path.

We will state this limitation and describe the possible real-world evaluation protocol in the revised manuscript.

Q2. Perturbation Classification Consistency

A2.

We thank the reviewer for identifying inconsistencies in our perturbation categorization. We agree that the original assignments of D07 and D13 were insufficiently precise. We will reclassify D07 under Observation and D13 under Execution, and consistently apply a unified categorization criterion across all perturbation types.

Following these corrections, we have recomputed the category-level statistics in Tables 2 and 3. Although some category-level averages change, the overall success rates and model rankings remain unchanged. Furthermore, the overall Feedback Utilization Gain remains identical for all five representative policies.

These results confirm that categorization corrections do not affect our main findings regarding heterogeneous robustness across perturbation conditions. We will update the corresponding tables and revise the affected category-level analyses in the manuscript to ensure consistency and accuracy.

AuthorsOpenReview ↗

Author Response to Q3-Q4

Q3. Recovery Intervention Protocols and Effects

A3.

We thank the reviewer for highlighting the need for clearer implementation details. We clarify the intervention protocols and their intended interpretation below.

1. Hold: Duration and Persistent Perturbations. Hold is activated after the first perturbation-affected environment transition, with a predefined exposure window of W=max⁡(1,round⁡(TˉBASE/8))W=\max(1,\operatorname{round}(\bar{T}_{\mathrm{BASE}}/8)), where TˉBASE\bar{T}_{\mathrm{BASE}} is the average nominal execution length for the corresponding policy–task pair. During Hold, zero-action commands are issued without new policy inference. The simulator continues advancing, and all waiting steps consume the original episode budget. Upon window expiration, only the perturbation's directly imposed modifications are reversed. For persistent changes (e.g., D01, D02, D18), subsequent physical consequences are preserved, meaning the scene is not restored to its pre-perturbation state. Normal execution then resumes with refreshed observations.

2. Hold vs. Perturbation Bypass. Unlike bypassing, Hold allows the perturbation to take effect before suspending execution, retaining residual physical consequences that may require subsequent adaptation. Nevertheless, we acknowledge that Hold combines action suspension and bounded perturbation exposure. However, successful task completion after Hold can still require the policy to accommodate residual state deviations and adjust its subsequent behavior to the modified task state.

3. Retry and Reset: Trigger Timing. Retry starts at episode step 0 and replays a nominally successful action sequence open-loop, with runtime perturbations triggered as usual but without policy replanning. Reset is applied once, immediately after the first perturbation-affected transition, restoring only the robot's initial joint configurations and velocities while preserving object states and environmental changes. Reset does not advance simulation time or extend the episode budget.

We will explicitly document the trigger timing, duration, termination conditions, restoration scope, and budget accounting of all three interventions in the revised manuscript, together with the clarifications of Hold diagnostic interpretation.

Q4. Live–Freeze Feedback Evaluation Protocol

A4.

We thank the reviewer for raising this concern. We agree that prolonged observation freezing could impair task execution and confound the comparison. However, our protocol employs a short, controlled visual-freezing window rather than freezing observations throughout the episode.

1. Controlled Freezing and Task Feasibility. Immediately before the first perturbation, we cache images from both cameras and replay them for a fixed window of W=max⁡(1,round(TˉBASE/8))W=\max(1,\mathrm{round}(\bar{T}_{\mathrm{BASE}}/8)) control steps (approximately 0.5–3.25 seconds), after which live visual feedback resumes. Only camera observations are frozen; proprioception, language inputs, physical simulation, and action execution remain active. Task execution therefore remains possible, as supported by Table 11: for example, π0.5\pi_{0.5} achieves 85.6% success under frozen feedback in D04.

2. Accounting for Freezing-Induced Degradation. We explicitly account for the impact of freezing under nominal execution through our difference-in-differences metric:

Gfeedback=(S1,live−S1,freeze)−(S0,live−S0,freeze).G_{\mathrm{feedback}}=(S_{1,\mathrm{live}}-S_{1,\mathrm{freeze}})-(S_{0,\mathrm{live}}-S_{0,\mathrm{freeze}}).

By subtracting the nominal live–freeze difference, GfeedbackG_{\mathrm{feedback}} measures how the benefit of updated visual feedback changes under perturbations relative to nominal execution, rather than attributing all freezing-induced degradation to perturbation handling.

We acknowledge that this design does not eliminate all potential interactions between observation freezing and perturbation difficulty. We will clarify the freezing duration, preserved observation modalities, and interpretation boundaries in the revised manuscript, restricting our conclusions to the relative feedback benefit under the controlled evaluation conditions.

AuthorsOpenReview ↗

Author Response to Q5-Q6

Q5. Scope of Cross-Task Robustness Transfer

A5.

We thank the reviewer for pointing out this ambiguity. We agree that the phrase "generalized to unseen perturbations" in RQ3 (Section 4.1) may suggest generalization to entirely unseen perturbation types, which extends beyond the scope of our current experiments.

Our intended objective in RQ3 is to investigate cross-task transfer of learned perturbation robustness. Specifically, in Section 4.5, we fine-tune π0.5 using 800 demonstrations from eight training tasks and evaluate it on disjoint tasks under five previously encountered perturbation families. As shown in Table 6, the fine-tuned policy achieves an average success rate of 70.1% on unseen tasks, outperforming the original π0.5 baseline (47.9%) by 22.2 percentage points. These results demonstrate that robustness learned from limited demonstrations can transfer across tasks within the evaluated perturbation families.

To eliminate this ambiguity, we will make the following revision:

  1. Section 4.1 (RQ3): Replace "Can robustness be learned and generalized to unseen perturbations?" with "Can perturbation robustness be learned from limited demonstrations and transferred to unseen tasks?"

Q6. Nominal Baselines Across Action Horizons

A6.

We agree that varying the execution horizon may affect nominal task performance, making it necessary to distinguish changes in general control capability from robustness to runtime perturbations.

1. Nominal success rates under different execution horizons.

We conducted additional evaluations under nominal conditions using the same four execution ratios as in RQ1-B. We fixed each policy's prediction horizon HH and varied the executed action length KK. Each configuration was evaluated on eight held-out tasks with 10 seeds, totaling 1,600 additional episodes.

Table R1: Nominal success rates (%) under different execution ratios K/HK/H.

ModelH1/81/41/21
OpenVLA-OFT876.2580.0088.7591.25
π0.51085.0091.2591.2596.25
Anchor-Align861.2566.2591.2590.00
FastWAM3287.5095.0090.0080.00
Cosmos Policy1686.2595.0092.5098.75
Average—79.2585.5090.7591.25

Each entry averages 80 nominal episodes. Executed chunk lengths are rounded to integers.

2. Quantifying perturbation-induced performance degradation.

To distinguish nominal performance changes from perturbation-induced degradation, we define the robustness drop for policy mm, perturbation pp, and execution ratio r=K/Hr=K/H:

Dm,p(r)=Sm,nom(r)−Sm,p(r),D_{m,p}(r)=S_{m,\mathrm{nom}}(r)-S_{m,p}(r),

where Sm,nomS_{m,\mathrm{nom}} and Sm,pS_{m,p} denote nominal and perturbed success rates, respectively. We further define the baseline-adjusted gain relative to K/H=1K/H=1:

Gm,p(r)=Dm,p(1)−Dm,p(r).G_{m,p}(r)=D_{m,p}(1)-D_{m,p}(r).

A smaller drop or positive gain indicates reduced perturbation-induced performance degradation relative to the corresponding nominal baseline.

3. Overall analysis across five policies and five runtime perturbations.

We compute these metrics across the five policies and five perturbations (D01, D02, D03, D05, and D18), with equal weighting.

Table R2: Average nominal performance, perturbed performance, and robustness degradation under different execution ratios.

K/HNominal SR (%) ↑Perturbed SR (%) ↑Robustness Drop (pp) ↓Gain (pp) ↑
1/879.2519.2060.05+9.62
1/485.5023.0062.50+7.18
1/290.7524.0566.70+2.97
191.2521.5869.670.00

Robustness Drop = Nominal SR − Perturbed SR. Gain = Drop at K/H=1K/H=1 − Drop at the corresponding execution ratio. All differences are measured in percentage points (pp).

The results yield two key observations.

First, intermediate replanning improves perturbed performance beyond nominal performance changes. Reducing K/HK/H from 1 to 1/2 increases perturbed SR from 21.58% to 24.05% (+2.47 pp), while nominal SR slightly decreases from 91.25% to 90.75% (−0.50 pp), yielding a baseline-adjusted gain of +2.97 pp.

Second, excessively frequent replanning does not necessarily improve overall performance. Although K/H=1/8K/H=1/8 produces the smallest robustness drop (60.05 pp), nominal SR falls to 79.25% and perturbed SR to 19.20%. Thus, its smaller drop should not be interpreted as superior robustness without considering the substantial degradation in nominal control.

We will incorporate these results into RQ1-B and clarify that execution horizon affects both nominal control and runtime robustness, with intermediate replanning offering a favorable trade-off in the aggregate. We thank the reviewer for this constructive suggestion.

Reviewer_E8s9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces LIBERO-Pro, a robustness benchmark extending LIBERO with 22 static perturbations and 20 runtime interventions. It contains 1,648 validated task–perturbation instances and 39,390 training trajectories. The authors evaluate 11 manipulation policies and find substantial performance degradation despite strong nominal success rates. Further experiments examine observation feedback, action execution horizons, recovery interventions, perturbation composition, and robustness transfer through fine-tuning. The benchmark addresses an important evaluation gap, particularly for disturbances occurring during execution. My main concerns are that several diagnostic conclusions exceed what the experiments establish, and key protocol details remain unclear.

Critical strengths
  1. Relevant problem and a useful focus on runtime robustness. High success under familiar conditions does not establish reliable behavior when conditions change during execution. The runtime interventions cover target relocation, sensor failures, action-stream disruptions, and recoverable grasp failures. This extends the evaluation beyond static distribution shifts and is the strongest aspect of the paper.
  2. Substantial benchmark construction and empirical coverage. The perturbation suite covers scene configuration, perception, robot state, execution, and language. Evaluating 11 policies across different model families provides a useful reference for future work. The six rounds of human verification also represent substantial effort to check that interventions are implemented correctly and tasks remain feasible.
  3. Experiments that investigate failure mechanisms. The paper goes beyond a leaderboard by studying feedback, execution horizons, recovery, and compound perturbations. The severe degradation under certain compositions is worth investigating. The fine-tuning study also offers preliminary evidence that the benchmark can support robustness development as well as evaluation.
Critical weaknesses

Weaknesses

  1. Evaluation diversity and perturbation calibration. The main evaluation uses only eight underlying tasks, several with closely related pick-and-place structures. The large number of perturbed instances therefore does not establish equally broad behavioral coverage. Perturbation magnitudes, durations, and trigger conditions are described qualitatively, without systematic severity sweeps or reference-controller performance. Matched static and runtime interventions can also differ in duration and exposure, so their comparison does not isolate online adaptation. I would like to see broader task-level validation, severity-dependent results, and a more explicit comparison with LIBERO-Plus.
  2. The feedback metric is overinterpreted. A negative difference-in-differences score, G_feedback, means that feedback provides less benefit under perturbation than under nominal execution. It does not mean that updated observations fail to improve perturbed performance. For example, Table 11 reports that OpenVLA-OFT under D11 improves from 45.0% with frozen feedback to 84.6% with live feedback, while G_feedback remains negative. This is a substantial benefit from updated observations. The discussion should distinguish direct feedback benefit from the change in that benefit relative to nominal execution. Identifying observation-to-action conversion as the main bottleneck would require additional evidence.
  3. Recovery and composition experiments need clearer controls. Hold is described as waiting until a perturbation ends, but several interventions involve persistent scene changes or injected outcome states. It is unclear what ends in these cases, whether any scene state is restored, and whether waiting consumes the episode budget. These details are essential for interpreting the large Hold gains. Replaying an unperturbed action sequence is also inherently vulnerable to changed object states, so its failure says relatively little about the mechanisms that prevent recovery. For composition experiments, constituent and composed cases should be matched in tasks, initial states, and perturbation parameters. Comparing a composition only with its weakest constituent does not distinguish accumulated failure from an adverse interaction between perturbations.
  4. Transfer claims and reproducibility need stronger support. Section 4.5 tests familiar perturbation families on held-out tasks. This supports cross-task transfer, rather than generalization to unseen perturbation types. Appendix C.4 also notes that the evaluated checkpoints may have encountered the evaluation tasks, so “unseen” should be qualified relative to the additional fine-tuning stage. The training description should clarify whether separate models are trained for each perturbation and how static and dynamic demonstrations are combined. A baseline fine-tuned on an equal amount of nominal data is needed to isolate the value of robustness data. The manuscript should also report evaluation episode counts, uncertainty estimates, and training-seed variability; otherwise, smaller differences and ranking reversals are difficult to assess.
Questions for authors

Clarification on feedback interpretation: A negative feedback gain does not necessarily mean that updated observations are ineffective. For example, Table 11 shows that OpenVLA-OFT under D11 improves from 45.0% with frozen feedback to 84.6% with live feedback, despite a negative score. What evidence supports identifying ineffective feedback utilization as the main bottleneck, beyond a reduction in feedback benefit relative to nominal execution?

Clarification on recovery interventions: How is Hold implemented for persistent interventions such as target relocation and injected drop states? What determines when waiting ends? Are any scene states restored, do observations continue updating, and does waiting consume the episode budget? These details are necessary to interpret the reported recovery gains.

Clarification on composition controls: Are composed perturbations and their constituent single perturbations evaluated using the same tasks, initial states, perturbation magnitudes, and runtime triggers? Are composed instances independently checked for feasibility? This is particularly relevant to the near-universal collapse on P02, P06, and T03.

Clarification on robustness transfer: In Table 6, is one model trained jointly across perturbations, or is a separate model trained for each perturbation or family? What is the total demonstration budget per model, and which evaluation tasks or perturbations were previously seen by the starting checkpoint? Is there a comparison against fine-tuning on an equal amount of nominal data to isolate the contribution of robustness demonstrations?

Clarification on statistical reliability: How many evaluation episodes and independent seeds support each reported result, and how are task and perturbation scores aggregated? What are the uncertainty estimates for the principal ranking reversals and transfer gains? Their statistical reliability affects the strength of the conclusions.

6 comments
AuthorsOpenReview ↗

Author Response to Q1–Q2

Response to Reviewer E8s9

We sincerely thank you for your positive feedback and valuable suggestions. Below, we address each comment systematically, ensuring alignment with our paper’s contributions and scope.

Q1. Task Coverage and Ranking Stability

A1.

In our original evaluation, we selected two held-out tasks from each of the four LIBERO suites, giving eight tasks in total. We tested 42 perturbation types on each task where applicable, with 30 rollouts per setting. This design balances evaluation cost and task coverage.

To further address this concern, we evaluated five models on the remaining 32 LIBERO tasks, covering all 42 perturbation types and 1,317 valid task–perturbation pairs. Together with the original evaluation, our experiments now cover all 40 LIBERO tasks.

The table below compares the results of four representative models on the original 8 tasks and the additional 32 tasks.

Table 1.

ModelOriginal 8 Tasks SRRankAdditional 32 Tasks SRRank
π0.570.14%170.87%1
FastWAM62.72%258.53%2
UniVLA56.76%356.69%3
OpenVLA35.11%437.65%4

The model rankings are identical across the two task sets (Spearman ρ = 1.0, Kendall τ = 1.0).

We further tested ranking stability using all 614,656 possible 8-task subsets from the additional 32 tasks, with two tasks selected from each LIBERO suite. The results show:

  • Mean Spearman ρ = 0.953, compared with the full 32-task ranking.
  • Pairwise Ranking Agreement = 96.06%.
  • Leave-One-Task-Out: The full ranking remains unchanged in all 32 cases.

These results provide strong evidence that the model rankings are stable across different task selections.

Q2. Perturbation Specifications, Feasibility, and Severity

A2.

We provide the following clarifications and additional evidence.

1. Perturbation Settings and Event Rules.

We will add a detailed perturbation configuration table to Appendix C.2 and C.3. It will specify the parameter values or ranges, sampling rules, trigger conditions, durations, and stopping rules for each perturbation.

2. Reference Solver and Task Feasibility.

We understand the reviewer's concern: Are low success rates caused by model limitations, infeasible tasks, or insufficient evaluation budgets?

To address this question, we provided GPT-6-Astra with ground-truth environment information and used it to generate executable reference programs. These programs successfully completed all valid evaluation instances, confirming that each instance has at least one feasible solution.

We will include Table 2 in the revised manuscript to compare the average execution steps of the reference solver with the evaluation step limits. Our results show that all evaluation budgets exceed the corresponding average reference steps. In addition, our released training data contain successful trajectories under perturbed conditions, providing further evidence of task feasibility.

Table 2. Average execution steps of the reference solver and evaluation step budgets.

LIBERO SuiteAverage StepsMax StepsEvaluation Budget
Spatial169.85205320
Object194.12230280
Goal142.52260300
Long305.41500520

3. Perturbation Severity and Scope.

The main scope of our work is to systematically evaluate model robustness across diverse perturbation mechanisms, including both static shifts and runtime interventions.

In the real world, many perturbations vary continuously, and it is difficult to define a unified severity scale across different perturbation types. For example, changes the tabletop texture, but there is no clear way to rank different textures by severity. Therefore, we select a set of well-defined, reproducible, and feasibility-verified evaluation settings based on perturbation mechanisms that may occur in real-world scenarios.

We agree that finer severity sweeps would provide additional insights and will treat them as an important direction for future research.

AuthorsOpenReview ↗

Author Response to Q3-Q4

Q3. Static vs. Runtime Comparison and Differences from LIBERO-Plus

A3.

  1. Static vs. Runtime Comparison

Static shifts and runtime interventions are two complementary robustness tests. They differ in trigger timing, duration, and exposure. Therefore, their success-rate differences alone cannot provide causal evidence of online adaptation. We will clarify this distinction and revise Finding 2 in Section 4.2 to focus on the observed robustness differences.

  1. Comparison with LIBERO-Plus.

Both LIBERO-Pro and LIBERO-Plus evaluate model robustness under distribution shifts, but they differ in their evaluation focus and perturbation coverage.

  1. Runtime interventions are our main contribution. We introduce 20 runtime interventions to evaluate additional capabilities beyond static robustness:
    1. Action-level Adaptation: Whether a model can adjust its actions when the target or path changes during execution.
    2. State-level Recovery: Whether a model can recover from failures, such as a failed grasp or a dropped object, and continue the task.
  2. We extend static perturbation coverage. For example, the average success rates under S03, S05, and S20 are only 23.5%, 21.9%, and 15.6%, respectively. These results reveal robust weaknesses not fully covered by existing benchmarks.
  3. The two benchmarks produce different model rankings. This suggests that LIBERO-Pro and LIBERO-Plus evaluate complementary aspects of model robustness.

We will add a detailed comparison in the revised manuscript to clarify these differences.

Q4. Direct and Relative Feedback Gains

A4.

We agree that a negative GfeedbackG_{\mathrm{feedback}} indicates a reduction in feedback benefit relative to nominal execution.

Specifically, GfeedbackG_{\mathrm{feedback}} measures the difference between the feedback benefit under perturbed and nominal conditions. As the reviewer correctly points out, OpenVLA-OFT under D11 achieves a substantial direct improvement of +39.6 percentage points with live feedback, despite a negative GfeedbackG_{\mathrm{feedback}} of −9.4 points.

We acknowledge that our original discussion conflated these two quantities and overstated the implications of negative difference-in-difference scores. Our results support observations below:

  1. Updated observations remain beneficial under perturbations. Positive direct feedback gains demonstrate that policies can leverage updated observations to improve task success.
  2. Feedback benefits can diminish under runtime distribution shifts. Predominantly negative GfeedbackG_{\mathrm{feedback}} values indicate their performance benefits are often smaller than those observed under nominal execution. This reveals the ability to benefit from feedback under nominal conditions does not guarantee that the same level of benefit is maintained under runtime distribution shifts.

Accordingly, we will revise the main finding in Section 4.3.

AuthorsOpenReview ↗

Author Response to Q5-Q6

Q5. Recovery Intervention Protocols and Effects

A5.

We clarify the three diagnostic interventions as follows:

  1. Hold. Hold introduces a finite perturbation exposure window of W=max⁡(1,round(TˉBASE/8))W=\max(1,\mathrm{round}(\bar{T}_{\mathrm{BASE}}/8)), where TˉBASE\bar{T}_{\mathrm{BASE}} is the average nominal execution length. During this window, the policy issues zero-action commands without new inference, while the simulator continues advancing. Every waiting step consumes the original episode budget, without horizon extension. Upon window expiration, only the perturbation's directly imposed modifications are reversed, including for persistent relocations and injected drop states; subsequent physical effects are preserved. Thus, Hold does not restore the entire scene to its pre-perturbation state. Updated observations are provided when policy execution resumes.
  2. Reset. Reset restores only the robot arm and gripper to their initial joint configurations, with joint velocities reset to zero. Object states, environmental changes, and ongoing perturbations remain unchanged. Reset incurs no additional environment steps or episode-budget extension.
  3. Retry. Retry performs a single open-loop replay of a previously successful nominal action sequence under the same initial configuration with runtime perturbations enabled, without additional policy inference. We agree that such replay is inherently vulnerable to changed task states. This is precisely its diagnostic purpose: to examine whether previously successful actions remain effective under perturbations. Our result supports an important distinction: robustness requires more than persisting with nominally successful behavior despite disturbances.

We acknowledge that Hold's improvements reflect the combined effects of suspending execution and limiting direct perturbation exposure, rather than the benefit of waiting alone. We will provide the complete implementation settings and clarify these interpretation boundaries in the revised manuscript.

Q6. Composition Controls and Robustness Metrics

A6.

We clarify the evaluation controls, feasibility verification, and interpretation of our composition metric.

1. Matched Evaluation Controls. For each composition and its constituent perturbations, we use identical evaluation tasks, paired initial states, and random seeds. Perturbation parameters, including magnitudes, durations, and runtime triggering rules, are preserved from the corresponding single-perturbation settings. The only intended difference is the coexistence of multiple perturbations.

2. Composition Feasibility Verification. Compositions are not constructed arbitrarily. We manually screen pairwise perturbation combinations for semantic, geometric, sensing, and execution conflicts, excluding incompatible combinations. Selected compositions undergo additional instance-level feasibility checks to ensure that the original task objective remains unchanged and physically achievable. We will provide the compatibility matrix and detailed validation procedures in the revised supplementary material.

3. Interpretation of the Weakest-Constituent Metric. Our metric evaluates whether robustness to individual perturbations is maintained when multiple perturbations coexist. A negative weakest-constituent gap indicates that the composed condition performs worse than every constituent individually. We agree that this metric does not isolate adverse interaction effects from accumulated failures and will explicitly clarify this limitation.

Importantly, our central finding remains valid: the severe performance degradation observed in P02, P06, and T03 demonstrates that strong robustness to individual perturbations does not guarantee robustness to their composition. This directly addresses RQ2's objective of assessing practical policy reliability under simultaneous distribution shifts.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.