- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces LIBERO-Pro, a robustness benchmark for embodied foundation models built upon LIBERO. It extends standard evaluation with 22 static distribution shifts and 20 runtime interventions, covering changes in environment, observations, robot states, execution conditions, and language. The authors evaluate 11 representative policies and further study feedback utilization, replanning frequency, recovery, compound perturbations, and robustness transfer. The paper shows that high nominal LIBERO performance does not necessarily imply robustness under distribution shifts, and that different models exhibit substantially different robustness profiles.
Critical strengths
-
Large-scale and carefully constructed evaluation. The benchmark covers a broad range of perturbations and evaluates 11 representative policies. The authors also conduct extensive manual verification of the perturbed task instances.
-
Runtime interventions are potentially useful. Disturbances introduced during task execution, such as object relocation, sensing failures, latency, and action-execution faults, test capabilities that are not fully captured by standard static OOD evaluation.
Critical weaknesses
-
Limited conceptual novelty over existing robustness benchmarks. A substantial part of the benchmark (including changes to objects, scenes, cameras, robot states, observations, and language) overlaps with prior robustness evaluation, most directly LIBERO-Plus (CVPR 2026), which has already demonstrated that strong nominal LIBERO performance can degrade significantly under systematic distribution shifts. The paper itself identifies runtime interventions as the main distinction from LIBERO-Plus. Therefore, the key novelty largely reduces to introducing perturbations during execution. While useful, I am not convinced that this alone constitutes a sufficiently strong conceptual advance for a new benchmark paper, particularly because many of the main conclusions like nominal success does not imply robustness and robustness varies substantially across perturbation categories are already expected from prior robustness studies.
-
The experiments do not convincingly establish that runtime robustness measures a fundamentally distinct capability. The paper interprets differences between matched static and runtime perturbations as evidence for online adaptation. However, static and runtime settings are not clearly controlled for perturbation duration, trigger time, remaining task horizon, or effective difficulty. Thus, the current results show that static and runtime evaluation behave differently, but do not cleanly establish that the difference specifically reflects closed-loop adaptation.
-
The evaluation diversity is limited despite the large number of perturbation instances. The main evaluation uses only eight held-out base tasks, two from each LIBERO suite, selected using one task split. Since many perturbation instances share the same underlying tasks, it is unclear how stable the reported rankings and conclusions are to task selection. Multiple splits or task-level uncertainty estimates would substantially strengthen the benchmark.
Questions for authors
-
Can the authors more clearly articulate what new scientific capability is measured by runtime interventions that cannot be captured by existing static robustness benchmarks?
-
Can the authors provide a matched static/runtime experiment controlling perturbation magnitude, duration, trigger point, and remaining task horizon?
-
How stable are the main model rankings across different choices of held-out LIBERO tasks? Are multiple splits or confidence intervals available?
-
In RQ3, the evaluation appears to hold out tasks while using semantically matched perturbation families during training. Would “cross-task robustness transfer” be a more accurate description than generalization to unseen perturbations?
6 comments
Author Response to Q3-Q4
Q3. Task Coverage and Ranking Stability
A3.
Our original evaluation uses a stratified task selection strategy. We select two held-out tasks from each of the four LIBERO suites (Spatial, Object, Goal, and Long) and evaluate all 42 perturbation types on each task. This design balances evaluation cost (8 tasks × 42 perturbations × 30 rollouts) and ranking stability.
To further address the reviewer's concern, we conducted additional experiments with 5 models on the remaining 32 LIBERO tasks, covering all 42 perturbation types and 1,317 valid task–perturbation combinations.
The following table compares the results of four representative models on the original 8 tasks and the additional 32 tasks.
| Model | Original 8 Tasks SR | Rank | Additional 32 Tasks SR | Rank |
|---|---|---|---|---|
| π0.5 | 70.14% | 1 | 70.87% | 1 |
| FastWAM | 62.72% | 2 | 58.53% | 2 |
| UniVLA | 56.76% | 3 | 56.69% | 3 |
| OpenVLA | 35.11% | 4 | 37.65% | 4 |
The model rankings are identical across the two task sets, with Spearman's ρ = 1.0 and Kendall's τ = 1.0.
We further examined ranking stability across different task selections. From the additional 32 tasks, we exhaustively evaluated 614,656 possible 8-task subsets, each containing two tasks from each LIBERO suite. The results show:
- Mean Spearman's ρ = 0.953.
- Pairwise Ranking Agreement = 96.06%.
- Leave-One-Task-Out: The complete ranking remains unchanged in all 32 cases (32/32).
These results provide direct quantitative evidence that our model rankings are highly stable across different task selections, supporting the reliability of the original evaluation.
Q4. Scope of RQ3: Cross-Task Robustness Transfer
A4.
We thank the reviewer for pointing out this inconsistency. We agree that “cross-task robustness transfer” more accurately describes the objective of RQ3. The phrase “generalization to unseen perturbations” in Section 4.1 was an editorial error, not a discrepancy in our experimental design.
As explicitly stated in Section 4.5, “We ask whether robustness can be learned from limited demonstrations and transferred to unseen tasks.” Accordingly, our experiments train policies on a limited set of training tasks and evaluate them on disjoint held-out tasks, while retaining semantically matched perturbation families (e.g., object pose, receiver pose, path obstacles, camera calibration, and robot-state variations) across training and evaluation.
Thus, our experimental design and analysis consistently investigate whether robustness learned from limited demonstrations transfers to unseen tasks. The results in Table 6 demonstrate that such cross-task robustness transfer is achievable, with our fine-tuned policy attaining 70.1% average success rate, compared with 47.9% for the original π0.5, an improvement of 22.2 percentage points.
We will correct the RQ3 description in Section 4.1 to “Can robustness be learned and transferred to unseen tasks?” and consistently use cross-task robustness transfer throughout the revised manuscript. This correction aligns with the existing experimental protocol and does not affect any reported results or conclusions.
A Friendly Follow-up on Our Rebuttal Responses
Thank you again for your valuable time and consideration. As the rebuttal period is approaching its deadline, we would greatly appreciate knowing whether our responses have adequately addressed your concerns. Any further feedback or suggestions from you would be invaluable in helping us improve our paper. We sincerely look forward to hearing from you.
Best regards,
Author Response to Q1–Q2
Response to Reviewer 8wkq
We sincerely thank you for your valuable suggestions. Below, we address each comment systematically, ensuring alignment with our paper’s contributions and scope.
Q1. LIBERO-Plus Comparison and the Value of Runtime Evaluation
A1.
We thank the reviewer for this suggestion. The added value of LIBERO-Pro can be explained from four aspects.
1. Necessity of Runtime Evaluation.
Runtime perturbations are important reliability risks in real-world robot deployment. Previous real-robot studies provide direct evidence:
These results show that runtime perturbations are practical reliability risks that should be evaluated, rather than artificial challenges introduced by benchmarks.
2. Evaluation of Additional Capabilities.
Static evaluation mainly measures whether a model can complete tasks under changed conditions. Runtime evaluation further examines how a model responds to unexpected changes and failures during execution. In particular, we evaluate two additional capabilities:
3. Differences from LIBERO-Plus.
Both LIBERO-Pro and LIBERO-Plus evaluate robustness under distribution shifts, but they differ in their evaluation focus and perturbation coverage.
4. Scientific Value Beyond Success Rates.
While both benchmarks reveal performance drops under perturbations, they examine different capability gaps. LIBERO-Plus mainly focuses on generalization under distribution shifts, while our runtime evaluation additionally examines action adaptation and failure recovery during execution.
Furthermore, our diagnostic experiments on feedback utilization, action update frequency, and recovery interventions help identify factors behind runtime failures and provide insights into recovery mechanisms beyond the overall success rate.
We will further clarify these differences and the added evaluation value of LIBERO-Pro in the revised manuscript.
Q2. Static–Runtime Comparison under Matched Perturbation Settings
A2.
Thank you for the suggestion. Static and runtime perturbations are inherently different: static perturbations occur before execution, whereas runtime perturbations occur during execution. Therefore, their durations, trigger points, and remaining task horizons cannot be fully matched.
Perturbation magnitude is a parameter we can directly match. To address the reviewer's concern, we conducted additional experiments with matched perturbation settings, covering seven perturbation pairs, three models, and eight tasks, with 6,598 valid evaluation episodes. For example, S01/D01 used identical target translations (8.28–12.81 cm) and rotations (16–18.3°), while S02/D02 used identical receiver translations (8.75–15 cm) and rotations (13.8–17.2°). The complete configurations will be provided in revised manuscript.
The table reports Static → Runtime success rates (%), averaged across applicable tasks and suites.
This further reinforces our argument in A1: runtime evaluation additionally probes execution-time capabilities, including Action-level Adaptation to unexpected changes and State-level Recovery from execution failures.
We will clarify this distinction in the revised manuscript, without attributing the observed differences solely to closed-loop adaptation.