- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces LIBERO-MOR, a robustness benchmark built on four LIBERO suites. It evaluates 22 perturbations across vision, language, proprioception, action execution, and environment, using three severity levels and five seeds for each of 40 tasks. Perturbed episodes are paired with clean references through a shared evaluation manifest. The authors report 93,800 rollouts across seven VLA systems and analyze absolute success rates, changes from clean performance, and relative robustness. The results show substantial performance degradation despite high clean success rates. Domain-level profiles and perturbation-specific analyses reveal different vulnerabilities across models, including severe failures under distractor instructions, motion blur, missing proprioception, and action delay.
Critical strengths
-
The benchmark covers several consequential parts of the robot interaction loop. Including proprioceptive corruption and execution disturbances alongside visual and linguistic perturbations makes the evaluation useful for examining failures beyond perception. The common task–seed grid and paired clean references provide a sensible foundation for controlled comparisons.
-
The evaluation is substantial and reveals differences obscured by clean success rates. Seven systems are evaluated under the same intended benchmark specification. For example, GR00T-N1.7 and AVA-VLA both achieve 95% clean success but differ by approximately 7.7 percentage points in aggregate perturbed success. Such comparisons demonstrate the practical value of evaluating robustness separately from nominal task completion.
Critical weaknesses
-
The consistency of the evaluation protocol needs stronger validation. Appendix A.8 reports that OpenVLA-OFT and π₀.₅ each include 1,404 perturbed episodes associated with an older manifest hash. Matching variant IDs and confirming valid execution do not establish that the underlying evaluation conditions remained identical. Additionally, Section 4.2 specifies a ten-step action horizon for all models, while Table 6 lists an eight-action chunk for AVA-VLA, without explaining the adapter behavior. These details matter because execution and replanning directly affect sensitivity to action perturbations. The authors should demonstrate equivalence between manifest versions and clarify the actual execution schedule for each model.
-
Several perturbations do not yet support an unambiguous interpretation of the reported failures. For ambiguous instructions, preserving the original success checker does not guarantee that the intended target remains identifiable after distinguishing attributes or relations are removed. For missing proprioception, Table 4 changes both dropout probability and the replacement rule across severity levels, confounding the interpretation of progressive degradation. Likewise, the striking distractor-instruction collapse lacks controls separating semantic distraction from prompt formatting or truncation effects. Since diagnostic interpretation is a central contribution, the paper would benefit substantially from representative instruction examples, checks of target identifiability, and targeted controls that isolate these factors.
Questions for authors
-
What exactly changed between the two manifest versions mentioned in Appendix A.8? Can you demonstrate that the 1,404 affected episodes per model used equivalent initial states, perturbation parameters, and randomization rules, or provide replacement results under the current specification?
-
Execution and input interfaces: How is AVA-VLA’s eight-action output reconciled with the stated ten-step execution horizon? Which proprioceptive fields does each model actually consume, and how are the perturbations applied to those fields? Do zero-strength interventions through the perturbation code path reproduce the clean reference results?
-
Language validity: How frequently do the ambiguous instructions retain a uniquely identifiable target? Can you provide representative original and perturbed instructions, together with controls using length-matched irrelevant text and alternative clause ordering, to clarify the cause of the distractor-instruction collapse?