acceptodds
Under review as a conference paper at ICLR 2027

LIBERO-MOR: A Multimodal Robustness Benchmark Across the Vision-Language-Action Pipeline

Abstract

Vision-language-action (VLA) models have achieved strong performance on robot manipulation benchmarks. Translating this progress into reliable deployment calls for systematic evaluation of how policies respond to variations throughout the robot interaction loop. We introduce LIBERO-MOR, a multimodal robustness benchmark across the vision-language-action pipeline. Built on standard LIBERO, it covers five deployment-facing domains: vision, language, proprioception, action execution, and environment. The benchmark comprises 22 deployment-inspired perturbations, each evaluated at three severity levels across 40 tasks under a paired clean–perturbed protocol, enabling controlled and reproducible comparisons. We evaluate seven representative VLA models spanning diverse action-generation mechanisms. Our experiments reveal substantial robustness gaps behind high clean-task success, together with shared vulnerabilities and distinct model-specific failure patterns across domains. Severity analysis further distinguishes progressive degradation, collapse at stronger interventions, and failures triggered at the mildest tested level. By connecting aggregate performance to pipeline domains, individual perturbations, and intervention strength, LIBERO-MOR provides a systematic framework for diagnosing VLA robustness and guiding the development of more reliable robot policies.

Primary area:
datasets and benchmarks
Keywords:
Vision-Language-Action Models, Robustness Evaluation, Embodied AI, Multimodal Learning, Robot Manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1GTpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LIBERO-MOR, a robustness benchmark built on four LIBERO suites. It evaluates 22 perturbations across vision, language, proprioception, action execution, and environment, using three severity levels and five seeds for each of 40 tasks. Perturbed episodes are paired with clean references through a shared evaluation manifest. The authors report 93,800 rollouts across seven VLA systems and analyze absolute success rates, changes from clean performance, and relative robustness. The results show substantial performance degradation despite high clean success rates. Domain-level profiles and perturbation-specific analyses reveal different vulnerabilities across models, including severe failures under distractor instructions, motion blur, missing proprioception, and action delay.

Critical strengths
  1. The benchmark covers several consequential parts of the robot interaction loop. Including proprioceptive corruption and execution disturbances alongside visual and linguistic perturbations makes the evaluation useful for examining failures beyond perception. The common task–seed grid and paired clean references provide a sensible foundation for controlled comparisons.

  2. The evaluation is substantial and reveals differences obscured by clean success rates. Seven systems are evaluated under the same intended benchmark specification. For example, GR00T-N1.7 and AVA-VLA both achieve 95% clean success but differ by approximately 7.7 percentage points in aggregate perturbed success. Such comparisons demonstrate the practical value of evaluating robustness separately from nominal task completion.

Critical weaknesses
  1. The consistency of the evaluation protocol needs stronger validation. Appendix A.8 reports that OpenVLA-OFT and π₀.₅ each include 1,404 perturbed episodes associated with an older manifest hash. Matching variant IDs and confirming valid execution do not establish that the underlying evaluation conditions remained identical. Additionally, Section 4.2 specifies a ten-step action horizon for all models, while Table 6 lists an eight-action chunk for AVA-VLA, without explaining the adapter behavior. These details matter because execution and replanning directly affect sensitivity to action perturbations. The authors should demonstrate equivalence between manifest versions and clarify the actual execution schedule for each model.

  2. Several perturbations do not yet support an unambiguous interpretation of the reported failures. For ambiguous instructions, preserving the original success checker does not guarantee that the intended target remains identifiable after distinguishing attributes or relations are removed. For missing proprioception, Table 4 changes both dropout probability and the replacement rule across severity levels, confounding the interpretation of progressive degradation. Likewise, the striking distractor-instruction collapse lacks controls separating semantic distraction from prompt formatting or truncation effects. Since diagnostic interpretation is a central contribution, the paper would benefit substantially from representative instruction examples, checks of target identifiability, and targeted controls that isolate these factors.

Questions for authors
  1. What exactly changed between the two manifest versions mentioned in Appendix A.8? Can you demonstrate that the 1,404 affected episodes per model used equivalent initial states, perturbation parameters, and randomization rules, or provide replacement results under the current specification?

  2. Execution and input interfaces: How is AVA-VLA’s eight-action output reconciled with the stated ten-step execution horizon? Which proprioceptive fields does each model actually consume, and how are the perturbations applied to those fields? Do zero-strength interventions through the perturbation code path reproduce the clean reference results?

  3. Language validity: How frequently do the ambiguous instructions retain a uniquely identifiable target? Can you provide representative original and perturbed instructions, together with controls using length-matched irrelevant text and alternative clause ordering, to clarify the cause of the distractor-instruction collapse?

Reviewer_yuPsOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

LIBERO-MOR evaluates VLA robustness under 22 perturbations spanning vision, language, proprioception, action execution, and environment. It applies three severity levels to 40 LIBERO tasks using paired clean–perturbed evaluations, reporting aggregate performance and detailed robustness profiles for seven policies.

Critical strengths

A systematic and reproducible evaluation protocol. The fixed task–seed manifest, matched clean references, and explicit perturbation parameters support controlled comparisons. Reporting absolute success alongside relative robustness helps distinguish nominal capability from degradation under perturbation.

Useful diagnostic granularity. Domain-level profiles, individual perturbation results, and severity curves reveal vulnerabilities obscured by clean-task scores. The substantial perturbed-performance drops and model-specific failure patterns demonstrate that the evaluated setting remains challenging.

Critical weaknesses

The incremental contribution over existing multimodal robustness evaluations is insufficiently established. RoBustVLA[1] already evaluate 17 perturbations across four modalities on LIBERO. LIBERO-MOR expands this coverage and standardizes reporting, but does not sufficiently demonstrate which important capabilities or failure modes its additions uniquely measure. The large rollout count establishes evaluation scale, while the case for a distinct benchmark requires clearer evidence of additional diagnostic value under matched comparisons.

The validity of some perturbations as robustness measurements is not adequately verified. The ambiguous-instruction operator removes attributes and relations with probabilities reaching 0.92 while retaining the original success checker. Such deletion may remove information necessary to identify the intended goal, confounding robustness with task under-specification. More generally, the paper does not establish that perturbed execution with zero intervention strength reproduces the clean reference, which would help exclude differences introduced by the perturbation pipeline itself.

[1] https://arxiv.org/abs/2510.00037 Guo, J., et al. (2026). On Robustness of Vision-Language-Action Model against Multi-Modal Perturbations. ICLR 2026.

Questions for authors
  1. Under matched checkpoints and execution settings, what important failures or model distinctions does LIBERO-MOR reveal beyond the closest existing multimodal robustness evaluation?
  2. How are ambiguous instructions checked to ensure that the intended goal remains identifiable? When multiple outcomes become reasonable, how does evaluation account for this ambiguity?
Reviewer_PGxgOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces LIBERO-MOR, a benchmark comprising 22 perturbations across vision, language, proprioception, action execution, and environment, evaluated at three severity levels on 40 LIBERO tasks. Across seven VLA models and 93,800 rollouts, the results reveal substantial robustness gaps despite high clean-task success, alongside model-specific sensitivities and degradation patterns.

Critical strengths
  • Relevant problem and systematic coverage. Evaluating disturbances across perception, state estimation, and execution reveals vulnerabilities that standard task-success metrics can overlook.

  • Controlled and comparable evaluation. Single-factor interventions use matched tasks, initial states, and seeds. Reporting absolute success, performance degradation, and relative robustness provides complementary perspectives on model performance.

  • Informative fine-grained analysis. Distractor instructions cause near-total failure in some models, while motion blur and missing proprioception produce distinct degradation curves. These findings offer more actionable diagnostic insights than aggregate rankings alone.

Critical weaknesses
  • The contribution over prior work requires clearer justification. Guo et al. already evaluate 17 perturbations across four modalities. This paper expands coverage and introduces a shared diagnostic protocol, but direct comparisons are needed to establish which scientific insights these additions uniquely enable.

  • The validity of some perturbations remains insufficiently established. Removing attributes and spatial relations may make the intended goal ambiguous; retaining the original success checker does not ensure that the task remains identifiable. Moreover, cross-domain vulnerability rankings depend on the selected operators and severity ranges, which lack a common calibration.

  • The reliability and interpretation of model differences remain limited. Evaluation uses only five seeds per task and provides no confidence intervals accounting for the paired design. The shared execution budget may also interact with model-specific operating requirements, leaving smaller ranking differences and non-monotonic responses insufficiently supported.

Questions for authors
  1. Under matched models, tasks, and execution budgets, what conclusions do the additional perturbations and severity levels reveal that the closest existing multimodal robustness benchmark cannot?

  2. Could human assessment or reference policies verify goal identifiability after language perturbations? Do the main vulnerability rankings remain stable under alternative severity ranges and domain weights?

  3. Could the authors provide task-clustered paired confidence intervals, zero-strength perturbation controls, and comparisons with native execution settings to establish that the key performance drops and model differences arise from the intended perturbations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.