acceptodds
Under review as a conference paper at ICLR 2027

Do Language and Visual Perturbations Compound? Interaction Effects and Backend Sensitivity in Compact Vision-Language-Action Model Evaluation

Abstract

Robustness evaluations of vision-language-action (VLA) models typically report per-perturbation success rates from a modest number of episodes, treating each episode's outcome as fixed and each perturbation as separable. Both assumptions bear directly on the validity of the resulting benchmark numbers. We test them in MiniVLA, a compact VLA with roughly 1B parameters, on LIBERO-90, using a pre-specified factorial design that crosses instruction and visual perturbations over 7 tasks (630 episodes). Individual episode outcomes are fragile: across 70 pairs of episodes that share the task, initial state, and instruction and run greedily on the same machine and backend, shifting only the random seed changes the outcome in 26% of pairs and the episode length in 63%, while aggregate success moves by two episodes. A preliminary comparison suggests that switching the inference backend at fp32 has a similarly sized effect. Within the factorial, visual perturbation dominates. Mild lighting and camera changes reduce success from 57% to 20%, while paraphrased instructions have little effect. Mismatched instructions drive success on the original task to zero, so this model does not ignore language, unlike recent reports for other VLAs. For paraphrase combined with mild visual perturbation, the only pairing with room above zero success, we find no detectable compounding, and we show how floor effects cap the super-additive loss such a design can detect. Sampled rollouts are no more diverse than a greedy policy perturbed with small Gaussian action noise. We recommend that small-sample VLA robustness evaluations report seeds and the inference backend, treat outcomes at a fixed initial state as stochastic by repeating episodes and reporting interval estimates, and test perturbations jointly at strengths that leave headroom above zero success.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language-action models, robustness evaluation, reproducibility, LIBERO, embodied AI, evaluation methodology, mixed-effects models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_UnKTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper analyzes the extent to which the robustness of Vision-Language-Action models must be considered jointly across both input modalities, namely text and image inputs. It tests a small VLA (MiniVLA) on a benchmark (LIBERO-90) to this end. The paper tests whether text and image corruptions act independently. Minor visual disturbances have a relatively strong effect, whereas rephrasing has little effect. In comparison, the seed and the hardware (CPU/MPS) have a surprisingly large impact.

Critical strengths

The paper emphasizes how important it is to repeat experiments across multiple seeds, which is far too often overlooked. The experiments are done carefully, with a pre-specified analysis plan and interval estimates. This is the main value of the paper for the community.

Critical weaknesses
  1. The study uses only one model and only one benchmark. All results come from MiniVLA on LIBERO-90, which ran on Apple Silicon. For an evaluation paper that gives recommendations for the whole field, this is too narrow. It is unclear whether the sensitivity to the seed, the dominance of visual disturbances, or the lack of interaction also apply to other VLAs or to other benchmarks and simulators.
  2. As far as I understand, the setup cannot really answer the title question. Mismatched instructions never succeed and strong visual corruptions succeeds almost never, so only the combination of paraphrase and mild visual perturbation leaves enough remaining success to measure compounding. And even there, since success cannot fall below zero, the part of the effect that exceeds the sum of the two individual effects can only be seen partially. Do I get this wrong? Please clarify.

The seed result is the most useful part of the paper, but similar gaps between runs have already been reported for VLAs [2]. These problems cannot be fixed with more analysis of the existing data. It would need more models and more benchmarks.

Further weaknesses:

  • The backend finding is not new. That results can differ between CPU and GPU backends even with identical seeds is known and documented, for example in the PyTorch reproducibility notes [1]. The comparison is also only preliminary.
  • The “Related Work” section is very incomplete. Most relevant is Rasouli et al. [2], who analyze VLA policies on BEHAVIOR-1K and report huge gaps between runs of the same task, which is close to the seed finding here. Further missing are a systematic positional robustness evaluation of OpenVLA [3], cross-model benchmarking experience with OpenVLA-OFT and others [4], worst-case search over physical variations (Eva-VLA [5]), scene fuzzing (VLATest [6]), RoboTwin 2.0 and its Plus variant [7], LIBERO-Safety [8] and adversarial attacks on VLAs [9]. OpenVLA-OFT is discussed three times, but its paper [10] is missing from the references.

[1] PyTorch documentation. Reproducibility. https://docs.pytorch.org/docs/stable/notes/randomness.html

[2] A. Rasouli et al. How VLAs (really) work in open-world environments. arXiv:2604.21192, 2026.

[3] Pang et al. Is OpenVLA truly robust? A systematic evaluation of positional robustness. IJCNLP-AACL, 2025.

[4] Y. Zhang et al. Experiences from benchmarking vision-language-action models for robotic manipulation. arXiv:2511.11298, 2025.

[5] Liu et al. Eva-VLA: evaluating vision-language-action models' robustness under real-world physical variations. arXiv:2509.18953, 2025.

[6] Z. Wang et al. VLATest: testing and evaluating vision-language-action models for robotic manipulation. FSE, 2025.

[7] T. Chen et al. RoboTwin 2.0: a scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. arXiv, 2025.

[8] R. Cui et al. LIBERO-Safety: a comprehensive benchmark for physical and semantic safety in vision-language-action models. arXiv:2606.23686, 2026.

[9] T. Wang et al. Exploring the adversarial vulnerabilities of vision-language-action models in robotics. ICCV, 2025.

[10] M. J. Kim, C. Finn, P. Liang. Fine-tuning vision-language-action models: optimizing speed and success. arXiv:2502.19645, 2025.

Questions for authors

Clarification and additional analysis (the answers could change my assessment):

  • Which result answers the title question? As I read it, floor effects leave only paraphrase plus mild visual perturbation to test compounding. If I missed something here, please clarify.
  • As far as I understand Section 5, the seed changes the positions of objects in the LIBERO scene. So the seed changes the scene itself and not only the numerics, while the abstract describes it as the same episode. How large is the effect if the scene is fixed and only the numerical seed changes? This matters for how the main finding should be read.
  • What is new compared to Rasouli et al. [2], who already report large gaps between runs of VLAs on BEHAVIOR-1K? Do your recommendations also apply there?

Suggestions (not part of my assessment):

  • Repeat the study with more models and benchmarks, and with perturbation strengths that leave room above zero success.
  • Complete the CPU/MPS comparison and add a run on CUDA, where most published VLA results are computed.
  • Two recent papers may be worth citing in a revision: LIBERO-VPro [11] and "Faster and Better?" [12]. The second one finds bugs in several VLA benchmarks, including LIBERO-Plus, that can change the ranking of methods. Both support your point that details of the benchmark influence the results.

[11] Li et al. LIBERO-VPro: benchmarking closed-loop visual robustness of robotic foundation models. arXiv:2609.24350, 2026. [12] Q. Chen et al. Faster and better? Benchmark bugs and design limitations distort the evaluation of vision-language-action acceleration. arXiv:2609.37771, 2026.

2 comments

Thank you for the detailed review and references!

Scope: We agree, please see point 3 of the general response.

Backend novelty: PyTorch does document numeric differences between CPU and GPU backends, but our addition is that in a closed-loop greedy policy at fp32 these differences change task outcomes, whereas in LLM decoding fp32 largely removes divergence across configurations (Yuan et al., 2025). We agree the comparison is preliminary and will describe it as such throughout.

Related work: These are all super helpful, thank you, we will add! "Faster and Better?" in particular supports our point that benchmark details can change results.

In response to your questions:

Which result answers the title question?

Paraphrase combined with mild visual perturbation is the only pairing with room to test compounding, and part of any super-additive effect is hidden because success cannot fall below zero. Our answer to the title question is less that the perturbations do not compound, more that this kind of design often cannot tell. Please see point 2 of the general response for the estimates (Δ = +4.3 pp, 95% CI [−11.4, +20.0], against a ceiling of 14.3 pp).

How large is the effect if the scene is fixed and only the numerical seed changes?

We agree that the seed changes the scene, and that the abstract's description of these as matched pairs overstated what was held fixed. Please see point 1 of the general response. The fixed-scene rerun we will add answers how large the effect is when only the numerical seed changes. We think the revised finding is still relevant for LIBERO users, since evaluations commonly treat initial states as fully specifying the start of an episode.

What is new compared to Rasouli et al. (2026), and do your recommendations also apply there?

Rasouli et al. report large run-to-run gaps for VLAs on BEHAVIOR-1K. Our contribution is narrower, in that we identify a specific source of this variation in LIBERO (fixture redraw at reset under a fixed initial state) that users can control for. We expect our recommendations to report seeds, repeat episodes, and report intervals to apply to their setting as well. We will add this reference.

Thank you for your response and for addressing my concerns. However, I still do not think the revisions and clarifications are sufficient for ICLR, and I therefore stand by my original recommendation.

Reviewer_jYFZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates two questions in the evaluation of compact VLA models: whether language and visual perturbations produce compounding effects, and whether episode outcomes are sensitive to random seeds and inference backends. Using MiniVLA, the authors conduct a 3×3 factorial experiment comprising 630 episodes across seven tasks from LIBERO-90. Interaction effects are analyzed using Firth-penalized mixed-effects logistic regression and bootstrap methods. The results show that visual perturbations account for the main performance degradation, while preliminary experiments also indicate sensitivity to the inference backend. Based on these findings, the paper provides recommendations for repeated evaluation, documentation of experimental conditions, and uncertainty reporting.

Critical strengths

S1. The paper shows that aggregate success rates can mask substantial variability in individual episode outcomes, providing useful insights for experimental design and result interpretation in small-sample VLA evaluation.

S2. The statistical analysis is detailed, and the authors provide a balanced discussion of the study’s limitations.

Critical weaknesses

W1. The experimental scope is narrow, limiting the generalizability of the findings. The study covers only one MiniVLA checkpoint, seven selected tasks, and one paraphrase per task. Moreover, the visual perturbations jointly alter lighting and camera position, making it difficult to determine whether the conclusions extend to other VLAs, language variations, or individual types of visual perturbations.

W2. The conclusion regarding sampling diversity is stronger than the available evidence supports. The experiment covers only two tasks, and the absence of a statistically significant difference in per-step action distance does not establish equivalence in behavioral diversity. The claim that sampling provides no richer behavioral variation than Gaussian action noise therefore requires more direct validation.

Questions for authors

Please see the weakness.

2 comments

Scope and the lighting/camera confound: One checkpoint, seven tasks, and one paraphrase per task limit how far the findings extend to other VLAs or language variations. We combined lighting and camera changes to keep the factorial at 3 × 3, which means we cannot attribute the visual effect to either factor. Please see point 3 of the general response for how we will narrow the claims.

Sampling diversity: We agree that a non-significant difference does not establish equivalence, and the per-step distance result is a null. The distinct-first-action count does show a difference, however: both noise conditions reach 10 of 10 while sampling averages 8.00 (difference +2.00, 95% CI [+1.00, +3.20]), so sampling is, if anything, less varied on that measure. We will revise the claim to say that on two tasks we did not detect greater diversity from sampling than from small Gaussian action noise, and remove the wording that implies the two are equivalent.

Reviewer_jYFZOpenReview ↗

Response to Authors

Thank you for your reponse. However, I still decided to keep the original rating score based on your current version of paper.

Reviewer_8c5mOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper evaluates the robustness of vision-language-action models, focusing on two hypotheses 1) whether language and visual perturbations have additive effects 2) whether nominally identical robotic episodes produce reproducible outcomes. The authors evaluate a single model, MiniVLA (about 1B parameters), on selected LIBERO-90 tasks. Their primary experiment employs a 3 × 3 factorial design covering three instruction conditions and three visual conditions. The paper reports that mild visual perturbations substantially reduce success rates, while instruction paraphrasing has a relatively small effect. The primary interaction analysis does not identify statistically significant compounding between paraphrasing and mild visual perturbations. The paper also reports that changing the random seed alters the outcomes of 18 of 70 matched episode pairs, while aggregate success changes by only two episodes. In addition, the authors perform a CPU-versus-MPS comparison and identify outcome differences in 7 of 34 completed pairs.

Critical strengths
  • The paper evaluates the compounding effects of language and visual perturbations, rather than considering only the success rates under individual perturbations.

  • The paper evaluates three instruction conditions (original, paraphrased, and mismatched) and three visual conditions (normal, mild, and strong).

  • The paper provides a statistical analysis of perturbation interactions, including confidence intervals.

  • The experimental setup is described in detail, including the model configuration, simulator versions, hardware, and evaluation settings.

  • The paper also reports a CPU-versus-MPS comparison to investigate the effects of inference backends on evaluation outcomes.

Critical weaknesses
  • The paper is poorly written and reads more like a collection of loosely connected experimental observations than a cohesive research study.

  • The main research question, whether language and visual perturbations compound, remains largely unanswered. The effect of paraphrasing is relatively small, while visual perturbations substantially reduce success rates, resulting in floor effects that limit the interaction analysis.

  • The main compounding analysis is restricted to one language perturbation (paraphrasing) and one visual perturbation level (mild). In addition, the visual perturbation combines lighting and camera-position changes, making it difficult to isolate their individual effects.

  • The paper evaluates only one VLA model, MiniVLA, without comparisons to other models such as OpenVLA or SmolVLA. This significantly limits the generalizability of the findings.

  • The criteria for selecting the seven tasks from LIBERO-90 are not sufficiently explained. In particular, it is unclear why task 46, which was included in the baseline evaluation, was excluded from the main experiment.

  • The paper reports that changing the random seed alters episode outcomes, but Section 5 explains that LIBERO redraws fixture positions during environment reset. The observed differences therefore seem to arise mainly from incomplete specification of the simulator state rather than from intrinsic nondeterminism in greedy VLA inference.

  • All experiments are run on Apple silicon, with no CUDA evaluation. The CPU-versus-MPS comparison is also preliminary, with only 34 completed episode pairs. This makes it difficult to generalize the findings to other hardware configurations.

  • The paper claims that MiniVLA does not ignore language, based on experiments with mismatched instructions. However, a wrong instruction disrupting the original task does not establish that the model understands or correctly follows the replacement instruction.

  • The paper does not provide direct experimental comparisons with existing VLA robustness benchmarks or evaluation protocols, such as RobustVLA.

Questions for authors
  • Why was task 46 included in the baseline evaluation but excluded from the main experiment?
  • Why were only 34 of the 80 CPU-versus-MPS episode pairs completed? Could the incomplete episodes bias the results?
1 comment

Thank you again for your detailed review!

Cohesion - We will revise the introduction so that each result maps to one of the two assumptions in the abstract (that an episode's outcome at a fixed setup is stable and that perturbations act separately).

Title question and floor effects - We agree that floor effects limit the interaction analysis. Please see point 2 of the general response.

Lighting and camera combined - We combined them to keep the factorial at 3 × 3, which means we cannot attribute the visual effect to either one. We will list this as a limitation.

One model, Apple Silicon only - Please see point 3 of the general response.

Seed result - We agree that the differences come from incomplete specification of the simulator state. Please see point 1 of the general response.

We also agree that the wording "does not ignore language" goes too far. Mismatched instructions show that the model's behavior depends on the instruction, in contrast to reports of instruction-invariant behavior in other VLAs, but they do not show that the model follows the replacement instruction. Section 4.1 already states this limit, and we will bring the abstract and contributions in line with it.

We will expand the comparison to RobustVLA in Related Work. RobustVLA covers more perturbation types across four modalities, including mixed perturbations, and reports success under each mix as a point estimate. Our design covers fewer perturbations but estimates the interaction directly with an interval.

In response to your questions:

  1. Why was task 46 included in the baseline evaluation but excluded from the main experiment?

The seven grid tasks (9, 18, 27, 30, 36, 37, 73) were fixed in the job-generation script before any grid episode ran, and were chosen to span high (9, 30, 37), medium (18, 36), and low (27, 73) clean success. Task 46 was in the clean-baseline set but not on the grid list, so its exclusion does not depend on any grid outcome. We will state this in the setup.

  1. Why only 34 of 80 CPU-vs-MPS pairs, and could this bias the result?

The full CPU leg was projected at about 40 hours, compared with about 2.6 hours on MPS, so we ran CPU as a scoped subset in our fixed job order (sorted by initial-state index, then task). The 34 pairs are the jobs that completed on both backends in this scoped run. Selection followed our job order, which fills the lowest initial-state indices first, rather than episode outcome. The subset is still not random, as it is concentrated in the lowest initial-state indices, and the paper's explanation in terms of hung episodes was incomplete. We will correct the text, note that the subset is not random, and complete the remaining 46 CPU episodes for the revision. This is also why we describe the backend comparison as preliminary.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their time and detailed responses. We found three major concerns, and we both address them here and refer back to them in individual replies.

  1. Seed result (8c5m, UnKT): We agree that the seed result reflects scene specification rather than inference numerics. Section 5 identifies the mechanism (LIBERO redraws fixture positions on every reset, and initial states do not record them), but the abstract and introduction describe the paired episodes as sharing everything the benchmark defines, which overstates what was held fixed. We will revise the framing to say that a LIBERO initial state does not fully specify the scene, so episodes at the same initial state are not paired across seeds or runs. In our data this changed the outcome in 18 of 70 pairs while aggregate success moved by two episodes. We will correct the abstract and introduction, and rerun the 70 pairs with fixture placement held fixed to separate scene variation from numerical variation.

  2. The title question (8c5m, UnKT): Mismatched instructions give zero success and strong visual corruption gives near-zero success, so paraphrase combined with mild visual perturbation is the only cell with room to test compounding. There the pre-specified mixed-effects model finds no detectable interaction (log-odds −0.568, 95% CI [−1.817, 0.637], p = 0.358, n = 280). On the probability scale Δ = +4.3 pp (95% CI [−11.4, +20.0]), but because success under both perturbations cannot fall below zero, Δ cannot exceed Δmax = 14.3 pp given the observed single-perturbation rates. The upper end of the interval lies beyond anything the design could produce, so the probability-scale result rules out no achievable super-additive loss, and the log-odds interaction is the informative test. We do not claim that the perturbations never compound. Our answer to the title question is that a design of this kind often cannot tell, which Section 4.2 quantifies. We will state this directly in the introduction.

  3. Scope (all reviewers): We agree that one model, one benchmark, and Apple Silicon only limit how far the findings generalize, and we cannot add models, benchmarks, or a CUDA run within the discussion period. We will narrow the recommendations to practices our results motivate rather than findings shown to hold across VLAs, and list the combined lighting and camera condition as a limitation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.