- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper analyzes the extent to which the robustness of Vision-Language-Action models must be considered jointly across both input modalities, namely text and image inputs. It tests a small VLA (MiniVLA) on a benchmark (LIBERO-90) to this end. The paper tests whether text and image corruptions act independently. Minor visual disturbances have a relatively strong effect, whereas rephrasing has little effect. In comparison, the seed and the hardware (CPU/MPS) have a surprisingly large impact.
Critical strengths
The paper emphasizes how important it is to repeat experiments across multiple seeds, which is far too often overlooked. The experiments are done carefully, with a pre-specified analysis plan and interval estimates. This is the main value of the paper for the community.
Critical weaknesses
- The study uses only one model and only one benchmark. All results come from MiniVLA on LIBERO-90, which ran on Apple Silicon. For an evaluation paper that gives recommendations for the whole field, this is too narrow. It is unclear whether the sensitivity to the seed, the dominance of visual disturbances, or the lack of interaction also apply to other VLAs or to other benchmarks and simulators.
- As far as I understand, the setup cannot really answer the title question. Mismatched instructions never succeed and strong visual corruptions succeeds almost never, so only the combination of paraphrase and mild visual perturbation leaves enough remaining success to measure compounding. And even there, since success cannot fall below zero, the part of the effect that exceeds the sum of the two individual effects can only be seen partially. Do I get this wrong? Please clarify.
The seed result is the most useful part of the paper, but similar gaps between runs have already been reported for VLAs [2]. These problems cannot be fixed with more analysis of the existing data. It would need more models and more benchmarks.
Further weaknesses:
- The backend finding is not new. That results can differ between CPU and GPU backends even with identical seeds is known and documented, for example in the PyTorch reproducibility notes [1]. The comparison is also only preliminary.
- The “Related Work” section is very incomplete. Most relevant is Rasouli et al. [2], who analyze VLA policies on BEHAVIOR-1K and report huge gaps between runs of the same task, which is close to the seed finding here. Further missing are a systematic positional robustness evaluation of OpenVLA [3], cross-model benchmarking experience with OpenVLA-OFT and others [4], worst-case search over physical variations (Eva-VLA [5]), scene fuzzing (VLATest [6]), RoboTwin 2.0 and its Plus variant [7], LIBERO-Safety [8] and adversarial attacks on VLAs [9]. OpenVLA-OFT is discussed three times, but its paper [10] is missing from the references.
[1] PyTorch documentation. Reproducibility. https://docs.pytorch.org/docs/stable/notes/randomness.html
[2] A. Rasouli et al. How VLAs (really) work in open-world environments. arXiv:2604.21192, 2026.
[3] Pang et al. Is OpenVLA truly robust? A systematic evaluation of positional robustness. IJCNLP-AACL, 2025.
[4] Y. Zhang et al. Experiences from benchmarking vision-language-action models for robotic manipulation. arXiv:2511.11298, 2025.
[5] Liu et al. Eva-VLA: evaluating vision-language-action models' robustness under real-world physical variations. arXiv:2509.18953, 2025.
[6] Z. Wang et al. VLATest: testing and evaluating vision-language-action models for robotic manipulation. FSE, 2025.
[7] T. Chen et al. RoboTwin 2.0: a scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. arXiv, 2025.
[8] R. Cui et al. LIBERO-Safety: a comprehensive benchmark for physical and semantic safety in vision-language-action models. arXiv:2606.23686, 2026.
[9] T. Wang et al. Exploring the adversarial vulnerabilities of vision-language-action models in robotics. ICCV, 2025.
[10] M. J. Kim, C. Finn, P. Liang. Fine-tuning vision-language-action models: optimizing speed and success. arXiv:2502.19645, 2025.
Questions for authors
Clarification and additional analysis (the answers could change my assessment):
- Which result answers the title question? As I read it, floor effects leave only paraphrase plus mild visual perturbation to test compounding. If I missed something here, please clarify.
- As far as I understand Section 5, the seed changes the positions of objects in the LIBERO scene. So the seed changes the scene itself and not only the numerics, while the abstract describes it as the same episode. How large is the effect if the scene is fixed and only the numerical seed changes? This matters for how the main finding should be read.
- What is new compared to Rasouli et al. [2], who already report large gaps between runs of VLAs on BEHAVIOR-1K? Do your recommendations also apply there?
Suggestions (not part of my assessment):
- Repeat the study with more models and benchmarks, and with perturbation strengths that leave room above zero success.
- Complete the CPU/MPS comparison and add a run on CUDA, where most published VLA results are computed.
- Two recent papers may be worth citing in a revision: LIBERO-VPro [11] and "Faster and Better?" [12]. The second one finds bugs in several VLA benchmarks, including LIBERO-Plus, that can change the ranking of methods. Both support your point that details of the benchmark influence the results.
[11] Li et al. LIBERO-VPro: benchmarking closed-loop visual robustness of robotic foundation models. arXiv:2609.24350, 2026. [12] Q. Chen et al. Faster and better? Benchmark bugs and design limitations distort the evaluation of vision-language-action acceleration. arXiv:2609.37771, 2026.
2 comments
Thank you for your response and for addressing my concerns. However, I still do not think the revisions and clarifications are sufficient for ICLR, and I therefore stand by my original recommendation.
Thank you for the detailed review and references!
Scope: We agree, please see point 3 of the general response.
Backend novelty: PyTorch does document numeric differences between CPU and GPU backends, but our addition is that in a closed-loop greedy policy at fp32 these differences change task outcomes, whereas in LLM decoding fp32 largely removes divergence across configurations (Yuan et al., 2025). We agree the comparison is preliminary and will describe it as such throughout.
Related work: These are all super helpful, thank you, we will add! "Faster and Better?" in particular supports our point that benchmark details can change results.
In response to your questions:
Which result answers the title question?
Paraphrase combined with mild visual perturbation is the only pairing with room to test compounding, and part of any super-additive effect is hidden because success cannot fall below zero. Our answer to the title question is less that the perturbations do not compound, more that this kind of design often cannot tell. Please see point 2 of the general response for the estimates (Δ = +4.3 pp, 95% CI [−11.4, +20.0], against a ceiling of 14.3 pp).
How large is the effect if the scene is fixed and only the numerical seed changes?
We agree that the seed changes the scene, and that the abstract's description of these as matched pairs overstated what was held fixed. Please see point 1 of the general response. The fixed-scene rerun we will add answers how large the effect is when only the numerical seed changes. We think the revised finding is still relevant for LIBERO users, since evaluations commonly treat initial states as fully specifying the start of an episode.
What is new compared to Rasouli et al. (2026), and do your recommendations also apply there?
Rasouli et al. report large run-to-run gaps for VLAs on BEHAVIOR-1K. Our contribution is narrower, in that we identify a specific source of this variation in LIBERO (fixture redraw at reset under a fixed initial state) that users can control for. We expect our recommendations to report seeds, repeat episodes, and report intervals to apply to their setting as well. We will add this reference.