- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces VLA-Retain, a benchmark for evaluating how targeted vision fine-tuning affects robustness across different perturbation dimensions in vision-language-action models. The authors evaluate five VLA architectures on 10,030 LIBERO-Plus episodes covering seven perturbation dimensions. They compare original and fine-tuned checkpoints under Camera-view, Sensor-noise, and Mixed training, and investigate alternative adaptation strategies. The main finding is that improvements on a target perturbation frequently cause performance regressions elsewhere, with negative mean off-target gains in 8 of 10 model-domain configurations. The study also demonstrates that broader training coverage and representation-preserving methods do not consistently eliminate these regressions.
Critical strengths
-
Comprehensive and carefully controlled evaluation. The study covers five VLA architectures, seven perturbation dimensions, and over 10,000 paired evaluation episodes. Comparing each fine-tuned checkpoint against its own baseline, using paired bootstrap confidence intervals, and controlling for source-domain forgetting provides convincing evidence that targeted adaptation can affect robustness beyond the target condition.
-
Important empirical findings with practical implications. The paper demonstrates that large target-domain improvements can coexist with substantial off-target regressions. The model-dependent behavior of adaptation strategies, particularly the contrasting results of DELTA-style preservation on X-VLA and OpenVLA-OFT, highlights a relevant challenge for reliable VLA deployment. These findings support evaluating adaptation through multidimensional robustness rather than target success alone.
Critical weaknesses
-
Limited adaptation and evaluation scope. Although seven perturbation dimensions are evaluated, targeted fine-tuning is performed only for Camera-view and Sensor-noise. Furthermore, all experiments use LIBERO-Plus simulation with the same 40 underlying manipulation tasks. Consequently, it remains unclear whether the observed trade-offs generalize to other adaptation targets, unseen manipulation tasks, or real-world conditions.
-
Limited training-seed coverage and explanatory analysis. The main benchmark evaluates one final checkpoint per configuration, while the three-seed analysis covers only two models on a smaller 560-episode subset. The reported bootstrap intervals account for task-level uncertainty but not training randomness, and only six of ten negative mean off-target gains have confidence intervals entirely below zero. Additionally, the paper does not investigate the representation changes responsible for these regressions, limiting its ability to explain the observed differences between architectures.
Questions for authors
-
Adaptation diversity. Why were only Camera-view and Sensor-noise selected as tuning targets? Is there evidence that adapting to other dimensions, such as Lighting, Background, or Language, produces similar cross-dimensional regressions? This would help establish how general the central finding is.
-
Statistical robustness. Can the authors provide additional training-seed results for the remaining three architectures? In particular, how stable are the smaller off-target regressions across independent fine-tuning runs?
-
Model-dependent retention. Can the authors provide additional analysis explaining why DELTA-style feature preservation improves X-VLA but severely degrades OpenVLA-OFT? For example, measuring representation drift between base and tuned visual encoders could help determine whether the effect reflects architectural differences or optimization instability.
3 comments
Thank you for your additional experiments, some of them have closed my concerns, but some of them still not covered.
Globally, idea of the paper appeals to me, but I still can't understand how useful your findings are. Your results and openings should be supported by a wider range of experiments on different benchmars such as RoboTwin 2.0, Colosseum V2, LIBERO MAX, VLA Agena.
For now I prefer to keep my score.
We thank the reviewer for the follow-up and are glad that the additional experiments resolved several concerns.
We would like to briefly clarify the practical value of our findings. Target success alone can hide the cost of an update under other conditions. Camera-view tuning raises X-VLA's success under camera shifts by 69.4 pp, yet the tuned model succeeds in only 6.3% of episodes once blur is added to the camera shift. InternVLA-A1.5 loses 12.4 pp under robot-state perturbations even though its success on the original tasks stays nearly unchanged. VLA-Retain exposes such hidden costs before deployment, and it shows that balanced training data covering the expected shifts reduces them.
The paired protocol does not depend on LIBERO, and we plan to apply it to RoboTwin 2.0, Colosseum V2, LIBERO MAX and VLA-Arena in the future.
Response to Reviewer eWVX
We thank the reviewer for the constructive feedback. We have conducted additional experiments and refer to the relevant content in the paper to answer the following questions.
Q1. Choice of adaptation targets.
Camera-view and Sensor-noise are the visual perturbations on which current VLAs are weakest. Averaged over the five base models, success is 46.5% and 66.0% under them, compared with about 89% and 90% under Lighting and Background. Language perturbations act on the instruction rather than the image, so they fall outside vision tuning.
We additionally tuned X-VLA and OpenVLA-OFT on Lighting. X-VLA again regresses off target, by 3.0 pp on average, although its Lighting success was already 96.7%. The regression is therefore not a by-product of a large target gain. OpenVLA-OFT again retains, with +0.9 pp. A third target thus reproduces both the regression and its model dependence.
Q2 and W2. Training-seed and statistical stability.
We extended the three-seed analysis to all five models. Camera-view tuning was retrained with seeds 42, 43 and 44, and each run was evaluated on the same 560-episode subset. The table reports the mean off-target change in pp.
The sign is identical across seeds for every model, including the smaller regressions of π0.5 and InternVLA-A1.5. We also evaluated all three X-VLA seeds on the full 10,030 episodes. Their mean off-target changes are −4.7, −4.6 and −6.3 pp, each with an interval below zero, and their standard deviation across seeds is only 1.0 pp. For the ten targeted configurations in the paper, the sign of every mean off-target change is also unchanged when any single task family is removed.
Q3. Why DELTA-style preservation differs between X-VLA and OpenVLA-OFT.
Following the reviewer's suggestion, we fed the same 280 observations, covering all 40 tasks and seven perturbations, to each checkpoint and compared the visual tokens it passes to the language model with those of its base model. Default denotes our standard LoRA tuning of the vision encoder and connector.
On X-VLA, DELTA-style keeps the features close to the base as designed. On OpenVLA-OFT, the same penalty weight fails to anchor them, so the degradation comes from this failure rather than from feature preservation itself. Lowering the weight restores both features and retention, which shows that the failure stems from optimization instability under a strong penalty rather than from an architectural limit. In both models, the checkpoint whose features drift most also retains worst, consistent with off-target regression tracking representation change. We will add this analysis to the paper.
W1. Unseen tasks and real-world settings.
LIBERO-Plus varies one perturbation at a time over the same tasks, which enables the paired per-dimension comparison at the core of VLA-Retain. Training and evaluation already use different perturbation instances. In future work, we plan to extend VLA-Retain to unseen tasks, other simulators and physical robots, applying the same paired protocol to test whether the cross-dimension effects hold beyond LIBERO.