acceptodds
Under review as a conference paper at ICLR 2027

VLA-Retain: Benchmarking Cross-Dimension Effects of Vision Tuning in VLAs

Abstract

Vision-language-action (VLA) models transfer broad visual and semantic knowledge from large-scale pretraining into robot control, but their manipulation performance remains sensitive to changes in observation conditions, which recent work addresses by fine-tuning the visual module on target-domain data. However, it remains unclear how such adaptation affects robustness beyond the target condition, as existing robustness benchmarks only evaluate models without fine-tuning and adaptation studies only report the target gain. Measuring this cross-dimension effect requires separating target gains from off-target changes, isolating adaptation effects from architectural differences, ruling out generic forgetting, and comparing tuning choices. Therefore, we introduce **VLA-Retain**, a paired benchmark that compares five open VLA models, tuned on Camera-view or Sensor-noise data, with their own base checkpoints on the same 10,030 LIBERO-Plus episodes spanning seven perturbation dimensions under varied training data and tuning strategies. Our study reveals four findings. First, targeted vision tuning is not local, and its target gain does not predict retention. Mean off-target gain is negative in 8 of 10 model–domain configurations. Under Camera-view tuning, X-VLA gains percentage points on the target but loses points on average elsewhere, whereas OpenVLA-OFT gains and points, respectively.Second, these regressions are not explained only by performance loss on the same tasks without perturbations. Third, broader training coverage raises overall gain over target-specific training but does not eliminate dimension-level regressions. Fourth, tuning strategies are model-dependent, as DELTA-style feature preservation raises X-VLA's mean off-target gain to points but lowers OpenVLA-OFT's to . Overall, these findings show that target-domain success alone is insufficient and highlight cross-dimensional retention as a key criterion for targeted VLA adaptation.

Primary area:
datasets and benchmarks
Keywords:
VLA, fine tuning, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eWVXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces VLA-Retain, a benchmark for evaluating how targeted vision fine-tuning affects robustness across different perturbation dimensions in vision-language-action models. The authors evaluate five VLA architectures on 10,030 LIBERO-Plus episodes covering seven perturbation dimensions. They compare original and fine-tuned checkpoints under Camera-view, Sensor-noise, and Mixed training, and investigate alternative adaptation strategies. The main finding is that improvements on a target perturbation frequently cause performance regressions elsewhere, with negative mean off-target gains in 8 of 10 model-domain configurations. The study also demonstrates that broader training coverage and representation-preserving methods do not consistently eliminate these regressions.

Critical strengths
  1. Comprehensive and carefully controlled evaluation. The study covers five VLA architectures, seven perturbation dimensions, and over 10,000 paired evaluation episodes. Comparing each fine-tuned checkpoint against its own baseline, using paired bootstrap confidence intervals, and controlling for source-domain forgetting provides convincing evidence that targeted adaptation can affect robustness beyond the target condition.

  2. Important empirical findings with practical implications. The paper demonstrates that large target-domain improvements can coexist with substantial off-target regressions. The model-dependent behavior of adaptation strategies, particularly the contrasting results of DELTA-style preservation on X-VLA and OpenVLA-OFT, highlights a relevant challenge for reliable VLA deployment. These findings support evaluating adaptation through multidimensional robustness rather than target success alone.

Critical weaknesses
  1. Limited adaptation and evaluation scope. Although seven perturbation dimensions are evaluated, targeted fine-tuning is performed only for Camera-view and Sensor-noise. Furthermore, all experiments use LIBERO-Plus simulation with the same 40 underlying manipulation tasks. Consequently, it remains unclear whether the observed trade-offs generalize to other adaptation targets, unseen manipulation tasks, or real-world conditions.

  2. Limited training-seed coverage and explanatory analysis. The main benchmark evaluates one final checkpoint per configuration, while the three-seed analysis covers only two models on a smaller 560-episode subset. The reported bootstrap intervals account for task-level uncertainty but not training randomness, and only six of ten negative mean off-target gains have confidence intervals entirely below zero. Additionally, the paper does not investigate the representation changes responsible for these regressions, limiting its ability to explain the observed differences between architectures.

Questions for authors
  1. Adaptation diversity. Why were only Camera-view and Sensor-noise selected as tuning targets? Is there evidence that adapting to other dimensions, such as Lighting, Background, or Language, produces similar cross-dimensional regressions? This would help establish how general the central finding is.

  2. Statistical robustness. Can the authors provide additional training-seed results for the remaining three architectures? In particular, how stable are the smaller off-target regressions across independent fine-tuning runs?

  3. Model-dependent retention. Can the authors provide additional analysis explaining why DELTA-style feature preservation improves X-VLA but severely degrades OpenVLA-OFT? For example, measuring representation drift between base and tuned visual encoders could help determine whether the effect reflects architectural differences or optimization instability.

3 comments
AuthorsOpenReview ↗

Response to Reviewer eWVX

We thank the reviewer for the constructive feedback. We have conducted additional experiments and refer to the relevant content in the paper to answer the following questions.

Q1. Choice of adaptation targets.

Camera-view and Sensor-noise are the visual perturbations on which current VLAs are weakest. Averaged over the five base models, success is 46.5% and 66.0% under them, compared with about 89% and 90% under Lighting and Background. Language perturbations act on the instruction rather than the image, so they fall outside vision tuning.

We additionally tuned X-VLA and OpenVLA-OFT on Lighting. X-VLA again regresses off target, by 3.0 pp on average, although its Lighting success was already 96.7%. The regression is therefore not a by-product of a large target gain. OpenVLA-OFT again retains, with +0.9 pp. A third target thus reproduces both the regression and its model dependence.

Q2 and W2. Training-seed and statistical stability.

We extended the three-seed analysis to all five models. Camera-view tuning was retrained with seeds 42, 43 and 44, and each run was evaluated on the same 560-episode subset. The table reports the mean off-target change in pp.

ModelSeed 42Seed 43Seed 44
X-VLA−3.96−5.00−7.08
OpenVLA-OFT+1.88+3.75+1.25
π0.5−0.83−1.67−1.04
NORA-1.5−10.62−7.50−6.67
InternVLA-A1.5−2.92−4.17−1.46

The sign is identical across seeds for every model, including the smaller regressions of π0.5 and InternVLA-A1.5. We also evaluated all three X-VLA seeds on the full 10,030 episodes. Their mean off-target changes are −4.7, −4.6 and −6.3 pp, each with an interval below zero, and their standard deviation across seeds is only 1.0 pp. For the ten targeted configurations in the paper, the sign of every mean off-target change is also unchanged when any single task family is removed.

Q3. Why DELTA-style preservation differs between X-VLA and OpenVLA-OFT.

Following the reviewer's suggestion, we fed the same 280 observations, covering all 40 tasks and seven perturbations, to each checkpoint and compared the visual tokens it passes to the language model with those of its base model. Default denotes our standard LoRA tuning of the vision encoder and connector.

ModelTuningCosine to baseMean off-target change (pp)
X-VLADefault0.69−4.6
X-VLADELTA-style0.99+1.2
OpenVLA-OFTDefault0.88+1.4
OpenVLA-OFTDELTA-style, weight 10.18−23.1
OpenVLA-OFTDELTA-style, weight 0.010.85+1.2

On X-VLA, DELTA-style keeps the features close to the base as designed. On OpenVLA-OFT, the same penalty weight fails to anchor them, so the degradation comes from this failure rather than from feature preservation itself. Lowering the weight restores both features and retention, which shows that the failure stems from optimization instability under a strong penalty rather than from an architectural limit. In both models, the checkpoint whose features drift most also retains worst, consistent with off-target regression tracking representation change. We will add this analysis to the paper.

W1. Unseen tasks and real-world settings.

LIBERO-Plus varies one perturbation at a time over the same tasks, which enables the paired per-dimension comparison at the core of VLA-Retain. Training and evaluation already use different perturbation instances. In future work, we plan to extend VLA-Retain to unseen tasks, other simulators and physical robots, applying the same paired protocol to test whether the cross-dimension effects hold beyond LIBERO.

Thank you for your additional experiments, some of them have closed my concerns, but some of them still not covered.

Globally, idea of the paper appeals to me, but I still can't understand how useful your findings are. Your results and openings should be supported by a wider range of experiments on different benchmars such as RoboTwin 2.0, Colosseum V2, LIBERO MAX, VLA Agena.

For now I prefer to keep my score.

We thank the reviewer for the follow-up and are glad that the additional experiments resolved several concerns.

We would like to briefly clarify the practical value of our findings. Target success alone can hide the cost of an update under other conditions. Camera-view tuning raises X-VLA's success under camera shifts by 69.4 pp, yet the tuned model succeeds in only 6.3% of episodes once blur is added to the camera shift. InternVLA-A1.5 loses 12.4 pp under robot-state perturbations even though its success on the original tasks stays nearly unchanged. VLA-Retain exposes such hidden costs before deployment, and it shows that balanced training data covering the expected shifts reduces them.

The paired protocol does not depend on LIBERO, and we plan to apply it to RoboTwin 2.0, Colosseum V2, LIBERO MAX and VLA-Arena in the future.

Reviewer_ETRUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces VLA-Retain, a benchmark for studying how visual tuning affects the cross-dimension generalization of vision-language-action (VLA) models. The authors evaluate five open-source VLA models across seven perturbation dimensions using LIBERO-Plus, comparing performance before and after targeted visual tuning. The results show that while tuning for a specific visual perturbation can substantially improve performance on the target dimension, it often degrades performance on other dimensions. The authors further investigate mixed training, alternative tuning strategies, and weight-merging methods, finding that these approaches can mitigate some regressions but do not provide a universally effective solution. Overall, the work highlights the trade-off between target-domain adaptation and cross-dimension knowledge retention in VLA models.

Critical strengths
  1. Practical relevance. The paper investigates how visual tuning affects the generalization of VLA models across different environmental perturbations, an important consideration for robust real-world deployment.
  2. Broad empirical evaluation. The study evaluates five open-source VLA models across multiple perturbation dimensions, systematically comparing target-dimension improvements with performance changes on non-target dimensions.
  3. Useful empirical findings. The results reveal potential cross-dimension regressions following targeted visual tuning and compare several mitigation strategies, including mixed training, alternative tuning configurations, and weight merging.
Critical weaknesses
  1. Limited representativeness of the tuning setup. The study focuses on tuning for individual perturbation dimensions, whereas practical fine-tuning often uses data covering multiple types of environmental variation. It remains unclear how well the findings generalize to such settings.
  2. Insufficient analysis of catastrophic forgetting. Performance degradation on the original domain after adapting to a new domain is a common phenomenon. The paper could more clearly distinguish general forgetting from cross-dimension negative transfer specific to different perturbations.
  3. Limited perturbation coverage and combinations. The benchmark does not fully capture realistic scenarios in which multiple perturbations occur simultaneously or interact with one another.
  4. Insufficient explanation of performance regressions. The paper observes model-dependent effects but does not fully explain whether these differences arise from architecture, pretraining, or the choice of trainable modules.
  5. Lack of a general mitigation strategy. The effectiveness of mixed training and weight-merging methods varies across models, and no approach consistently improves target performance while preserving robustness across all dimensions.
  6. Unclear relationship between perturbation types and tuning modules. The study primarily focuses on visual tuning, but different perturbations may require adapting different components. It remains unclear whether visual perturbations can be addressed by tuning the vision encoder alone, and whether models such as OFT require broader or joint tuning across modules and perturbation dimensions.
Questions for authors
  1. Tuning setup: Why does the study focus on tuning for individual perturbation dimensions? Would similar cross-dimension regressions persist when training data covers multiple perturbations?
  2. Catastrophic forgetting: How do the authors distinguish general domain forgetting from negative transfer specific to different perturbation dimensions? Could additional baselines help isolate these effects?
  3. Module selection: Do different perturbation types require tuning different model components? For example, can visual perturbations be addressed by tuning the vision encoder alone, while language or robot-state perturbations require adapting other components?
  4. Training paradigm: Does OFT's distinct training paradigm affect its ability to retain knowledge across dimensions? Would joint tuning across multiple perturbation dimensions improve its generalization?
  5. Real-world deployment: Since visual, linguistic, object-layout, and robot-state variations often co-occur in practice, do the findings and proposed mitigation strategies generalize to such compound perturbations?
4 comments
AuthorsOpenReview ↗

Response to Reviewer ETRU (Part 1)

We thank the reviewer for the constructive feedback. We have conducted additional experiments and refer to the relevant content in the paper to answer the following questions.

Q1. Single-perturbation tuning and multi-perturbation data.

We tune on one perturbation at a time to align with current targeted adaptation work [1, 2, 3], which adapts the vision module to a single shift. This setting also ties every off-target change to one known training shift and makes the cause of each regression easier to analyze.

Section 5.3 analyzes Mixed tuning on the full LIBERO-Plus training set, which covers five perturbation types. Mixed tuning raises overall success above Camera-only tuning for all five models, but regressions remain. InternVLA-A1.5 still declines on Language and Robot State, and both declines persist after subtracting its change on original LIBERO. Mixed tuning also gives up 13.1 pp of X-VLA's Camera gain relative to Camera-only tuning.

We further trained X-VLA on balanced Camera and Noise data. It achieves the highest overall gain we observed, +14.4 pp, and retains nearly all of both target gains, yet Language still drops by 3.1 pp. Broader coverage changes where regressions occur but does not remove them, so adaptation should be evaluated across all dimensions.

[1] W. Li et al. VLA Models Are More Generalizable Than You Think: Revisiting Physical and Spatial Modeling. arXiv:2512.02902, 2026.

[2] D. Yin and J. Zhang. VLAGuard: A Framework for Evaluating and Mitigating Physical Attention Hijacking in Vision-Language-Action Robots within Wireless Sensor Networks. arXiv:2608.01028, 2026.

[3] J. Zhang et al. Structure-Aware Robust Fine-Tuning: Defending Vision-Language-Action Robots against Physical Attention Hijacking. arXiv:2608.03231, 2026.

Q2. Distinguishing general forgetting from perturbation-specific transfer.

Section 5.2 evaluates every pair on original LIBERO and subtracts this change from each perturbation dimension. After Camera-only tuning, InternVLA-A1.5 changes by only −0.3 pp on original LIBERO but drops by 12.4 pp on Robot State, a regression that a standard forgetting metric would miss.

Following the reviewer's suggestion, we added a continued-tuning baseline that tunes X-VLA on unperturbed original LIBERO demonstrations with the same LoRA scope and budget. Compared with this baseline, Camera-only tuning still loses 7.9 pp on Sensor Noise after accounting for the original LIBERO change. Forgetting and perturbation-specific regression are therefore distinct effects, and VLA-Retain separates them. We will add this baseline to Section 5.2.

Q3. Which modules to tune for different perturbations.

For Camera-view, tuning the vision encoder alone is sufficient. Section 5.4 compares where the LoRA adapters are placed within the visual module. Our default setting, Dual LoRA, adapts both the vision encoder and the connector that maps visual features into the language model. Adapting only the vision encoder, Backbone-only in Figure 6, achieves the same +69.4 pp Camera gain on X-VLA and comes within 1.5 pp of the default on OpenVLA-OFT. Adapting only the connector falls well short on X-VLA.

Language and robot-state perturbations act on inputs that bypass the vision encoder, so our study focuses on visual perturbations as tuning targets. Updating only the vision module still changes behavior beyond it, because its output feeds the language and action modules under every condition. We tuned each of the five models separately on Camera-view and Sensor-noise, which gives ten tuned models. Compared with its base model, success under language perturbations drops in nine of them and success under robot-state perturbations drops in eight. VLA-Retain is designed to reveal such effects on perturbations that vision tuning does not target.

AuthorsOpenReview ↗

Response to Reviewer ETRU (Part 2)

Q4. OpenVLA-OFT and joint tuning.

To compare models under the same conditions, we restrict tuning to the vision module for all five models. Within this setting OpenVLA-OFT already retains best. Its mean off-target change stays positive after Camera-view, Sensor-noise and a new Lighting tuning experiment, and across three training seeds, while it gains 39.3 pp on Camera-view. OFT therefore keeps its robustness without tuning beyond the vision module.

Its training paradigm is a plausible reason for this behavior. OFT is the only model in our study that predicts continuous actions with a regression head, while the other four generate actions through flow matching, and it has the smallest visual-module share among the five models (Table 1). Its tuned visual features also stay closer to its base model than X-VLA's do, with a cosine similarity of 0.88 versus 0.69 on identical inputs. These differences are consistent with architecture shaping retention, in line with the model dependence reported in Section 5.1.

Joint tuning across perturbation types further improves OFT. Under Mixed tuning (Section 5.3), its overall success rises by 10.4 pp, with large gains on Camera-view, Sensor-noise and Background, and no dimension declines by more than 0.2 pp.

Q5. Compound perturbations.

We added a paired compound test on X-VLA. The same 80 task instances are run under four conditions, where blur is Gaussian blur from the sensor-noise family. Success rates (%) are shown below.

X-VLA modelCleanCameraBlurCamera and Blur
Base95.017.543.811.3
Camera-only98.892.527.56.3
Noise-only92.516.397.516.3
Mixed96.375.092.530.0
Balanced Camera and Noise95.085.087.530.0

Camera-only tuning reaches 92.5% under the camera shift alone but falls to 6.3% once blur is added, no better than the base model's 11.3%. Single-perturbation success therefore does not predict robustness to compound shifts. Mixed and balanced Camera and Noise tuning both reach 30.0%, so the benefit of broader coverage carries over to compound shifts, though compound success stays far below single-shift success. We will add this test to VLA-Retain.

W4 and W5. Causes of regression and general mitigation.

We agree that these are important directions. Our results suggest that regression depends on the tuning target and the tuned module as well as the architecture. Weight interpolation already improves retention and overall gain on both models we tested (Section 5.4), and balanced Camera and Noise data gives X-VLA the largest overall gain we observed. In future work, we plan to use VLA-Retain to study the separate effects of architecture and pretraining and develop a method that works across different architectures.

Thank you for the rebuttal and additional experiments. Some of my concerns have been partially addressed. However, my main concerns about the paper's novelty, depth of insight, and generalizability remain.

The main finding that targeted fine-tuning may degrade performance on other distributions is well known, and the paper offers little new insight beyond this observation. Moreover, the results are highly model-dependent and primarily limited to LIBERO-Plus, with insufficient validation across benchmarks.

Overall, the rebuttal does not fundamentally change my assessment of the paper's contribution, and I keep my original rating.

We thank the reviewer for the follow-up and for acknowledging the additional experiments.

Novelty and insight.

The well-known effect is forgetting on the source distribution, while VLA-Retain measures regressions on other perturbations that forgetting metrics miss. After Camera-view tuning, InternVLA-A1.5 changes by only −0.3 pp on the original tasks but loses 12.4 pp under robot-state perturbations. Compared with continued tuning on unperturbed data, Camera-view tuning still costs X-VLA 7.9 pp on Sensor Noise after accounting for the change on the original tasks. The benchmark also yields insights beyond this observation. Target gain does not predict retention, single-perturbation success does not predict robustness to compound shifts, and balanced training data covering the expected shifts gives the largest overall gain we observed while retaining nearly all target gains.

Model dependence.

Model dependence is itself a finding that motivates per-model evaluation. It is stable, since the sign of the mean off-target change is identical across three training seeds for every model. It also means that a retention method validated on one model can fail on another, as the same feature-preservation penalty that raises X-VLA's mean off-target success by 5.8 pp destabilizes the training of OpenVLA-OFT.

Generalizability.

Within LIBERO-Plus, our findings hold across five architectures, three tuning targets, three training seeds and compound perturbations. The paired protocol itself does not depend on LIBERO, and we plan to apply it to other benchmarks.

Reviewer_HDFROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates how targeted vision fine-tuning affects VLA robustness across different perturbation dimensions. The problem is meaningful, and the evaluation is systematic. However, the contribution is primarily empirical, with limited algorithmic novelty and mechanistic understanding.

Critical strengths
  1. Meaningful research problem. The paper highlights an overlooked issue in VLA adaptation: improvements on a target perturbation may compromise robustness under other conditions.

  2. Sound evaluation methodology. Paired comparisons between base and fine-tuned models, together with separate target and off-target metrics, provide a clear assessment of adaptation effects.

  3. Systematic empirical evaluation. The study covers five VLA models, seven perturbation dimensions, and over 10,000 episodes. Source-domain controls, bootstrap confidence intervals, and additional sensitivity analyses strengthen the empirical findings.

Critical weaknesses

1.The contribution is mainly a new evaluation perspective built upon existing benchmarks and adaptation techniques, rather than a fundamentally new methodology or explanatory framework.

  1. Experiments are confined to LIBERO simulation, with only two targeted adaptation domains and shared underlying task definitions across training and evaluation.

  2. Insufficient mechanistic understanding. Although the paper demonstrates cross-dimensional regressions, it provides limited insight into their causes or why retention strategies behave differently across architectures.

  3. Some conclusions require stronger statistical support. Only six of ten configurations show statistically supported negative mean off-target gains. The claim that mixed training cannot eliminate regressions also relies partly on very small performance drops without corresponding uncertainty estimates.

Questions for authors
  1. Could the authors investigate the mechanisms underlying cross-dimensional degradation, particularly the role of visual representation drift and interference with downstream action prediction?

  2. Would the findings generalize to other targeted perturbations, unseen manipulation tasks, or different robotic environments?

  3. Are the small regressions observed under mixed training statistically meaningful? Could the authors provide corresponding confidence intervals?

  4. What explains the substantially different effects of DELTA-style regularization across models? Could the results be sensitive to feature-matching locations or regularization hyperparameters?

1 comment
AuthorsOpenReview ↗

Response to Reviewer HDFR

We thank the reviewer for the constructive feedback. We have conducted additional experiments and refer to the relevant content in the paper to answer the following questions.

Q1. Representation drift and action prediction.

We fed the same 280 observations, covering all 40 tasks and seven perturbations, to the base and tuned checkpoints and compared both the visual tokens passed to the language model and the first predicted action chunk. Camera-view tuning lowers X-VLA's feature cosine similarity to the base model to 0.69. Although the action module is frozen, the predicted actions on identical inputs change as well. Among the off-target perturbations, Sensor Noise shows the largest shift in predicted position and also the largest success drop, 10.7 pp. Tuning thus changes not only the visual features but also the actions that the frozen downstream modules predict from them.

When DELTA-style preservation keeps the features close to the base, this interference shrinks as well. It raises X-VLA's feature cosine to 0.99, cuts the position and rotation deviation of its predicted actions by 30% and 44%, and turns its mean off-target change from −4.6 to +1.2 pp. We discuss OpenVLA-OFT in our response to Q4.

Q2. Generalization to other targets, tasks and environments.

We added Lighting as a third tuning target on X-VLA and OpenVLA-OFT. X-VLA again regresses off target, by 3.0 pp on average, although its Lighting success was already 96.7%, while OpenVLA-OFT again retains with +0.9 pp. A new target thus reproduces both the regression and its model dependence. A new compound test further shows that single-perturbation gains do not carry over. After Camera-view tuning, X-VLA succeeds in 92.5% of episodes under the camera shift alone but in only 6.3% when Gaussian blur is added, no better than the base model's 11.3%.

Training and evaluation already use different perturbation instances, and in future work we plan to extend VLA-Retain to unseen tasks, other simulators and physical robots with the same paired protocol.

Q3 and W4. Statistical support.

For each of the five models, we computed the change in success on each of the seven perturbation dimensions after Mixed training, together with paired 95% bootstrap CIs over the 40 task families. Of these 35 changes, five have intervals entirely below zero, all from two models.

ModelDimensionChange (pp)95% CI
NORA-1.5Language−4.29[−7.37, −1.20]
NORA-1.5Object Layout−4.26[−7.67, −1.01]
InternVLA-A1.5Robot State−3.16[−5.25, −1.13]
InternVLA-A1.5Language−2.93[−5.35, −0.60]
InternVLA-A1.5Background−2.51[−4.96, −0.31]

Under Mixed training, no dimension of X-VLA, OpenVLA-OFT or π0.5 has an interval entirely below zero, while NORA-1.5 and InternVLA-A1.5 retain clear regressions, another instance of model dependence. We will report these intervals in Section 5.3.

For the ten targeted configurations, the sign of every mean off-target change is unchanged when any single task family is removed. Three independent X-VLA training runs on the full benchmark also each give an off-target interval below zero.

Q4. Model dependence of DELTA-style regularization.

On identical inputs, DELTA-style keeps X-VLA's visual tokens close to the base, with a cosine similarity of 0.99. On OpenVLA-OFT, the same penalty weight of 1 fails to anchor them. Their cosine to the base falls to 0.18 and their norm shrinks to 0.43 of the base. OFT's degradation therefore comes from this failure, caused by optimization instability under the strong penalty, rather than from feature preservation itself.

The result is sensitive to the penalty weight. On the full benchmark, lowering OFT's weight from 1 to 0.01 moves its mean off-target change from −23.1 to +1.2 pp, on par with standard tuning at +1.4 pp, and restores a feature cosine of 0.85. Both models are matched at the same functional location, the visual tokens passed to the language model, and we plan to study matching at earlier layers of the vision encoder in future work.

W1. Contribution.

Robustness benchmarks evaluate fixed models, and adaptation studies typically report only target gains. VLA-Retain pairs each tuned model with its own base and separates target, off-target and source-domain changes. This exposes effects neither line of work can detect, such as target gains that hide off-target losses and regressions that forgetting metrics miss. Our new drift analysis further links retention to how far tuning moves the visual representation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.