acceptodds
Under review as a conference paper at ICLR 2027

UAM: A Dual-Stream Perspective on Forgetting in VLA Training

Abstract

Vision–language–action (VLA) models typically adapt a pretrained vision–language model (VLM) to robot control to transfer broad multimodal knowledge. Yet action fine-tuning can erode the competence that motivates this initialization, a loss we call the embodiment tax. Can a VLA reduce this loss while allowing its semantic backbone to adapt, without auxiliary vision–language co-training? Inspired by the recognition and visuomotor pathways of biological vision, we investigate a representational-bottleneck hypothesis: a semantic pathway that must also supply control-specific visual features may face competing adaptation demands. We study this hypothesis through the Unified Action Model (UAM), which couples a semantic expert, a parallel visual Dorsal Expert, and an action expert using mixture-of-transformers. Starting from aligned understanding–generation pretraining, we compare Dorsal initialization, visual versus query inputs, and visual-dynamics supervision. The strongest tested design pairs generative initialization with future-observation prediction from robot trajectories, providing a complementary learning target while all three experts remain trainable. Under matched initialization and action-training settings, UAM retains 96.7% of pretrained scores across four reported multimodal metrics, versus 54.8% without the Dorsal pathway, while also improving OOD action performance by 35.0% with no auxiliary vision–language replay, VLM freezing, or gradient blocking is used during adaptation. After training on only 3,000 real-robot trajectories, UAM achieves the highest average success among the compared variants on semantic generalization tasks involving unseen objects, novel target compositions, and instruction variation. Together, these results support aligned generative pretraining and visual-dynamics supervision as a means to improve the retention–control trade-off in unified multimodal models. Demos can be found at https://sites.google.com/view/unified-action-modelour anonymous website.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied AI, Unified Models, VLA

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pZ7pOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies forgetting of pretrained multimodal capabilities during vision-language-action adaptation. Building on BAGEL and the three-expert formulation inherited from BagelVLA, UAM couples semantic, generative Dorsal, and action experts through MoT. The Dorsal branch receives future-observation prediction supervision and provides complementary visual features for action generation, while all three experts remain trainable without gradient blocking or auxiliary vision-language replay. The main contribution is a controlled study of how Dorsal initialization, input modality, visual-dynamics supervision, and semantic-expert trainability affect retention and control. Validation includes multimodal benchmarks, CALVIN and RoboTwin simulations, and real-robot semantic generalization tests after training on 3,000 trajectories. The strongest configuration retains 96.7% of pretrained scores across four matched multimodal metrics, compared with 54.8% without the Dorsal branch, while improving control and generalization to unseen objects, novel object–target combinations, and instruction variations.

Critical strengths
  1. Systematic ablation study

    The paper examines how Dorsal initialization, visual versus query inputs, future-observation supervision, and semantic-expert trainability affect multimodal retention and robot control. These comparisons provide useful empirical evidence about forgetting during VLA adaptation.

  2. Joint evaluation of retention and control

    The paper evaluates both pretrained multimodal capabilities and embodied performance, showing that strong retention can coexist with improved control while the semantic expert remains trainable, without auxiliary vision-language replay.

Critical weaknesses
  1. The embodied evaluation is relatively limited

    Experiments on VLABench, LIBERO-Pro, or LIBERO-Plus would help validate broader generalization and robustness.

  2. The analysis of forgetting is not very detailed

    The ablations and attention maps show improved retention, but do not fully explain why the Dorsal expert reduces forgetting.

Questions for authors
  1. What technical changes does UAM introduce beyond BagelVLA, apart from the study of forgetting and retention?
  2. What evidence beyond the attention maps shows that the Dorsal expert reduces interference with the semantic expert during action training?
  3. Are additional results available on VLABench, LIBERO-Pro, or LIBERO-Plus to show whether the improvements extend beyond the reported tasks?
Reviewer_s9vHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the loss of pretrained vision-language understanding during VLA training. It evaluates UAM, which adds a visual Dorsal pathway alongside a semantic expert and an action expert, using generative initialization and future-observation prediction to support robot control. Through design ablations, multimodal benchmarks, simulation, and real-robot tests, the paper reports strong retention of VLM capabilities alongside improved manipulation performance, including on tasks with unseen objects and varied instructions.

Critical strengths
  1. The paper addresses a consequential problem and measures it directly. Under comparable robot-training settings, UAM retains substantially more of its pretrained multimodal understanding than the No-Dorsal baseline while keeping the semantic expert trainable.

  2. The evaluation goes beyond retention scores. The authors compare Dorsal designs in simulation and on real robots, including tasks with unseen objects, new object–target combinations, and varied instructions. UAM’s strong results make the proposed design practically interesting, even though the source of its robot gains needs clearer analysis.

Critical weaknesses

Although UAM substantially improves the retention of VLM understanding, the experiments do not show that this retention explains its robot performance gains. Dorsal variants with similarly high retention achieve markedly different OOD success rates. In particular, adding future-observation supervision raises retention only from 94.2% to 96.7% but yields much larger gains on several real-robot OOD tasks. This suggests that the Dorsal pathway’s predictive training may be an important driver of those gains, beyond the preservation of VLM benchmark performance. Comparisons with the No-Dorsal baseline also change model capacity. The paper reports both retention and control results, but does not disentangle the contribution of preserved semantic understanding from that of the added pathway, its initialization, and its supervision.

Questions for authors

Please address the concerns raised in the Critical Weaknesses section.

Reviewer_S8L1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduced the concept of embodiment tax, which is about finetuning on VLMs for robotics tasks can degrade the base model's vision and language understanding capabilities. The authors proposed UAM, which couples a semantic expert, a visual Dorsal Expert, and an action expert via MOT structure. The experiments show that UAM preserves most of the base model's VL capability while also improving robotic manipulation and OOD generalization.

Critical strengths
  • The paper addresses a fundamental limitation of prior approaches and makes a meaningful contribution to the field.
  • The proposed approach introduces a novel perspective, and it is technically sound, and the design choices are well-justified.
Critical weaknesses
  • The UAM introduces an additional 7B Dorsal Expert, which increases the model from ~9B to ~16B parameters. It is kinda unclear whether the performance gain comes from dual-stream specialization or increased capacity. This might limit the central representational bottleneck hypothesis.
  • The paper provides evidence for semantic experts focusing on object recognition while Dorsal expert focusing on control-based visual representations, yet the evidence is in Fig 6 the qualitative visualization, however it doesn't seem to be a solid way to establish that the two pathways encode functionally different visual information or that this specialization causes reduced forgetting.
  • It is unclear whether the benefits generalize to other pretrained VLM and generative backbones.
Questions for authors

Could the authors provide stronger evidence that the improvements come from dual-stream specialization rather than increased model capacity, and the two pathways learn distinct representations beyond the qualitative visualizations in Fig. 6, and that these benefits generalize to other pretrained backbones?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.