- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This manuscript investigates how limited language diversity in teleoperated robot learning datasets affects VLA language representations. It first extends Wanna et al. (2026) investigation to the larger oxe-magic-soup mixture (Section 3). It then studies the evolution of language embeddings across the LLM -> VLM -> VLA pipeline. The main conclusion is that embeddings don't drift much from LLM -> VLM, but do so from VLM-VLA, and that VLA adaptation over-specializes language representations to the narrow training vocabulary. The authors conclude with recommendations to practitioners.
Critical strengths
Systematic comparison of LLM -> VLM -> VLA: a critical strength of this paper is the comparison of the language embeddings (untied vs. tied) across the LLM -> VLM -> VLA pipeline for two different VLA architectures. I think this is consequential because if there was a lot of drift from LLM -> VLM, then I would argue the findings are quite limited.
Critical weaknesses
No causal evidence: the analyses use released checkpoints only, so the authors cannot attribute the embedding distortion to limited language diversity directly. There is also a confounder with traniing length (>20 epochs for VLA post-training but only 1-2 epochs for VLM stage). If there were comparisons between training with the original oxe-magic-soup dataset, and a language augmented dataset, that would yielded a much stronger paper. Right now, the only evidence we have is that VLA post-training a VLM overfits on the small language diversity in the dataset, which is expected.
Findings predictable and limited novelty: the findings that VLA language-embeddings change a. lot when we overfit to a dataset of limited diversity is expected. The recommendations (more diverse data, staged training, etc.) largely restate prior work (e.g., Wanna et al. 2026) and are never evaluated. The paper extends Wanna et. al's work in Section 3, and the paper offers little actionable insight beyond existing literature.
Questions for authors
-
Did you run any controlled fine-tuning experiments on small datasets (like just bridgev2) comparing original instructions to paraphrased / diversified ones? does this reduce the LLM->VLM->VLA sensitivity?
-
Is there a way to separate the effect of training epochs (over 20 for VLAs vs. 1-2 for the VLM) from the effect of limited language diversity? Otherwise, its possible that we just tremendously overfit the VLM, rather than the language diversity itself being the cause.