acceptodds
Withdrawn from ICLR 2027

Limited Linguistic Diversity in VLA Training Distorts Language Representations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language-action (VLA) models inherit rich language representations from pretraining, yet remain brittle to variation in natural-language commands. We study how this brittleness is reflected in the representations themselves. We first show that the Open X-Embodiment Magic Soup mixture relies on a strikingly small and repetitive vocabulary that does not cover the natural variation in robot commands: even simple paraphrases of benchmark instructions often contain words absent from it. We then trace the language embeddings of OpenVLA and -FAST through vision-language and robot training. Robot training changes the embeddings considerably more than vision-language training does, and in both models these changes carry a direct imprint of the data. For OpenVLA, whose training data is public and whose embeddings are untied, only tokens that appear in the training commands change; among them, tokens appearing in more varied commands change less, while tokens seen in only a few commands can still be substantially reshaped. For -FAST, whose training data is not public, the most heavily trained tokens stand out clearly enough from the rest of the vocabulary that we can partially recover its training vocabulary. Rather than adapting the language space as a whole, robot training thus specializes it to the narrow set of commands observed during training. This specialization also shows in how robust VLA models are to small input corruption: a single-character typo in an object name, which preserves the meaning of a command but changes its tokens, alters VLA representations substantially more than those of the corresponding language and vision-language models, suggesting that VLA representations depend more on the exact tokens than on the intended meaning. Together, these results link the narrowness of robot language data to changes in the VLA representations.

Primary area:
interpretability and explainable AI
Keywords:
vision-language-action, linguistic diversity, token embeddings

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FhbDOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This manuscript investigates how limited language diversity in teleoperated robot learning datasets affects VLA language representations. It first extends Wanna et al. (2026) investigation to the larger oxe-magic-soup mixture (Section 3). It then studies the evolution of language embeddings across the LLM -> VLM -> VLA pipeline. The main conclusion is that embeddings don't drift much from LLM -> VLM, but do so from VLM-VLA, and that VLA adaptation over-specializes language representations to the narrow training vocabulary. The authors conclude with recommendations to practitioners.

Critical strengths

Systematic comparison of LLM -> VLM -> VLA: a critical strength of this paper is the comparison of the language embeddings (untied vs. tied) across the LLM -> VLM -> VLA pipeline for two different VLA architectures. I think this is consequential because if there was a lot of drift from LLM -> VLM, then I would argue the findings are quite limited.

Critical weaknesses

No causal evidence: the analyses use released checkpoints only, so the authors cannot attribute the embedding distortion to limited language diversity directly. There is also a confounder with traniing length (>20 epochs for VLA post-training but only 1-2 epochs for VLM stage). If there were comparisons between training with the original oxe-magic-soup dataset, and a language augmented dataset, that would yielded a much stronger paper. Right now, the only evidence we have is that VLA post-training a VLM overfits on the small language diversity in the dataset, which is expected.

Findings predictable and limited novelty: the findings that VLA language-embeddings change a. lot when we overfit to a dataset of limited diversity is expected. The recommendations (more diverse data, staged training, etc.) largely restate prior work (e.g., Wanna et al. 2026) and are never evaluated. The paper extends Wanna et. al's work in Section 3, and the paper offers little actionable insight beyond existing literature.

Questions for authors
  1. Did you run any controlled fine-tuning experiments on small datasets (like just bridgev2) comparing original instructions to paraphrased / diversified ones? does this reduce the LLM->VLM->VLA sensitivity?

  2. Is there a way to separate the effect of training epochs (over 20 for VLAs vs. 1-2 for the VLM) from the effect of limited language diversity? Otherwise, its possible that we just tremendously overfit the VLM, rather than the language diversity itself being the cause.

Reviewer_FTMCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper asks how fine-tuning a VLM into a VLA reshapes the language representations the model inherited from pretraining, given how narrow the language in robot training data is. It makes four contributions:

  1. A full linguistic audit of the OXE Magic Soup+ mixture used to train OpenVLA. This extends Wanna et al. (2026) from 4 datasets to all 27. The mixture has about 969K episodes but only about 182K unique commands and 4,477 unique words, many of them typos. As a sanity check, 49% of LIBERO-Para paraphrases contain at least one token absent from the mixture.
  2. Embedding-drift analysis across LLM → VLM → VLA for OpenVLA (untied embeddings) and π0-FAST (tied). VLA training moves embeddings much more than VLM training does. In OpenVLA only the roughly 11% of tokens that appear in training commands change. How far a token moves depends non-monotonically on its frequency: tokens that appear in many distinct commands move less than mid-frequency tokens.
  3. A vocabulary-recovery method for π0-FAST, whose training data is undisclosed. Softmax pressure pushes almost all tied embeddings along one shared direction (98% have cosine > 0.8 to it). Tokens that deviate from this direction mark the input vocabulary. This lets the authors estimate an active vocabulary under 7,000 tokens and surface plausible undisclosed training words: the prompt template, groceries, and failure or episode vocabulary.
  4. A representation-robustness probe. A single-character typo in the object noun changes VLA last-token representations, both in direction (cosine) and in geometry (CKA), far more than in the matching LLM and VLM. Appendix J links this drift to failures on LIBERO with OpenVLA: within-task AUC is 0.75 on LIBERO-Goal (paraphrases) and 0.58 on LIBERO-Object (typos).
Critical strengths
  • Timely question, useful framing. Many papers show, behaviorally, that VLAs are brittle to language. This one goes inside the model and connects that brittleness to properties of the training data. The “active vocabulary” concept is simple and practically useful.
  • The data audit is a solid community resource. Table 4’s per-dataset breakdown is a concrete reference. Kuka, for example, is 12.7% of the mixture but has a single command, and TOTO has one word. The LIBERO-Para coverage check (49% of paraphrases contain unseen tokens) makes the problem concrete.
  • The tied-embedding analysis of π0-FAST is the most original part. Separating input-side updates from uniform softmax pressure by measuring deviation from a common direction is clever. Recovering plausible training vocabulary from weights alone (Table 9) is an interesting result in its own right, with implications for data auditing and data leakage. The RANSAC-style check in footnote 3 is a good robustness touch.
  • Comparing two architectures with different embedding-sharing choices is well motivated. The authors correctly interpret what each embedding matrix can and cannot show (Sec. 4.1, Appendix E).
  • Appendix J is a welcome step toward behavioral grounding. Its design is sensible: paired trials from identical initial states, restriction to tasks that succeed under the original instruction, and within-task rank normalization. It also compares against LIBERO-Para’s own keyword-distance metric.
  • Limitations are discussed candidly, and the AI-use disclosure is clear.
Critical weaknesses
  1. Several headline findings follow from the architecture or are already known.
  • That only in-vocabulary tokens change in untied input embeddings is a property of sparse gradients, as the paper acknowledges (“by construction”).
  • That softmax pressure shifts all output or tied rows along a shared direction is the known representation-degeneration effect (Gao et al., 2019).

These are correctly described, but they make up much of Sec. 5.1–5.2, so the novel empirical content is thinner than the framing suggests. The paper would benefit from separating more clearly what is mechanical from what is a genuine finding.

  1. The title and abstract make causal claims; the evidence is correlational. “Limited linguistic diversity … distorts language representations” is causal, but every analysis uses released checkpoints, and Sec. 7 itself says the results are not causal claims. Several confounds remain open:
  • VLM and VLA stages differ in epochs (1–2 vs. 27), learning rate, objective, and data modality.
  • Larger VLA drift could therefore reflect training length rather than lexical narrowness.

The obvious controlled experiment is missing, and it is affordable at LIBERO scale. For example, fine-tune OpenVLA or π0-FAST on a LIBERO suite with and without paraphrase augmentation, and/or with frozen or regularized embeddings, then measure embedding drift, typo sensitivity, and success rate. This would test the causal claim and the Sec. 6 recommendations, which are currently untested.

  1. The “context diversity, not frequency” conclusion (Sec. 5.2, Fig. 2) is under-controlled.
  • The number of unique commands is strongly correlated with frequency. The paper needs a regression or partial-correlation analysis showing diversity explains drift after controlling for frequency.
  • An optimizer-level explanation is not ruled out. With Adam, rarely updated rows have small second-moment estimates, so their occasional updates are relatively large in normalized terms. Consistent gradients on very frequent tokens behave differently. This alone could produce the non-monotonic curve.
  • The paper should state the optimizer and weight-decay settings. Notably, the 10.7% changed-token figure implies no decoupled weight decay on the embeddings.
  • The footnote concedes that unique-command counts may poorly proxy linguistic context, since many commands are near-duplicates. A better measure would help, for example the number of distinct n-gram contexts or the entropy of neighboring tokens.
  • Table 8’s single-command tier contains tokens like _https, _posterior, and _efficiently. This suggests a few long, unusual commands repeated many times, which is an artifact worth characterizing rather than evidence about context richness.
  1. The robustness probe (Sec. 5.3) lacks a reference point. That VLA final layers change more under a typo does not by itself show heightened sensitivity to meaning-preserving noise. VLA late layers are specialized toward action prediction and may simply amplify any input change. Two controls are needed:
  • A meaning-changing perturbation (a different object noun) as a reference scale. The key quantity is the ratio of typo drift to semantic-change drift: does the VLA treat “pxt” as more like “pot” or more like “towel”?
  • A synonym or paraphrase condition.

Further concerns:

  • Raw cosine similarity of LLM hidden states is known to be dominated by a few outlier dimensions and anisotropy. Near-unity cosine for LLMs and VLMs may therefore be uninformative. Mean-centered or standardized activations should be used.
  • Inputs also differ across stages: no image for the LLM, different prompt templates. This should be stated explicitly in the main text.
  • The CKA caveat in Appendix I is welcome but somewhat undermines the CKA panel of Fig. 4.
  1. Behavioral validation is limited.
    Appendix J covers only OpenVLA, LIBERO-fine-tuned checkpoints rather than the OXE-trained models analyzed in the main text, and a typo AUC of 0.58 is weak. The n reported for LIBERO-Goal (390) also needs reconciling with the stated inclusion criterion. Given its importance, a condensed version of this appendix belongs in the main paper.

  2. The π0-FAST vocabulary-size estimate rests on an unverified assumption.
    The estimate assumes |N_i|/|N_o| = |S_i|/|S_o|, which is not justified. The method could be validated on a setting with known ground truth, for example a tied-embedding model (PaliGemma) briefly fine-tuned on a known command set, by reporting precision and recall of the recovered vocabulary.

  3. Scope. Only two autoregressive VLAs are studied. Diffusion and flow-matching action heads (π0, GR00T) and models co-trained with VLM data (π0.5, ChatVLA) are excluded. The exclusion is reasonable for cleanliness, but it limits how far the conclusions generalize. Co-trained models would be a natural test of the recommendations.

Questions for authors

Minor issues

  • Kim et al. 2026a and 2026b are the same LIBERO-Para paper cited twice.
  • The paper alternates between “Magic Soup” and “Magic Soup+”.
  • Sec. 4.2’s claim that chunked prediction gives language a “less frequent gradient signal” is speculative and unsupported.
  • Fig. 12 (Appendix H) has no explanation of the axes or of the color coding (dashed vs. solid boxes).
  • The Fig. 13 caption is terse and grammatically incomplete.
  • Table 3 should note that OpenVLA overwrites the last 256 vocabulary rows with action tokens (31,743 vs. 32,000 tokens in Table 7).
  • The result that LLM → VLM changes 95.8% of input tokens, but only slightly, deserves a sentence of interpretation.
  • Appendix B’s claim that frequent words become “more differentiated” should be compared against a matched random token set, not only against “all changed tokens.”
  1. What optimizer and weight-decay settings were used in each stage? Can the frequency/diversity curve be reproduced in a small controlled fine-tune where the number of steps is matched across conditions?
  2. How does typo-induced drift compare with drift induced by replacing the object with a different valid object?
  3. Can the π0-FAST vocabulary-recovery method be validated on a model with known training data?
  4. Have you tried freezing the language embeddings during LIBERO fine-tuning? Even a single result would substantially strengthen Sec. 6.
Reviewer_oMrNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how the narrow language of robot training data affects the language representations of VLAs. An audit of the OXE Magic Soup mixture shows a very small, repetitive vocabulary. Tracing token embeddings from LLM to VLM to VLA in OpenVLA and π0-FAST, the authors find that VLA training mainly reshapes the small set of tokens seen in training. For π0-FAST, the training vocabulary can even be partially recovered from the weights. Finally, a single-character typo in an object name changes VLA representations much more than LLM/VLM representations.

Critical strengths
  1. This paper provides an original, representation-level angle on a known problem. Language brittleness in VLAs is usually studied behaviourally. Looking directly at what VLA adaptation does to the inherited embeddings is novel.
  2. The idea of recovering π0-FAST's undisclosed training vocabulary from the weights is particularly creative.
Critical weaknesses
  1. The causal claim in the title is not tested. All results come from released checkpoints. The VLM and VLA stages differ in epochs, objective and data, which the paper itself acknowledges, so the drift cannot be attributed to linguistic diversity. Neither a controlled fine-tuning run nor a comparison with a VLA trained on richer language (e.g. π0.5) is provided, so the recommendations in Sec. 6 are untested.
  2. In the typo probe, VLAs may simply be more sensitive to any change in the command, and no semantic-change control or unrelated-command baseline is given. The "diversity rather than frequency" claim is read off density plots, without a statistic that separates the two strongly correlated factors.
  3. The link to behaviour is weak. The only behavioural evidence (App. J) uses different, LIBERO-fine-tuned checkpoints and covers OpenVLA only.
Questions for authors
  1. In Sec. 5.3, do the LLM, VLM and VLA see the same prompt template and image? How are the typos chosen?
  2. Could you add a semantic-change control and an unrelated-command baseline to Fig. 4?
  3. Would a small controlled run (e.g. fine-tuning OpenVLA on LIBERO with vs. without paraphrase augmentation) support the title claim?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.