acceptodds
Under review as a conference paper at ICLR 2027

LOOK: Latent Knowledge Is Overlooked, Not Overwritten, in Vision-Language-Action Models

Abstract

Failures of vision-language-action (VLA) models on concepts unseen in robot demonstrations are often attributed to semantic damage during action training.We find that target knowledge can be weakened yet remain useful for control: the surviving signal can be overlooked by the action pathway. We study this possibility with CrossBind, which measures target identification and closed-loop behavior separately. Across three action-pretrained backbones, an unfitted likelihood readout identifies placement targets on 0.56 to 0.69 of episodes against a 1/3 chance level. Yet same-checkpoint pairing in two task-adapted policies reveals only weak associations between correct identification and correct placement. This discrepancy motivates a semantics-to-action access bottleneck. RECALL tests this account by giving the retained signal an explicit path to action. With the backbone frozen, demonstration target indices and the original action loss train the action expert and interface; the inherited readout selects targets at inference. At fixed policy weights, replacing a location-only condition with readout-selected targets raises success from 0.215 to 0.438 on π0.5, from 0.056 to 0.722 on InstructVLA,and from 0.049 to 0.514 on X-VLA. These results show why preserving semantics must be accompanied by attention to its use: retained target knowledge can support behavior when given an effective route to action

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models,Language Grounding for Manipulation, semantic Readout from Pretrained VLMs

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_cwPROpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

his paper studies why vision-language-action models fail on concepts absent from robot demonstrations. CrossBind measures target identification separately from task execution. An inherited likelihood readout recovers useful target knowledge across three model families, but its correctness weakly predicts placement in two task-adapted policies. RECALL freezes the action-pretrained backbone and trains an interface and action expert with demonstration target indices. At inference, the inherited readout selects the regions supplied to the expert. A key strength is linking semantic measurement to controlled policy interventions. With policy weights fixed, replacing location-only inputs with selected targets raises success from 21.5% to 43.8% on π0.5, 5.6% to 72.2% on InstructVLA, and 4.9% to 51.4% on X-VLA. However, the main policy results use an earlier data recipe, and each configuration has one training run. Retained semantics can aid control, but the original failure mechanism remains less firmly established.

Critical strengths
  1. Useful separation of knowledge and behavior. The same-checkpoint analysis tests whether target identification predicts placement. This is more informative than comparing aggregate readout accuracy with task success.
  2. Clear intervention evidence. Five input conditions share policy weights and evaluation states. Redirected-target and separate object and destination controls confirm sensitivity to the selected referents.
  3. Consistent gains across three architectures. RECALL reuses inherited scoring without fitting a semantic probe. Each backbone benefits from readout-selected targets over location-only conditioning.
Critical weaknesses
  1. The strongest shortcut guarantees cover a different recipe. Full-factorial crossing supports the dataset audit, while the main policies use earlier placement-crossed data. This limits how directly those guarantees support the behavioral diagnosis.
  2. The original failure mechanism remains partly unresolved. The readout uses isolated crops, and RECALL trains a new target-conditioned pathway. These results establish useful retrieval, but also change visual binding demands relative to the original policy.
  3. Training stability is not established. Each configuration has one training run. Episode-level intervals describe evaluation uncertainty, but do not show whether the gains and weak baseline associations persist across independently trained checkpoints.
Questions for authors
  1. Do the behavioral findings persist when policies are trained on the full-factorial dataset?
  2. Does a full-scene readout yield the same weak association with the original policy's placement outcomes?
  3. How stable are the main within-policy gains and baseline associations across independent training seeds?
Reviewer_rUxnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether semantic failures in VLA models stem from lost knowledge or its ineffective use in action generation. Experiments on CrossBind, a controlled benchmark that separately evaluates semantic target identification and closed-loop execution, show that pretrained semantic knowledge remains recoverable after action training but is weakly associated with task success. The proposed RECALL method explicitly routes recovered target information to the action expert, improving task success across three VLA architectures.

Critical strengths
  1. By utilizing semantic readouts using likelihood scores, the paper provides a clear approach to distinguishing semantic knowledge retention from its utilization in action generation.

  2. CrossBind employs a systematic factorial design to eliminate dataset shortcuts, enabling more reliable evaluation of semantic binding.

  3. The paper provides detailed comparisons and ablation studies to validate CrossBind's design and examine the effectiveness of RECALL, strengthening the experimental evidence.

Critical weaknesses
  1. The task setting is overly simplistic. Identifying block colors and image contents only captures a limited range of visual understanding abilities, while semantic knowledge encompasses much broader capabilities that are difficult to evaluate in the current setting.

  2. RECALL relies on predefined candidate regions and target location annotations during training, limiting its applicability to diverse tasks and open-ended environments. This raises concerns about its generality as a method for leveraging semantic knowledge in VLM backbones.

  3. The lack of real-world robot experiments makes it unclear whether the findings and conclusions generalize to real-world deployment.

  4. Several figures, particularly Figures 3(b) and 4, suffer from layout or rendering issues that make them difficult to read and hinder understanding of the presented content.

Questions for authors
  1. Why were the three VLA models trained on different subsets rather than the full factorial CrossBind dataset? Could potential shortcuts remain in the actual training data? Further analysis is needed, such as examining associations between instructions, block arrangements, and image triples in each training set.

  2. According to Tables 4 and 5, InstructVLA achieves 72.2% task success with RECALL, despite readout accuracies of only 26.0% for picking and 68.8% for placement. Although evaluated on different episode subsets, this discrepancy raises questions about whether the improvements can be fully attributed to better utilization of retained semantic knowledge. Further analysis is needed to explain this phenomenon and assess the validity of the central conclusion.

Reviewer_LQgZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper suggest that VLA failures can be attributed to semantic signals that survive adaptation yet remain poorly accessed by action generation. They propose controlled experiments to validate this hypothesis and show that indeed task failures occur when semantic signals remain. Moreover, identifying semantic information is only weakly correlated with task success.

Critical strengths

While related work study VLA failures due to damage to pretrained semantics during action adaptation, the paper suggest an interesting decomposition of VLA failures: when a robot fails to follow an instruction, is it because the object information is missing, or because the action-generation mechanism fails to use knowledge that is still available? This is an interesting view, and the approach could lead to better understanding VLA mechanisms.

The paper conducted controlled experiments, showing that semantic information of objects and targets degrades but still remain under adaptation. Furthermore, they showed only a weak correlation between perceptive correctness and task success.

Critical weaknesses
  1. While the readout experiment provide useful information, the readout method may introduce potential issues that are not properly addressed. First, a binary classification readout of choosing the highest probability among fixed locations is not feasible at real-life scenarios, second it is biased towards one correct answer out of all the possibilities.

  2. The recall method completely shifts the task. Since ground truth information is provided, the model may learn shortcuts that help solving the task. This is different than methods that encourage preserving semantic information while keeping the inference path similar. Therefore, the results are less surprising, and the inference modification suffers from the readout issues mentioned in (1) - is this method applicable for real-life scenarios where there isn't a closed set of options?

  3. The experiments are limited only to the controlled experiments settings. It will be interesting to see results on more realistic robotic settings.

  4. The references section is missing, so there is no way to link refs. Moreover, the required "AI USE STATEMENT" section, and code for reproducibility are missing.

Questions for authors

How do you suggest to study existence of semantic information in less "controlled" environments? Can this method generalize to real life scenarios?

Reviewer_A8GMOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper argues that VLA model failures on unseen concepts stem not only from semantic damage during action training, but from an access bottleneck: useful knowledge survives yet is poorly routed to action generation. The authors introduce CrossBind, a controlled pick-and-place benchmark with factorial shortcut removal, and show via unfitted likelihood readouts that three VLA backbones (π0.5, InstructVLA, X-VLA) retain above-chance target identification (0.56–0.69 vs. 1/3). Episode-level pairing reveals weak behavioral association (φ ≈ 0.03–0.06). RECALL, their intervention, routes readout-selected targets to a learned action interface with the backbone frozen, yielding large gains (e.g., 0.056→0.722 on InstructVLA) over location-only baselines at fixed policy weights.

Critical strengths

The “overlooked vs. overwritten” distinction is a genuinely important reframing. The standard narrative attributes VLA generalization failures to catastrophic forgetting of VLM semantics. Demonstrating recoverable-yet-unused knowledge across three architecturally diverse lineages shifts the burden of proof and opens a new design axis. Second, the within-policy condition replacement methodology (Table 4) is unusually clean. Comparing self vs. loc-only at fixed weights, instructions, and evaluation states isolates the contribution of semantic identity from spatial cues, providing strong evidence that the retained signal has control value rather than merely being decodable in a non-behavioral sense.

Critical weaknesses

The benchmark is extremely narrow: one tabletop task, discrete block-on-image placement, and a small fixed candidate set. Whether the access bottleneck accounts for failures in richer settings (cluttered scenes, longer horizons, continuous targets) is entirely open, and the factorial design that makes CrossBind clean also limits ecological validity. Second, the weak-association evidence (φ = 0.03, p = 0.69 for π0.5) is not statistically significant and could reflect readout noise rather than a genuine bottleneck. The paper’s causal claim rests more on RECALL’s gain than on the pairing analysis, but the pairing is presented as a key motivating finding. A non-significant result with n = 288 leaves room for the simpler explanation that the readout is too noisy to predict episode outcomes, rather than that semantics is systematically overlooked.

Questions for authors

How sensitive is the readout accuracy to the captioning prompt template and scoring procedure? Small wording changes can shift VLM likelihoods substantially. Have you tested alternative templates, and would a meaningfully different prompt change your conclusions about how much semantics survives? Several VLAs already co-train with image-text objectives (RT-2, π0). Your Section 2 mentions these, but Table 2 does not break out whether co-trained checkpoints show a smaller access bottleneck than purely action-finetuned ones. Can you comment on whether co-training already partially addresses the bottleneck, and if so, what RECALL adds beyond what co-training provides? RECALL requires a fixed, pre-mined candidate set and ground-truth target indices at training time, which limits applicability to enumerable environments. What is your view on extending the approach to continuous, open-world settings where candidate regions cannot be pre-specified? Is the contribution primarily diagnostic, or do you see a path toward a practical method?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.