- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks whether language-conditioned robot policies follow instructions or infer the task from the scene. Holding the scene fixed, the authors change the instruction in one of three ways: they swap the target for another visible object, replace object nouns with random nouns, or substitute a random sentence. They evaluate two VLAs (pi0.5, GR00T N1.7) and two world-action models (FastWAM, LingBot-VA) on LIBERO, RoboTwin 2.0, and six real-world scenes.
The paper makes three claims. First, under in-scene target replacement, policies overwhelmingly act on the original target: 86-98% of LIBERO-Object trials and 81-94% of real-world trials. Second, the failure isn't simple language insensitivity. Perturbations reduce success rates and change intermediate action readouts from a layerwise "action lens," and linear probes recover the instructed target from instruction-token representations with near-perfect accuracy. Third, the authors offer correlates of why encoded language fails to control action: a modest attention-based instruction contribution to the action expert, a qualitative example where target-token attention stays on the original object, and UMAP and shared semi-NMF analyses showing a post-fine-tuning shift from target-organized to scene-organized representations. They call this a "representation-to-control grounding gap." Validation is behavioral (closed-loop rollouts) plus representational analyses on 60 LIBERO and 36 real-world scene-target conditions.
Critical strengths
-
The behavioral evaluation is clean and broad. Interventions are tightly matched, with fixed initial states and simulator seeds, and with fixed generation noise in the action lens. The study spans both VLA and WAM paradigms and includes real hardware. Extending the target-replacement failure to world-action models is a useful finding, because it shows that video-prediction objectives alone don't fix target selection. This is the main reason the paper is informative despite overlap with LIBERO-CF and LangGap.
-
The question is well posed, and the authors frame their claims carefully. The paper separates language sensitivity, information accessibility, and behavioral adherence, and it consistently avoids claiming causality it hasn't shown. That separation is a useful conceptual contribution for evaluating VLAs, even though the mechanistic evidence doesn't yet fully support it.
Critical weaknesses
-
The "encoded" half of the thesis rests on a probe that is close to guaranteed to succeed. The probes read mean-pooled hidden states at the instruction-token positions, and the instruction contains the target noun, so recovering it there is expected at nearly any layer of nearly any model. The base checkpoint's 100% accuracy is consistent with this. The paper cites Hewitt & Liang but includes none of the controls that work recommends: no control tasks, no layer-0 or random-init baselines, and no held-out-noun splits. With N = 60, D in the thousands, and an unspecified train/test split, accuracy says little. The informative test would probe the action pathway (action-expert or action-query states) to see whether target identity reaches the computation that produces actions. This matters to my assessment because "encoded but not in control" is the paper's central claim and main point of novelty over prior behavioral work, and as tested it may reduce to "the word appears in the prompt."
-
Part III asks "why" but answers only with correlates, and the diagnostics are never validated against a policy that does follow instructions. The attention-contribution measure attributes contribution by source-token position. In pi0.5's bidirectional prefix, instruction-position states already carry visual information, and the ~20% share comes from roughly 15 tokens against hundreds of image tokens, so calling it "weak" depends on an unstated baseline. The visual-attention claim rests on one qualitative example, the action lens on a single rollout per model, and the UMAP and semi-NMF analyses on 60 points with an unbalanced scene x target design. No interventional test is reported, such as activation patching between target conditions or instruction-token knockout or amplification. There is also no positive control, such as a model fine-tuned with multiple targets per scene, to show that the diagnostics track instruction following. Without these, readers can't tell whether the internal signatures explain the failure or are generic properties of fine-tuned VLAs. Prior work (Grant et al.; VLA-Trace) already reports decodable-but-unused prompt information, which raises the bar for what this paper needs to add.
Questions for authors
-
How were the probe train/test splits constructed? Does target accuracy hold when entire target nouns are held out, and when probing action-expert states rather than instruction-token states?
-
Did the real-world training data include multiple instructed targets per scene? If so, the real-world failure is considerably stronger evidence than LIBERO-Object, where each scene has exactly one training task and scene identity predicts the target perfectly. If not, both settings share this identifiability confound, and the failure is partly expected from the data.
-
How is "Tend" (a committed approach) determined: automatically or by human judgment, and with what agreement? How many rollouts per condition were run for the LIBERO target-replacement experiment?
-
Why are e_ij vectors summed across layers before taking the norm (Eq. 2), rather than computed per layer as in Kobayashi et al.? Please also explain the 1.7% instruction contribution for pi0.5 on RoboTwin.
Suggestions that would substantially strengthen a revision:
-
Run activation patching that transplants instruction-token states from target-B runs into target-A runs, layer by layer, to locate where (if anywhere) behavior can be redirected.
-
Add a positive-control model trained with several instructed targets per scene, and show whether the diagnostics (instruction contribution, scene-versus-target organization) shift as instruction following improves.
-
Give a quantitative attention-alignment metric across all replacements, plus a directional action-lens metric (for example, predicted displacement toward the new versus the original target) with a reference scale.
2 comments
Gentle Reminder
Hello! Thank you for your valuable suggestions. We’ve addressed the issues raised in your review and updated the revised version accordingly. If you think any part could use additional clarification or expansion, we’d be delighted to discuss.
Kind regards,
The Authors
Thank you for the detailed review and recognition of our controlled evaluation and conceptual distinctions. We address related comments below.
On the fully crossed real-world set, we train target probes on five scenes and test on the sixth. For scene probes, we train on five targets and test on the sixth. Each split retains all output classes. Probes are fitted per layer (Fig. 3), testing transfer across the other factor rather than unseen target classes.
We added probes on visual-token states and action-token states within the action expert, rather than language-token states, using 60 matched LIBERO-Object conditions. Layer-wise ridge probes select regularization by five-fold cross-validation within training data. Accuracies average over 18 layers and six class-stratified train/test splits:
Target information is recoverable in the fine-tuned action expert.
Our probe tests accessibility, not improvement over lexical embeddings. Randomly relabeling target categories only permutes output classes, leaving the linear task unchanged; that word-type control gives no informative selectivity comparison. Decoding may reflect lexical retention. We combine it with organizational and behavioral evidence.
Original real-world training used one target per scene, with 50 trajectories per task across six scenes. We added two controls to compare successful and unsuccessful instruction following:
LIBERO-Object control: Training with two targets per scene enables instruction-conditioned selection between trained targets. On held-out scene–target combinations, however, the policy selects a scene-trained target although the requested target and manipulation were demonstrated elsewhere. Internal analyses of these failures retain the original qualitative representational patterns.
Real-world positive control: Every task is demonstrated in every training scene, with 50 demonstrations per scene–task pair. Success is close to 100% in training scenes, 43.6% in similar unseen scenes, and 15.8% in substantially different scenes. In the successful training-scene condition, UMAP/shared semi-NMF show target-organized representations without the original target-to-scene shift, despite near-perfect target/scene decoding.
The positive control shows that successful and unsuccessful instruction following exhibit different representational organization despite similarly high target decodability. Scene-oriented organization is associated with failure in the evaluated settings, rather than being a universal property of fine-tuned VLAs. Attention provides complementary evidence: it follows the instructed object in successful LIBERO-Goal conditions but favors the original target in the failure example, while aggregate instruction contribution remains comparable to other LIBERO tasks. These comparisons strengthen the diagnostic value beyond probe accuracy alone.
We manually labeled Tend based on whether the end effector reached the object's location, regardless of task completion. It measures target-directed behavior separately from Success. Target replacement uses 20 rollouts per scene–replacement-target combination.
Equation 2 sums output-projected attention-weighted vectors across heads/layers before taking norms, preserving reinforcement and cancellation. Aggregating per-layer norms instead summarizes magnitudes without cross-layer cancellation.
The 1.7% instruction contribution on RoboTwin is consistent with its visually distinctive task settings: the current scene provides strong cues about which task to execute, allowing the policy to infer the expected behavior from visual context. Consequently, instruction tokens contribute relatively little to action generation under our measure.
Following your suggestions, we have added per-layer contribution analysis following Kobayashi et al., a quantitative attention-alignment metric, and an action-lens metric. Detailed definitions, evaluation protocols, and results are provided in the revision.
Thank you for the helpful suggestions. We hope the new controls and probes strengthen your assessment of the distinction between accessible target information and generalizable instruction control. Details and additional results are provided in the revision.