acceptodds
Desk rejected at ICLR 2027

Encoded but Not in Control: Revealing the Grounding Gap in Vision-Language Robot Policies

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Instruction following is central to language-conditioned robot policies: language should determine what to do when the same scene permits multiple valid actions. Yet successful execution alone cannot establish whether a policy follows the instruction or infers the task from the scene. We study this ambiguity through scene-preserving instruction interventions, using valid target substitutions, arbitrary nouns, and unrelated sentences while holding the scene fixed. We evaluate vision-language-action (VLA) policies and world-action models (WAMs) in simulation and in real-world experiments. Our analysis addresses three questions: (a) Does task success imply instruction following? When instructions request a different visible object, all evaluated policies predominantly approach and pick up the incorrect original target associated with the scene. (b) Is this failure caused by language insensitivity? Instruction perturbations affect task performance. A layerwise action lens shows intermediate action predictions respond to these perturbations. Linear probes accurately recover instructed targets, indicating modified instructions are encoded despite rarely determining target selection. (c) Why does encoded language fail to control action? Attention analysis indicates weak instruction-token contributions to action generation. Target-token attention can remain focused on the original object, revealing a mismatch between target encoding and visual grounding. UMAP and shared non-negative matrix factorization show target information remains accessible within representations increasingly organized by scene identity. Our findings expose a grounding gap concealed by nominal success and provide a diagnostic framework. They further establish a concrete criterion for progress: policies should reliably follow valid changes in user intent, even when they conflict with scene-favored behavior.

Primary area:
interpretability and explainable AI
Keywords:
Model Interpretability, Robot Foundation Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pjmbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether language-conditioned robot policies follow instructions or infer the task from the scene. Holding the scene fixed, the authors change the instruction in one of three ways: they swap the target for another visible object, replace object nouns with random nouns, or substitute a random sentence. They evaluate two VLAs (pi0.5, GR00T N1.7) and two world-action models (FastWAM, LingBot-VA) on LIBERO, RoboTwin 2.0, and six real-world scenes.

The paper makes three claims. First, under in-scene target replacement, policies overwhelmingly act on the original target: 86-98% of LIBERO-Object trials and 81-94% of real-world trials. Second, the failure isn't simple language insensitivity. Perturbations reduce success rates and change intermediate action readouts from a layerwise "action lens," and linear probes recover the instructed target from instruction-token representations with near-perfect accuracy. Third, the authors offer correlates of why encoded language fails to control action: a modest attention-based instruction contribution to the action expert, a qualitative example where target-token attention stays on the original object, and UMAP and shared semi-NMF analyses showing a post-fine-tuning shift from target-organized to scene-organized representations. They call this a "representation-to-control grounding gap." Validation is behavioral (closed-loop rollouts) plus representational analyses on 60 LIBERO and 36 real-world scene-target conditions.

Critical strengths
  1. The behavioral evaluation is clean and broad. Interventions are tightly matched, with fixed initial states and simulator seeds, and with fixed generation noise in the action lens. The study spans both VLA and WAM paradigms and includes real hardware. Extending the target-replacement failure to world-action models is a useful finding, because it shows that video-prediction objectives alone don't fix target selection. This is the main reason the paper is informative despite overlap with LIBERO-CF and LangGap.

  2. The question is well posed, and the authors frame their claims carefully. The paper separates language sensitivity, information accessibility, and behavioral adherence, and it consistently avoids claiming causality it hasn't shown. That separation is a useful conceptual contribution for evaluating VLAs, even though the mechanistic evidence doesn't yet fully support it.

Critical weaknesses
  1. The "encoded" half of the thesis rests on a probe that is close to guaranteed to succeed. The probes read mean-pooled hidden states at the instruction-token positions, and the instruction contains the target noun, so recovering it there is expected at nearly any layer of nearly any model. The base checkpoint's 100% accuracy is consistent with this. The paper cites Hewitt & Liang but includes none of the controls that work recommends: no control tasks, no layer-0 or random-init baselines, and no held-out-noun splits. With N = 60, D in the thousands, and an unspecified train/test split, accuracy says little. The informative test would probe the action pathway (action-expert or action-query states) to see whether target identity reaches the computation that produces actions. This matters to my assessment because "encoded but not in control" is the paper's central claim and main point of novelty over prior behavioral work, and as tested it may reduce to "the word appears in the prompt."

  2. Part III asks "why" but answers only with correlates, and the diagnostics are never validated against a policy that does follow instructions. The attention-contribution measure attributes contribution by source-token position. In pi0.5's bidirectional prefix, instruction-position states already carry visual information, and the ~20% share comes from roughly 15 tokens against hundreds of image tokens, so calling it "weak" depends on an unstated baseline. The visual-attention claim rests on one qualitative example, the action lens on a single rollout per model, and the UMAP and semi-NMF analyses on 60 points with an unbalanced scene x target design. No interventional test is reported, such as activation patching between target conditions or instruction-token knockout or amplification. There is also no positive control, such as a model fine-tuned with multiple targets per scene, to show that the diagnostics track instruction following. Without these, readers can't tell whether the internal signatures explain the failure or are generic properties of fine-tuned VLAs. Prior work (Grant et al.; VLA-Trace) already reports decodable-but-unused prompt information, which raises the bar for what this paper needs to add.

Questions for authors
  1. How were the probe train/test splits constructed? Does target accuracy hold when entire target nouns are held out, and when probing action-expert states rather than instruction-token states?

  2. Did the real-world training data include multiple instructed targets per scene? If so, the real-world failure is considerably stronger evidence than LIBERO-Object, where each scene has exactly one training task and scene identity predicts the target perfectly. If not, both settings share this identifiability confound, and the failure is partly expected from the data.

  3. How is "Tend" (a committed approach) determined: automatically or by human judgment, and with what agreement? How many rollouts per condition were run for the LIBERO target-replacement experiment?

  4. Why are e_ij vectors summed across layers before taking the norm (Eq. 2), rather than computed per layer as in Kobayashi et al.? Please also explain the 1.7% instruction contribution for pi0.5 on RoboTwin.

Suggestions that would substantially strengthen a revision:

  1. Run activation patching that transplants instruction-token states from target-B runs into target-A runs, layer by layer, to locate where (if anywhere) behavior can be redirected.

  2. Add a positive-control model trained with several instructed targets per scene, and show whether the diagnostics (instruction contribution, scene-versus-target organization) shift as instruction following improves.

  3. Give a quantitative attention-alignment metric across all replacements, plus a directional action-lens metric (for example, predicted displacement toward the new versus the original target) with a reference scale.

2 comments

Thank you for the detailed review and recognition of our controlled evaluation and conceptual distinctions. We address related comments below.

Q1) Probe splits and action-pathway decoding

On the fully crossed real-world set, we train target probes on five scenes and test on the sixth. For scene probes, we train on five targets and test on the sixth. Each split retains all output classes. Probes are fitted per layer (Fig. 3), testing transfer across the other factor rather than unseen target classes.

We added probes on visual-token states and action-token states within the action expert, rather than language-token states, using 60 matched LIBERO-Object conditions. Layer-wise ridge probes select regularization by five-fold cross-validation within training data. Accuracies average over 18 layers and six class-stratified train/test splits:

StatesTarget: BaseTarget: FTScene: BaseScene: FT
Visual99.26%96.20%100.00%99.35%
Action12.31%55.46%33.43%98.06%

Target information is recoverable in the fine-tuned action expert.

Our probe tests accessibility, not improvement over lexical embeddings. Randomly relabeling target categories only permutes output classes, leaving the linear task unchanged; that word-type control gives no informative selectivity comparison. Decoding may reflect lexical retention. We combine it with organizational and behavioral evidence.

Q2) Multi-target controls and representational differences

Original real-world training used one target per scene, with 50 trajectories per task across six scenes. We added two controls to compare successful and unsuccessful instruction following:

  • LIBERO-Object control: Training with two targets per scene enables instruction-conditioned selection between trained targets. On held-out scene–target combinations, however, the policy selects a scene-trained target although the requested target and manipulation were demonstrated elsewhere. Internal analyses of these failures retain the original qualitative representational patterns.

  • Real-world positive control: Every task is demonstrated in every training scene, with 50 demonstrations per scene–task pair. Success is close to 100% in training scenes, 43.6% in similar unseen scenes, and 15.8% in substantially different scenes. In the successful training-scene condition, UMAP/shared semi-NMF show target-organized representations without the original target-to-scene shift, despite near-perfect target/scene decoding.

The positive control shows that successful and unsuccessful instruction following exhibit different representational organization despite similarly high target decodability. Scene-oriented organization is associated with failure in the evaluated settings, rather than being a universal property of fine-tuned VLAs. Attention provides complementary evidence: it follows the instructed object in successful LIBERO-Goal conditions but favors the original target in the failure example, while aggregate instruction contribution remains comparable to other LIBERO tasks. These comparisons strengthen the diagnostic value beyond probe accuracy alone.

Q3) Tend annotation and rollout counts

We manually labeled Tend based on whether the end effector reached the object's location, regardless of task completion. It measures target-directed behavior separately from Success. Target replacement uses 20 rollouts per scene–replacement-target combination.

Q4) Contribution aggregation and RoboTwin results

Equation 2 sums output-projected attention-weighted vectors across heads/layers before taking norms, preserving reinforcement and cancellation. Aggregating per-layer norms instead summarizes magnitudes without cross-layer cancellation.

The 1.7% instruction contribution on RoboTwin is consistent with its visually distinctive task settings: the current scene provides strong cues about which task to execute, allowing the policy to infer the expected behavior from visual context. Consequently, instruction tokens contribute relatively little to action generation under our measure.

Q5) Additional contribution analysis and quantitative metrics

Following your suggestions, we have added per-layer contribution analysis following Kobayashi et al., a quantitative attention-alignment metric, and an action-lens metric. Detailed definitions, evaluation protocols, and results are provided in the revision.

Conclusion

Thank you for the helpful suggestions. We hope the new controls and probes strengthen your assessment of the distinction between accessible target information and generalizable instruction control. Details and additional results are provided in the revision.

Gentle Reminder

Hello! Thank you for your valuable suggestions. We’ve addressed the issues raised in your review and updated the revised version accordingly. If you think any part could use additional clarification or expansion, we’d be delighted to discuss.

Kind regards,
The Authors

Reviewer_ebWDOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether the nominal task success of language-conditioned robot policies implies instruction following. Keeping the visual scene fixed, the authors replace the specified target with another visible object or swap in random nouns or random sentences, evaluating two VLAs (π0.5\pi_{0.5}, GR00T N1.7) and two world-action models (FastWAM, LingBot-VA) across LIBERO, RoboTwin 2.0, and six real-world scenes. Under target replacement in LIBERO-Object, 86.5%–98.3% of rollouts still execute actions toward the original target (Table 1)—a finding that holds true for the two VLAs on physical hardware. Although instruction corruption does impact success rates, most notably on LIBERO-Goal (Table 2), and a layerwise action lens reveals that intermediate action predictions respond to textual prompts, probing analyses point to a disconnect: while linear probes successfully decode the instructed target from instruction-token representations, the attention-based contributions of instruction tokens to actual action generation remain relatively small. UMAP visualizations and a shared semi-NMF analysis further suggest that task fine-tuning reorganizes instruction representations around scene configurations rather than semantic commands. The authors call this a "representation-to-control grounding gap."

Critical strengths
  1. The behavioral evidence is clean and broad. Scene-preserving target replacement effectively isolates target selection from the rest of the task pipeline, demonstrating this failure mode for both VLAs and WAMs in simulation, and for the two VLAs on a real robot (Table 1). Extending this observation to WAMs is useful, as it shows that a video-prediction training objective alone does not guarantee instruction following.

  2. The finding that task-specific fine-tuning reorganizes instruction-token representations from a target-centric structure to a scene-centric one (Figs. 6 and 7) is the most interesting part of the analysis. It suggests that narrow fine-tuning can weaken how the pretrained model uses language, which is relevant to how post-training data should be designed.

Critical weaknesses
  1. The paper does not separate its proposed explanation from a simpler data explanation. In LIBERO-Object, as far as I can tell, each scene layout is paired with a single target during training (the original target in each set of Table 5), so in these scenes the instruction is redundant and nothing in training rewards using it for target selection. For the real-world experiments the training data is not described at all (number of demonstrations, and whether any scene was demonstrated with more than one target), and I could not find the number of rollouts behind Table 1. Without a training intervention, for example fine-tuning with the same scene paired with several targets and re-running the behavioral and representational analyses, the results cannot distinguish "policies fail to ground encoded language" from "the fine-tuning data never required grounding". The behavioral finding itself has been reported before (LIBERO-CF and LangGap, which the paper cites), so much of the added value rests on this explanation.

  2. The mechanistic evidence is weak. The linear probes decode the instructed target from mean-pooled hidden states of the instruction tokens themselves. Since the target word is part of these tokens, near-ceiling accuracy with N = 60 (LIBERO) or 36 (real world) samples and features of a few thousand dimensions is expected, and it says little about whether target information reaches the action pathway. There are no control tasks or selectivity measures, which the cited probing literature recommends, and no probes on action-query or visual-token states. The attention-contribution, UMAP and NMF analyses are correlational, and the paper itself notes several times that they do not establish a causal role. A causal test, such as patching instruction-token or target-token states from another target and checking whether behavior switches, is missing.

Questions for authors

Clarifications Requested:

  1. How was the real-world training dataset collected (specifically, the number of demonstrations per scene and the number of distinct target objects demonstrated within each scene)? And how many rollouts underlie each entry in Table 1?
  2. What is the linear probe classification accuracy measured at the input embedding layer, as well as on a baseline control task?

Key Experiments Needed:

  1. Fine-tune a VLA on LIBERO-Object (or the real-world dataset) such that each visual scene is paired with multiple distinct target objects during training. Does the target-replacement failure mode disappear under this multi-target regime, and do the probe, attention, and UMAP/NMF results change accordingly?
  2. Can the instructed target be decoded from action-query or visual-token states, and does an intervention on the instruction representations redirect the policy's behavior?
3 comments

Thank you for the careful review and recognition of our behavioral and representational evidence. We combine related comments below.

Q1) Real-world setup

The original real-world dataset used six scenes with one demonstrated target per scene and 50 complete trajectories per task, totaling 300 demonstrations. Evaluation uses 20 closed-loop rollouts per scene–replacement-target combination, aggregated in Table 1.

Q2) Additional experiments and findings

We conducted the suggested LIBERO-Object intervention with two targets per scene. The policy distinguishes the trained targets according to the instruction. However, when the requested target was demonstrated elsewhere but not paired with the current scene during training, the policy selects a target it was trained to manipulate in the current scene. Probe, attention, and UMAP/shared semi-NMF analyses of these held-out failures retain the original qualitative patterns.

A complementary real-world control demonstrates every task in every training scene, with 50 demonstrations per scene–task pair. It achieves close to 100% training-scene success, 43.6% success in visually similar unseen scenes, and 15.8% in substantially different unseen scenes. Both target and scene remain nearly perfectly decodable, but UMAP/shared semi-NMF show target-organized representations, without the target-to-scene shift observed in the original failure setting.

These comparisons show that our representational diagnostics distinguish successful and unsuccessful instruction-following conditions despite similarly high target decodability. Attention provides complementary evidence: it follows the instructed object in successful LIBERO-Goal conditions, contrasting with its original-target bias in the failure example. Thus, the observed scene-oriented organization is associated with failure in the evaluated settings, rather than being a universal property of fine-tuned VLAs.

Together, these results establish that multi-target training supports instruction following on trained combinations but does not guarantee generalization to held-out combinations, while strengthening the diagnostic value of our internal analyses.

Q3) Probe validity and action-pathway decoding

Instruction-token decoding reflects retention of language-related information.

We added visual-token and action-expert probes on the same 60 LIBERO-Object conditions. Separate layer-wise ridge probes select regularization by five-fold cross-validation within training data. Accuracies below average over 18 layers and six class-stratified train/test splits, with ten classes for each task:

StatesTarget: BaseTarget: FTScene: BaseScene: FT
Visual99.26%96.20%100.00%99.35%
Action12.31%55.46%33.43%98.06%

Pooling uses valid image tokens or action positions at the final denoising step. Scene identity is decoded substantially more accurately than target identity.

Regarding embedding/control-task comparisons, our probe asks whether information remains accessible, rather than whether deeper states outperform lexical embeddings. Hewitt & Liang's word-type control tests arbitrary word-to-label memorization. Here, a one-to-one random relabeling of target categories only permutes output classes, leaving the multiclass linear classification problem unchanged. That control does not provide an informative selectivity comparison for this task. We complement accessibility with pathway probes and representational comparisons.

Conclusion

Thank you for the concrete suggestions. We hope the training controls and action-pathway probes strengthen your assessment of our additional finding: preserved target accessibility can coexist with substantial changes in scene–target organization. Details and additional results are provided in the revision.

The authors have partially addressed my concerns. Therefore, I have decided to increase my rating.

Thank you again for your suggestions!

Best,
Authors

Reviewer_bjzROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper investigates a grounding gap in vision-language robot policies: after fine-tuning, instructions can fail to control actions even when they are encoded. To justify this, the paper follows a three-step logic. First, scene-preserving instruction interventions are conducted, revealing that the policy conducts persistent selection of the original scene-associated targets while ignoring the instructed ones. Second, action prediction and textual probing analyses are conducted, showing that the instruction information is actually not ignored but encoded. Finally, attention analysis and representation decomposition show weak language contributions and strong scene domination in the output. Based on the results, the paper places the use of encoded language at the center of the grounding problem.

Critical strengths
    1. The paper discusses a meaningful problem in robotic instruction following: policies may persist in training-stage behaviors despite changed instructions, raising concerns about whether language reliably controls their actions in practice.
    1. The paper presents a relatively systematic experimental analysis, combining instruction interventions, linear probing, attention analysis, and representation decomposition across simulation and physical experiments to characterize the gap between encoding and control.
Critical weaknesses
    1. (Critical) As discussed in the paper, [1] and [2] already demonstrate instruction-following failures under fixed-scene semantic changes, whose methodology is used in the first step analysis in the paper. Meanwhile, [3] points out that decodable prompt information despite behavioral invariance, which shares the similar methodology as the second step in the paper. Although target-specific probing and shared semi-NMF in the third step of the paper provide additional characterization, the paper does not sufficiently establish what new understanding these analyses deliver beyond previously studied visual shortcuts and the dissociation between instruction encoding and behavioral control.

[1] When vision overrides language: Evaluating and mitigating counterfactual failures in vlas. arXiv preprint arXiv:2602.17659, 2026.

[2] Langgap: Diagnosing and closing the language gap in vision-language-action models. arXiv preprint arXiv:2603.00592, 2026.

[3] Not all features are created equal: A mechanistic study of vision-language-action models. arXiv preprint arXiv:2603.19233, 2026.

    1. (Critical) The details of the implementation of fine-tuning, particularly for the real-world experiments, remain insufficient. Because the interpretation depends on learned scene–task associations, these details are essential for justifying the soundness of the analysis.
    1. The paradigm of fixed-scene target substitutions has its reasonableness for justifying the existence of the instruction following problem, while it does not reflect how significant it is in real problems, since during testing, it would be not that likely the scenes remain exactly the same to the ones during training. Since the paper is more about problem spotting than method proposing, it would be stronger to include more complete experimental analysis about this. Controlled changes of the scenes, including object positions, distractors, and layouts beyond the training dataset would largely address this issue.
    1. Some major analysis steps can only justify correlations, rather than causality. For example, target decoding from instruction tokens may reflect lexical retention rather than visual grounding, smaller attention contributions do not imply limited causal influence, and scene-dominated variance does not establish scene-dominated decisions. Although the authors acknowledge these limitations, controlled interventions or comparisons with successful instruction-following policies could substantiate the proposed explanation significantly.
Questions for authors
  • (Critical) Following Weakness 1, please clarify the novel findings beyond the existing work [1-3].

  • (Critical) Following Weakness 2, please provide more details of fine-tuning.

  • Following Weakness 3, experiments with controlled changes of the scenes, including object positions, distractors, and layouts beyond the training dataset would be suggested.

  • Following Weakness 4, controlled interventions or comparisons with successful instruction-following policies would be suggested.

4 comments

Thank you for the thoughtful review and for recognizing the importance of reliable instruction control and our systematic analysis. We address questions below.

Q1) Contributions beyond prior work

We thank the reviewer for highlighting this critical point. Building on LIBERO-CF/LangGap’s behavioral findings and Grant et al.’s encoding–behavior dissociation, our work provides three additional contributions:

  1. Extending counterfactual target-selection failures to world-action models, showing that visual-dynamics modeling does not guarantee instruction following. FastWAM and LingBot-VA remain directed toward the original target in 98.3% and 96.3% of LIBERO-Object trials, respectively, while success on the newly instructed target is only 0.0% and 1.3% (Table 1).

  2. Distinguishing instruction sensitivity during action generation from successful behavioral redirection, beyond prompt decodability alone. With observations, robot states, and initial generation noise matched, our layerwise action lens shows that instruction changes affect intermediate action readouts across layers and denoising steps (Fig. 2), yet closed-loop rollouts retain the original-target preference (Table 1).

  3. Identifying a fine-tuning-associated shift toward scene-organized instruction representations despite preserved target recoverability. While both target and scene identities remain highly decodable (Fig. 3), shared semi-NMF with variance partitioning reveals increased scene-unique and decreased target-unique explained variance (Fig. 7), complemented by intermediate-layer scene clustering in UMAP (Fig. 6).

Together, these findings extend the known encoding–control dissociation by identifying what changes despite preserved decodability: the relative organization of target and scene information. This provides a specific representational characterization accompanying instruction-following failures that probe accuracy alone cannot reveal.

Q2) Real-world experimental setup

The original π0.5/GR00T N1.7 real-world experiments used six scenes, one demonstrated target per scene, and 50 complete trajectories per task: 300 demonstrations. Target-replacement evaluation uses 20 rollouts per scene–replacement-target combination. Regarding scene–task associations, we added a real-world control with every task demonstrated in every training scene, using 50 demonstrations per scene–task pair.

Q3) Generalization to unseen scenes

We additionally evaluated the real-world multi-task policy in scenes excluded from training: success is close to 100% in training scenes, 43.6% in visually similar unseen scenes, and 15.8% in substantially different unseen scenes.

In LIBERO-Object, a policy trained on two targets per scene could follow instructions for both trained targets. On held-out scene–target pairings, however, it selected a target trained in the current scene, even though the requested target and its manipulation had been demonstrated elsewhere.

Together, these results distinguish instruction following on trained combinations from transfer to new combinations and scenes. Requiring language during training supports the former but does not ensure the latter.

Q4) Positive controls and action-pathway probes

In the successful real-world control, both target and scene remain nearly perfectly decodable, yet representations are target-organized rather than showing the original target-to-scene shift.

On LIBERO-Goal, attention follows the instructed object. Additional LIBERO action-token probes show target accuracy increasing from 12.31% to 55.46% after fine-tuning, while scene accuracy increases from 33.43% to 98.06%. This extends the analysis into the action expert.

These comparisons show that successful and unsuccessful instruction-following policies exhibit different representational organization. This indicates that scene dominance is associated with instruction-following failure in the evaluated settings, rather than being a universal consequence of fine-tuning.

Conclusion

Thank you for the constructive suggestions. We hope the training controls, successful-policy comparisons, and action-pathway probes clarify our contribution and strengthen your assessment of its empirical support. Details and additional results are provided in the revision.

Gentle Reminder

Hi there! Thanks again for the thoughtful feedback. We’ve addressed the questions you raised and incorporated the corresponding updates into the revised version. If there’s anything you feel could benefit from further clarification, we’d be more than happy to continue the discussion.

Best regards,
The Authors

Thanks for the responses. I appreciate the new experimental results. However, I share the same concern with pjmb that the current paper focuses on spotting phenomena while lacking insightful analysis to answer "why". Due to this reason, I would like to keep my score unchanged.

AuthorsOpenReview ↗

Response to the Reviewer bjzR

Thank you for reviewing the new results and for asking us to explain why policies revert to the scene-associated target. As discussed in our response to reviewer pjmb, we added multi-target training controls and action-expert probes to help explain this behavior.

  • The LIBERO-Object control shows when the policy falls back on the scene. With two demonstrated targets per scene, the policy followed instructions for either trained target. For held-out scene–target pairings, it reverted to a target demonstrated in the current scene, and its representations were organized by scene.
  • Scene information is more strongly represented than target information in the action expert. After fine-tuning, scene decoding from action-token states rose from 33.43% to 98.06%, while target decoding rose from 12.31% to 55.46%. Scene identity thus became substantially more recoverable than instructed target identity within the action pathway itself.
  • Successful instruction following is associated with target-organized representations. In the real-world positive control, we demonstrated every target in every training scene, making the instruction necessary to select the target. Success was close to 100% in those scenes, and the representations were organized by target.

These results suggest that scene-organized representations were associated with instruction-following failures, while target-organized representations were associated with successful instruction following. After fine-tuning, scene identity was also more readily decoded than instructed target identity from action-expert states. Together, these findings offer insight into why policies revert to the scene-associated target instead of following the current instruction.

We hope these results could clarify your concerns, and we welcome any future discussion.

Best,

Authors

Comments on the paper

1 comment

We thank all three reviewers for their careful and constructive feedback. All three found the paper clear and easy to follow. Reviewer bjzR highlighted the importance of reliable instruction control and the systematic analysis. Reviewer ebWD highlighted the breadth of the behavioral evidence and the shift toward scene-organized representations after fine-tuning. Reviewer pjmb highlighted the tightly matched interventions and the distinction between language sensitivity, information accessibility, and behavioral adherence.

The new text and appendix additions are marked in blue in the revised manuscript.

General Clarifications

  • Multi-target training and generalization. We added two training controls with multiple targets per scene. In LIBERO-Object, we demonstrated two targets per scene. In the real world, we demonstrated every target in every training scene. Multi-target training enabled instruction-based selection on trained scene–target pairs, while failures persisted on held-out pairs in LIBERO-Object and success dropped in unseen real-world scenes. The real-world policy achieved close to 100% success in training scenes, 43.6% in visually similar unseen scenes, and 15.8% in substantially different unseen scenes.

  • More evidence from the internal analyses. Despite high target decodability in both settings, representations were target-organized in the successful real-world control and shifted toward scene organization in the held-out LIBERO-Object failures. The positive control thus shows that the two behavioral outcomes can be distinguished by how target and scene information are organized. We also added two quantitative measures: attention retained on the original target in the representative replacement example and the change in predicted end-effector motion direction.

  • Contribution beyond prior work. We extend fixed-scene target-replacement failures to world-action models and show that instruction changes can affect intermediate action predictions even when closed-loop target selection remains biased toward the original target, and quantify a fine-tuning-associated shift toward scene-organized representations despite preserved target decodability.

Additional Results and Manuscript Changes

  • For Reviewer bjzR. We documented the original real-world training dataset and added the fully crossed multi-target control, its evaluation in unseen scenes, and its UMAP and shared semi-NMF analyses (App. H.2.1, pp. 28–29).
  • For Reviewer ebWD. We added the two-target LIBERO-Object control and analyzed representations for held-out scene–target pairs (App. H.1, p. 28). We also added a successful LIBERO-Goal attention comparison (App. H.2.2, pp. 29–30) and visual-token and action-expert probes (App. J, pp. 30–31, Table 8 on p. 31). After fine-tuning, the action-expert probes recover the instructed target with 55.46% accuracy and scene identity with 98.06% accuracy.
  • For Reviewer pjmb. We added layer-wise attention contributions (App. K, pp. 31–32) and a directional action-lens metric based on net end-effector displacement (App. I, pp. 30–31). At object-selection keyframes in the representative replacement example, the original target region received 78.6% as much attention as under the original instruction (Sec. 6.2). Our individual reply also specifies the rollout count and the manual annotation criterion for Tend.

Conclusion

We thank the reviewers for suggesting the multi-target controls and additional analyses. The new controls show where instruction following succeeds and where it fails to transfer, while the added probes and analyses clarify how those outcomes relate to the information available within the policy. We welcome further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.