acceptodds
Under review as a conference paper at ICLR 2027

When Images Read the Instruction: Bidirectional Attention in Vision-Language-Action Models

16 views

Abstract

Embodied reasoning and manipulation require rich interaction between visual, linguistic, and proprioceptive modalities, yet the dominant VLA backbone is a causal decoder that encodes the image before the instruction or robot state enters the context. Bidirectional attention is a natural remedy, as all tokens can freely attend to each other. Popular VLAs already adopt bidirectional observation encoding, but its effect has not been tested in a controlled comparison. In this work, we systematically study how bidirectional attention shapes representations for embodied tasks. We build matched paradigms from one causal Qwen3-VL-2B that vary in their embodied pretraining and attention mask: causal, flipped to bidirectional at policy time, or bidirectional and learned during pretraining. We show that only a learned attention mask improves manipulation over causal pretraining on the same data when using a final layer acton head. Activation patching, closed-loop instruction transplants, attention-path knockouts, and object-level attention measurements trace this difference. The attention mask decides where the instruction is represented: every bidirectional mask moves the instruction into the image tokens, and the policy reads its task from there. At what stage the mask is learned decides whether the instruction's attention stays on the task objects. A learned mask preserves the object selectivity that embodied pretraining creates, while a flipped mask does not. Our results suggest that bidirectional attention in VLA backbones should be aligned during pretraining rather than added at policy time.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vlm, vla, robot learning, embodied reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yUq5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how attention structure and embodied pretraining affect instruction conditioning in vision-language-action models. It compares six training configurations, including causal attention, bidirectional attention introduced during policy training, PrefixLM pretraining, and encoder–decoder conversion. The evaluation combines downstream policy performance with activation patching, closed-loop feature transplantation, attention knockouts, and object-attention measurements. The results suggest that bidirectional embodied pretraining can improve instruction-conditioned visual representations and policy performance when the action head reads the final backbone layer, with benefits depending on the backbone and action-head design.

Critical strengths
  • Closed-loop transplantation provides behavioral evidence beyond attention visualization. Transplanting image features computed under a different instruction redirects the policy toward the donor task while keeping the observed scene fixed. This shows that instruction information carried by visual representations can influence task execution, strengthening the evidence beyond correlations in attention maps.

  • The unsuccessful bidirectional configurations are informative. P3 and P4 allow images to attend to instructions, yet their object-attention lift and policy performance do not consistently improve over causal configurations. These comparisons show that enabling the connection alone is insufficient. The additional backbone and action-head experiments also expose meaningful limits to the positive findings.

Critical weaknesses
  • The resulting design guidance remains narrow relative to existing bidirectional VLAs. π0 already uses bidirectional attention between image and instruction tokens, so the main contribution here is explaining when and why this design helps. However, P3 has lower object-attention lift than P1, the Qwen3.5-0.8B conversion performs worse than its causal baseline, and the policy advantage largely disappears with a layer-wise action head. These experiments vary different factors and do not establish a unified account of when bidirectionality helps or hurts. Although the paper acknowledges several limitations, the evidence provides limited guidance beyond the principal 2B, final-layer-head setting.

  • The interventions do not establish why the changed information flow improves policy performance. Closed-loop transplantation demonstrates that instruction-conditioned image features can control behavior, while inference-time knockouts reveal dependencies of trained checkpoints. These are useful mechanistic findings, but they do not explain why this fusion route should outperform fusion through subsequent text or action tokens. The trained configuration comparisons establish performance differences under the tested recipes, but do not isolate instruction information in image tokens as the cause of those differences. Likewise, the object-attention measurements do not establish that stronger object selectivity causes higher task success.

  • P6’s gains are not fully separated from additional pretraining capacity and computation. P6 uses approximately 3.5B parameters during embodied pretraining, compared with approximately 2.1B for P2 and P5. Matching downstream backbone size and pretraining examples therefore does not match pretraining resources. The larger causal model in Table 6 reaches comparable embodied reasoning performance, but no downstream policy results are reported for it. Consequently, the contribution of additional pretraining resources to P6’s policy gains remains unresolved. This concern applies specifically to P6 and does not invalidate the P5 comparison.

Questions for authors
  1. How do you explain the differing outcomes across P3, the 0.8B backbone, and the layer-wise action head? Beyond the stated restriction to final-layer heads, what conditions determine whether bidirectional embodied pretraining is beneficial?

  2. Can you provide matched causal/bidirectional policy comparisons at different sizes within the same backbone family, or vary the amount of embodied pretraining data and report downstream success? Figure 5 tracks reasoning performance during conversion, but does not establish how the downstream advantage changes with model or data scale.

3 I prefer to train a downstream policy from the existing 4B causal checkpoint in Table 6, using the same downstream data, action-head type, and training protocol. Since its reasoning score matches P6, this would clarify whether comparable reasoning performance translates into comparable control performance, and what advantage P6 offers relative to enlarging the causal backbone.

4 comments

Thank you for the thoughtful review, and in particular for asking for one account of the negative cases. Changes in the revised PDF are purple, and green where they address several reviewers.

W1 / Q1 (one account of when bidirectionality helps). We agree that the paper presented these cases as separate caveats. We conduct one additional experiment per case. First, for the flip P3 we measure where in depth its image tokens read the instruction, train it for 60k instead of 30k steps, and apply the same kind of flip on a bimanual robot (ABC-Sim). Second, for Qwen3.5-0.8B we train a second seed and repeat the comparison on SmolVLM2-500M, a full-attention model of similar size, as well as a larger model (Qwen3-VL-4B). Third, for the layer-wise head we add a matched control. Our initial experiments compared against the non-pretrained model. Fourth, we block image-to-instruction attention in pi0.5 in early or late layers.

CaseNew experimentResult
P3 flipshare of image-to-instruction attention in layers 19-27flips 0.29 to 0.38, learned masks 0.06 to 0.13
P3 vs P1 at 30k and 60k steps-2.0 at 30k, +2.7 at 60k (s42)
same flip on ABC-Sim (P4 vs P2)-20.4 [-26.7, -14.2]
Qwen3.5-0.8Bsecond seed, P6 vs P2-3.3 (s42), -0.7 (s43)
SmolVLM2-500M, P5 and P6 vs P2+5.7 [+3.3, +8.1] and +2.9 [+0.3, +5.5]
Layer-wise headmatched control, P5 and P6 vs P2+3.5 [+2.0, +5.1] and +2.0 [+0.4, +3.7]
pi0.5 (LIBERO, 96.5)route blocked in all / early / late layers69.0 / 71.5 / 97.0
pi0 (LIBERO, 89.5)route blocked in all / early / late layers31.5 / 66.0 / 89.0

All cases follow one rule. Bidirectional attention helps when image tokens read the instruction in the early layers, through a mask on full-attention layers that the backbone learned before policy training. P3 learns the mask only at policy time, reads the instruction late and needs twice the budget to catch up, and the same flip collapses on a different embodiment. Qwen3.5-0.8B is mostly linear attention, where conversion is not a mask change. On every full-attention backbone we tested, the gains replicate. The layer-wise head follows the rule once it has the matched control. pi0 and pi0.5 also rely on the early layers. We hypothesize that the learned mask lets the backbone compute task-conditioned image features early, and that this is learned best before the policy sees few demonstrations with fixed instructions. In the revision, the Discussion states this one rule instead of separate caveats, and the cases are in Appendix K.

W2 (why fusion through image tokens rather than text or action tokens?). Thank you, this is exactly what our earlier tests could not show. We conduct four additional experiments. First, we retrain P5 and P6 with only the image-to-instruction route removed, so the instruction still reaches the text tokens and the head. Second, we train causal policies with the instruction before the images. Third, on ABC-Sim we block the route at inference in early or late layers. Fourth, on ABC-Sim we stop the action head from reading the instruction tokens, alone and together with the image route block.

Route removed (pp)P5P6
LIBERO-Plus (training time)-3.9 [-5.5, -2.2]-3.2 [-4.7, -1.6]
LIBERO-PRO, paraphrased (training time)-2.9 [-7.0, -0.1]-2.1 [-4.5, -0.3]
LIBERO-PRO, new goals (training time)-1.0 [-3.7, +1.9]-5.5 [-9.0, -2.4]
RoboCasa, reworded (training time)-3.7 [-7.5, -0.1]-7.7 [-11.5, -4.1]
ABC-Sim, layers 0-18 at inference-14.8 [-19.8, -9.8]
ABC-Sim, layers 19-27 at inference+2.9 [-2.1, +7.7]
ABC-Sim, head cannot read the instruction-8.8 [-15.8, -1.7]-2.9 [-7.7, +1.9]
ABC-Sim, head and image route blocked-18.3 [-25.4, -11.2]-20.6 [-25.8, -15.4]
Policy (LIBERO-Plus %)Image tokens read the instruction viaSuccess
P2 causal PTcannot76.0
P2, instruction first (s42)token order75.0
P1 plain, instruction firsttoken order76.0
P5 PrefixLMlearned mask78.3
P5, route removedcannot74.4

Removing only the image route costs most or all of the gain on every benchmark, although the text tokens and the head still see the instruction. On ABC-Sim, blocking the early route removes the whole 16-point gain of P6. So fusion through later text tokens is not an equivalent substitute. The head test shows this directly. When the action head of P6 cannot read the instruction tokens, P6 loses only 2.9 points and stays 12.7 points above P2, while the same restriction costs the causal P2 16.7 points. Only when the image route is blocked as well does P6 fall to the level of P2. So in P6 the instruction reaches the actions through the image tokens. A new probe shows what the route adds. In P5 and P6 the image tokens of the target object change with the instruction from about layer 4 on, so they encode which object the task names. We hypothesize that the early route produces text-conditioned image features, mainly for the wrist camera, so the head receives tokens that already encode which object to approach. A causal policy has to learn this binding from demonstrations, which is slower and does not transfer to new wordings. In the revision, the route ablation is the causal test of RQ4 in the main text, and the instruction-order controls and the probe are in Appendix K. We further hypothesize that the route emerges through the modality the policy relies on most. In our policies this is the wrist camera (blocking the route only for the wrist camera costs most of the gain), which is why other features are merged into it.

W3 / Q3 (P6 capacity, policy from the 4B causal checkpoint). Thank you for the precise suggestion. We conduct two additional experiments. First, we train policies from the existing 4B causal checkpoint with the same data, head and protocol (2 seeds). Second, we pretrain PrefixLM and conversion variants of Qwen3-VL-4B on the same data and budget and train policies from them.

Backbone (LIBERO-Plus %)Causal PTPrefixLMConversion
Qwen3-VL-2B76.078.378.2
Qwen3-VL-4B73.979.580.3

The 4B causal policy is 4.3 points below 2B P6, although its embodied VQA score matches P6. More causal capacity does not reproduce the gain. P5 has the same parameters as P2 and gains as much as P6, and at 4B both learned masks gain more than at 2B. Over 12 backbones, embodied VQA scores do not predict policy success (r = -0.32). We hypothesize that P6 offers an attention pattern, not capacity: image tokens that read the instruction, which a causal model with images first cannot provide. In the revision, the 4B results are in the new RQ1 table, and P5 is stated as the parameter-matched control.

Q2 (model size and pretraining data). We ran both. First, the 4B comparison above. Second, we train policies from intermediate pretraining checkpoints after 1M, 2M and 4M of 6.5M samples (s42).

LIBERO-Plus (pp)P5 vs P2P6 vs P2
Qwen3-VL-2B, 6.5M samples+2.4 [+1.0, +3.7]+2.2 [+0.8, +3.7]
Qwen3-VL-4B, 6.5M samples+5.5 [+3.6, +7.5]+6.4 [+4.8, +8.0]
Qwen3-VL-2B, 1M samples+2.8 [+0.6, +5.0]+4.0 [+1.9, +6.2]
Qwen3-VL-2B, 2M samples+2.5 [+0.9, +4.2]+0.7 [-1.5, +2.9]
Qwen3-VL-2B, 4M samples+1.4 [-0.8, +3.7]-1.0 [-3.5, +1.6]

The gain is larger at 4B than at 2B. P5 is ahead from the first million samples on, which fits the result that a short generic PrefixLM stage recovers most of the gain. P6 is unstable in the middle of pretraining. We hypothesize that this is because conversion is a larger architectural change, and the policy has to adapt to the new encoder-decoder architecture. In the revision, both results are in Appendix K.

Thank you for the additional experiments. The training-time ablation somehow addresses my mechanistic concern. However, I still have three remaining reservations:

  1. Token order. The findings in K.6 motivate a causal alternative: placing the instruction before the images so that image tokens can attend to it. K.5 tests this order during policy training, but leaves open whether using it throughout both pretraining and policy training could match bidirectional attention.

  2. Practical significance. In the main 2B setting, the bidirectional models improve LIBERO-Plus success by about 2–2.5 percentage points over causal pretraining. I find this gain modest as a basis for a broader training recommendation.

  3. Scaling. The larger gap at 4B partly reflects a lower causal baseline (73.9% at 4B versus 76.0% at 2B). This makes it difficult to conclude that bidirectional attention becomes more beneficial with model scale.

Therefore, I keep my original rating.

Reviewer_LNN6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper asks whether bidirectional attention in vision-language-action (VLA) models helps embodied policy learning, and if so, when and why. Starting from one causal VLM, Qwen3-VL-2B, the authors build six matched paradigms that cross embodied pretraining (none, causal, PrefixLM, encoder-decoder conversion) with the policy attention mask (causal, bidirectional imposed at policy time, bidirectional learned during pretraining). Through activation patching, a closed-loop instruction transplant, attention-path knockouts, and object-attention lift, the paper argues that bidirectional attention moves the instruction into image tokens, that the policy reads its task from those tokens, and that only a learned mask preserves object-selective instruction attention.

Critical strengths
  1. The six paradigms isolate the attention mask from the stage at which it is learned, holding the initial backbone, policy data, action head, schedule, and executed trainable parameters matched. This is a real step beyond prior work that changes the mask together with the backbone, corpus, or objective.

  2. Activation patching, the closed-loop transplant, reverse-path knockouts, and object-attention lift together form a coherent investigative approach that could be reused by the community to study representation in VLAs.

Critical weaknesses
  1. Appendix I shows that with a layer-wise action head the causal policy catches up and the conversion gain disappears (LIBERO-Plus 78.1 vs 78.9; RoboCasa pick-and-place 62.8 vs 57.3). Appendix I also reports a Qwen3.5-0.8B control that reverses the result (conversion 72.3 vs causal 75.6). These are not incidental ablations—they directly bound the headline claim that learned bidirectional masks help VLAs. The paper relegates them to the appendix and dismisses them in a single sentence, while the main text and abstract present the finding as general. This is a framing problem, but it is also a soundness problem: the claimed causal effect of the learned mask is conditional on a specific action head and model family, and the paper does not establish how broad the condition is.

  2. The P5/P6 vs P2 contrast is the central evidence for "the stage of learning matters," but each pretraining variant is trained once, and the two policy seeds cannot capture run-to-run variance in the pretraining stage. The authors acknowledge this in Appendix C but still present the main-text conclusion as robust. For a 4-point effect, this is an inadequate variance control.

  3. The selectivity advantage holds only on the agent view; on the wrist view the paradigms are comparable. Figure 8 shows PrefixLM/π0.5 make small image-token edits that carry the full effect, while conversion rewrites the tokens more heavily—this sits in tension with the RQ4 account that centers on selectivity. The mechanism chain is suggestive but not closed.

Questions for authors
  1. The Qwen3.5-0.8B control reverses the headline result. How do you reconcile this with the claim that a learned mask improves manipulation? Does the effect depend on model scale or family, and can you provide at least one additional backbone with multiple seeds?

  2. With a layer-wise action head, the causal policy catches up and the conversion gain disappears. Should the recipe be stated explicitly as "for final-layer action heads only"?

  3. Each pretraining variant is a single run. Can you estimate pretraining-run variance with additional seeds or checkpoint variance, and does the P5/P6 vs P2 contrast survive?

4 comments

Thank you for the detailed review and for recognising the matched design. Changes in the revised PDF are orange, and green where they address several reviewers.

W1 / Q1 / Q2 (action head and backbone bound the claim). We agree that these results should be highlighted in the main manuscript. We note that the experiments with Qwen3.5-0.8B can not be put in the same family as the others. Qwen3.5 models are hybrid attention and state-space models, with the majority of layers being linear attention layers. Thus, making these layers bidirectional is not trivial and outside of our studied setting. We will make this clearer in the main manuscript. Instead, we conduct additional experiments across non-hybrid full-attention models of different families and sizes. First, we add the matched control that Appendix I was missing. Appendix I compared against plain Qwen3-VL-2B instead of causal pretraining, which was not a controlled experiment. We train P2 with the layer-wise head (2 seeds). Second, we block image-to-instruction attention at inference in the layer-wise policies, in early or late layers. Third, we repeat the full matched comparison on Qwen3-VL-4B and SmolVLM2-500M.

Setting (success %)P2 causal PTP5 PrefixLMP6 conversionP5 vs P2P6 vs P2
Final-layer head (paper)76.078.378.2+2.4 [+1.0, +3.7]+2.2 [+0.8, +3.7]
Layer-wise head, LIBERO-Plus75.779.277.7+3.5 [+2.0, +5.1]+2.0 [+0.4, +3.7]
Layer-wise head, RoboCasa66.469.465.9+3.1 [+0.9, +5.0]-0.5 (n.s.)
SmolVLM2-0.5B59.965.562.7+5.7 [+3.3, +8.1]+2.9 [+0.3, +5.5]
Qwen3-VL-4B73.979.580.3+5.5 [+3.6, +7.5]+6.4 [+4.8, +8.0]
Layer-wise head, route blocked at inference (pp)all layerslayers 0-18layers 19-27
P5-51.2-39.0-4.1
P6-50.0-46.4-0.9

With the correct baseline the gain holds under the layer-wise head, and the merging in early layers carries the gains as with the final-layer head. Plain Qwen3-VL-2B is unusually strong with this head (77.3), which made the original comparison look like a null. The gain also holds from 0.5B to 4B in two families. We hypothesize that a head reading intermediate layers can partly replace fusion in a plain VLM, but it cannot create the early route that the learned mask provides. Appendix I is corrected and the results move into the main text (RQ1). We believe that these new findings across action heads and different model sizes and families strengthen the generalizability claim significantly and show that our assumptions hold broadly in different settings. We also point to the new dataset results in the general response, further highlighting consistent gains across different datasets, including bimanual setups.

W2 / Q3 (single pretraining run per variant). We agree that a single pretraining run per variant did not show whether the gain is robust to the pretraining seed, and thus conduct two additional experiments. First, we rerun all three 2B pretraining runs with a second seed and train both policy seeds on each (2 x 2 design). Second, we train policies from intermediate pretraining checkpoints after 1M, 2M and 4M of 6.5M samples (s42) to investigate how different dataset pretraining budgets affect downstream performance.

LIBERO-Plus (pp)P5 vs P2P6 vs P2
Original pretraining seed+2.4 [+1.0, +3.7]+2.2 [+0.8, +3.7]
New pretraining seed+1.8 [+0.5, +3.1]+1.0 [-0.5, +2.6]
Both (4 pairs)+2.1 [+1.0, +3.2]+1.7 [+0.4, +2.9]
1M samples (s42)+2.8 [+0.6, +5.0]+4.0 [+1.9, +6.2]
2M samples (s42)+2.5 [+0.9, +4.2]+0.7 [-1.5, +2.9]
4M samples (s42)+1.4 [-0.8, +3.7]-1.0 [-3.5, +1.6]

Both learned masks beat causal pretraining over the 2 x 2 design. P5 stays ahead at every checkpoint, while P6 is less stable during pretraining. We hypothesize that PrefixLM only relaxes the mask on the causal weights, while conversion has to reorganize the encoder first. In the revision, the main comparison uses the pooled 2 x 2 numbers, the checkpoint curve is in Appendix K, and the P6 instability is stated as a limitation.

W3 (agent vs wrist view, Fig. 8 vs RQ4). Thank you, this pointed to a real gap. To further pinpoint the source of the gains we conduct three additional experiments. First, we retrain P5 and P6 with the image-to-instruction route blocked for all image tokens, only the wrist camera, only the agent camera, or only some layers, and measure success and agent-view lift. Second, we measure for every bidirectional policy how much of its image-to-instruction attention happens in late layers. Third, we measure how image tokens and actions change when RoboCasa instructions are reworded, and evaluate reworded instructions in closed loop.

Blocked during trainingP5 success (pp)P6 success (pp)P5 agent liftP6 agent lift
All image tokens-3.9-3.2-0.63-0.92
Layers 0-18 only-2.8-3.7-0.53-0.89
Layers 19-27 only (lift s42)+0.9+0.7-0.05-0.35
Wrist camera only-2.5-3.3+0.05+0.02
Agent camera only (s42)-0.7+1.5-0.55-0.47
P2 causal PTP5 PrefixLMP6 conversion
Image-token change under rewording, final layer00.070.29
Action change under rewording0.220.180.18
Success with reworded instructions (%)53.361.555.9

Blocking the agent view lowers agent-view selectivity but not success. Blocking the wrist view lowers success but not selectivity. So the gain comes from the wrist view reading the instruction in the early layers, and selectivity is a by-product. Across the 8 bidirectional policies, the less image-to-instruction reading happens late, the higher the success (rank correlation -0.90). A new probe confirms what the route computes. In P5 and P6 the image tokens of the target object change with the instruction from about layer 4 on, while P4 shows no such signal and P3 shows it only in the late layers. This also resolves the tension with Fig. 8. The small edits of P5 and pi0.5 are what an early, targeted read looks like. P6 rewrites its image tokens more but does not change its actions more, and it is less robust to rewording in closed loop. We hypothesize that the early route produces text-conditioned wrist features that encode which object to approach. In the revision, RQ4 is rewritten around the wrist-view route, Fig. 4 now shows where in depth the image tokens read the instruction, and the tables and the probe are in Appendix K. The route ablation gives the same result on LIBERO-PRO, reworded RoboCasa and ABC-Sim. We hypothesize that image and language features are merged early on, and that later layers refine the merged features for action generation.

We hope this answers all questions and concerns and are happy to provide more clarifications!

Reviewer_LNN6OpenReview ↗

maintain my rating

Thank you for the extensive additional experiments. The revised response provides stronger evidence regarding the effects of different attention masks. However, my main concerns remain unresolved.

First, the proposed approach primarily relies on modifying the attention mask to enable bidirectional information flow. While the authors provide further analyses of its internal mechanisms, it remains unclear what fundamental advantage this design offers over existing multimodal fusion strategies.

Second, the evaluation is still largely based on comparisons among the authors' own variants, with insufficient comparisons against representative existing VLA methods. Based on what I've learned recently, the performance of this method on the LIBERO dataset does not yet seem sufficient to demonstrate that it offers any practical advantages.

Overall, the additional experiments improve the empirical analysis, but do not sufficiently strengthen the methodological novelty or demonstrate competitive performance against existing approaches. I therefore maintain my rating.

Thank you for your reply and for acknowledging the additional experiments. We would like to clarify two points.

Advantage over existing fusion strategies. Our paper does not propose a new fusion method. Bidirectional attention is already the fusion strategy of prominent VLAs such as π0, π0.5 or FLOWER, but whether and why it helps had not been tested in a controlled way. We provide this test at matched data, action head and parameters, and compare the learned mask against the main alternative fusion routes: fusion through later text tokens in a causal backbone (learned bidirectional masks +2.4 / +2.2) and fusion through a layer-wise action head (+3.5 / +2.0). Other fusion designs add modules on top of a causal backbone, such as cross-attention layers, learned query or readout tokens, or gates on the visual tokens. Compared with these, we only change the attention pattern and introduce no new parameters (PrefixLM), and the PrefixLM backbone remains a fully functional VLM: on 12 embodied spatial-reasoning benchmarks it scores as high as causal pretraining (0.660 vs. 0.657). A learned bidirectional mask is also complementary to other fusion approaches: it changes only which tokens may attend to each other, so it can be combined with any of these designs.

Practical advantage. We respectfully disagree that a state-of-the-art score on LIBERO or LIBERO-Plus is necessary to demonstrate the practical benefit of our work. We do not aim for the best possible performance on any benchmark, but study a mechanism (attention masks) and its effect on behavior generation. We show that learning a bidirectional mask before policy training increases performance and can be applied to different VLA backbones. Future research can build on these findings, which we believe is a substantial practical insight for the research community. A comparison with published VLAs would establish how a full system ranks on LIBERO-Plus. Our controlled comparisons instead isolate what practitioners gain by changing the attention mask during pretraining. The largest gains were +8.1 points on reworded instructions and +16.0 points on ABC-Sim. Across the settings we tested with full-attention backbones, we observed no statistically significant degradation relative to causal pretraining. For VLAs with a compatible backbone, this change requires no redesign of the action head, so state-of-the-art methods can adopt it during their own training.

To make sure we address your concern, could you kindly point us to the existing fusion methods you are referring to? We would be glad to discuss how our findings relate to them.

We are happy to answer further questions.

Reviewer_xRnVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the impact of bidirectional attention on Vision-Language-Action (VLA) models through six controlled configurations based on Qwen3-VL-2B. Experiments show that bidirectional attention improves manipulation performance when learned during embodied pretraining, while introducing it only during policy training can degrade performance. Through several mechanistic analyses, the authors demonstrate that bidirectional attention enables image tokens to encode task instructions, while pretraining helps preserve object-level attention selectivity.

Critical strengths
  1. The paper investigates an important problem concerning cross-modal interaction and attention mechanisms in VLA models.
  2. Multiple techniques, including activation patching, instruction transplantation, and attention-path knockouts, provide valuable insights into how instruction information influences actions.
Critical weaknesses
  1. The paper frames its flip vs. learned-mask contrast as the embodied analogue of LLM2Vec [1], so the main finding (flipping hurts, learning the mask helps) is largely expected. The novelty beyond prior work should be clarified.
  2. The causal role of object-level attention selectivity remains unclear. The link between selectivity and performance is only correlational across six configurations (Fig. 4), without an intervention.
  3. Comparisons with causal models using instruction-first token ordering or a short bidirectional adaptation stage are missing, making it difficult to isolate the benefits of bidirectional attention itself.
  4. Improvements over causally pretrained models are modest. As the authors acknowledge, the advantage disappears with a layer-wise action head and reverses in the Qwen3.5-0.8B control, limiting the generality of the findings.
Questions for authors
  1. Given LLM2Vec, which results in the paper would the authors say were not predictable from the language-model literature?
  2. Can the authors provide more direct evidence that object-level attention selectivity contributes causally to the performance improvements?
  3. Would an instruction-first causal model or a short bidirectional adaptation stage achieve comparable performance?
5 comments

Thank you for the careful review. We will answer each point below. Changes in the revised PDF are in blue and green where they address several reviewers.

W1 / Q1 (novelty vs LLM2Vec). We made the distinction and novelty clearer in the revised manuscript. LLM2Vec enables bidirectional attention in a decoder LLM and adapts it with masked next token prediction to obtain text embeddings. They do not study how this affects a policy's representation, which attention paths matter, or behavior generation. In our work, we instead show that bidirectional attention helps VLAs and thoroughly study the mechanism why it does so. The part that LLM2Vec predicts, that a short bidirectional adaptation helps, also holds for VLAs (W3: 77.4 vs 76.0 for causal PT). Results that do not follow from the LLM literature include:

  • That bidirectional attention helps for behavior generation in the first place.
  • Where the task is stored in a multimodal model. With a learned mask the instruction moves into the image tokens. Patching only the image tokens from a run with a different instruction makes the robot execute the donor task in closed loop in 78 to 93% of episodes (P5 and P6, both seeds), while the same wrong instruction without patching never does (0 / 200). This results in more task-specific image token representations, improving downstream behavior generation and robustness.
  • Which attention path causes the gain, tested at training time. New: we retrained P5 / P6 with only image-to-instruction attention blocked, everything else unchanged. Both lose their gain and fall to causal PT. Blocking it only in layers 0-18 costs as much as blocking it in all layers, while blocking it only in layers 19-27 costs nothing. As before, the wrist view carries most of it. Thus, fusion happens early on. We hypothesize that the later layers refine these representations towards action prediction.
  • A production VLA uses the same path. We additionally conducted a pi0.5 inference-time knockout of image-to-instruction attention. Blocking it in the early layers reproduces 91% of the full drop, while blocking the route in later layers does not result in a performance drop.

We believe this goes substantially beyond the findings of LLM2Vec and will highlight this distinction in the revised manuscript, placing greater emphasis on insights specific to behavior generation.

W2 / Q2 (causal role of object selectivity). Thanks for pointing this out. We agree that the experiments in their current form only showed a correlation. To improve upon this, we conduct three additional experiments. First, we steer selectivity directly during rollouts by adding a bias to the attention from the instruction to the object tokens (or to the background), with a random region of the same size as control. Second, we retrain P5 and P6 with the image-to-instruction attention blocked by the mask, in all layers or only in parts of the model. Third, we test whether the image tokens encode which object the instruction names, by running the same frames with their own and with a foreign instruction.

ExperimentResult
Steering selectivity at test timesuccess changes by less than 1 pp in every paradigm, same as the random region
Route blocked in training, all layersP5 -3.9 pp, P6 -3.2 pp (back to causal PT)
Route blocked in layers 0-18 onlyP5 -2.8 pp, P6 -3.7 pp
Route blocked in layers 19-27 onlyP5 +0.9 pp, P6 +0.7 pp (n.s., full gain over causal PT kept)
Route blocked for the wrist camera onlyP5 -2.5 pp, P6 -3.3 pp
Route blocked for the agent camera onlyP5 -0.7 pp, P6 +1.5 pp (agent-view selectivity drops as much as with full blocking)
Target-object probe, AUROC own vs foreign instruction (causal = 0.5)P5 and P6 0.59 to 0.63, peak at layers 14 to 17. pi0.5 0.60. P4 0.5 at all layers, P3 only in late layers (0.58)

These new results show that selectivity does not cause the gain but is instead a by-product of bidirectional attention. Instead, the early image-to-instruction route results in the increased performance, mainly through the wrist camera. We hypothesize that the route produces task-conditioned image features, so the image tokens already encode which object the task names. In the revision, RQ4 is rewritten around the route with the new route-ablation table, Fig. 4 now shows where in depth the image tokens read the instruction, and all three experiments are in Appendix K.

W3 / Q3 (instruction-first causal model, short bidirectional adaptation). To answer whether token ordering recovers the gain and if a shorter adaptation stage suffices in initializing the attention for downstream policy learning, we conduct two additional experiments. First, we train causal policies with the instruction placed before the images, so the image tokens can read it through the causal mask. Second, we run a short PrefixLM adaptation. We adapt plain Qwen3-VL-2B as a PrefixLM for 1,440 steps on generic VQA data without embodied data, and then train our policy on top (2 seeds).

PolicyHow image tokens read the instructionLIBERO-Plus success (%)
P1 plaincannot74.0
P1, instruction firsttoken order76.0
P2 causal PTcannot76.0
P2, instruction first (s42)token order75.0
P2, instruction before and after (s42)token order75.2
Short PrefixLM adaptationlearned mask77.4
P5 PrefixLMlearned mask78.3

Instruction first helps the plain backbone but not the causally pretrained one, and it stays below P5. It uses the same route, since blocking image-to-instruction attention costs it 53.8 points. This shows that simple token reordering helps the plain backbone. However, rich interaction through bidirectional exchange between modalities is what drives the gains. A shorter pretraining phase recovers most gains but is still behind a fully learned mask. We hypothesize that longer pretraining is necessary on pretraining-like data, as learning the attention pattern in a different domain (action generation with a flow-matching head) tries to learn two unseen patterns concurrently, negatively impacting the optimization. In the revision, both experiments are in Appendix K and the Discussion states this condition.

W4 (gains modest and not general). We agree that the paper did not show how general the gain is. To strengthen the claims and insights, we conduct additional experiments. First, we repeat the full matched comparison on different model sizes and families: Qwen3-VL-4B and SmolVLM2-500M. Second, we add the matched control for the layer-wise head, which Appendix I was missing. Third, we train a second seed of the Qwen3.5-0.8B pair. Finally, we evaluate on three harder settings: a bimanual robot (ABC-Sim), LIBERO-PRO and RoboCasa with reworded instructions.

Setting (success %)P2 causal PTP5 PrefixLMP6 conversionP5 vs P2P6 vs P2
SmolVLM2-0.5B59.965.562.7+5.7 [+3.3, +8.1]+2.9 [+0.3, +5.5]
Qwen3-VL-2B (paper)76.078.378.2+2.4 [+1.0, +3.7]+2.2 [+0.8, +3.7]
Qwen3-VL-4B73.979.580.3+5.5 [+3.6, +7.5]+6.4 [+4.8, +8.0]
Qwen3-VL-2B, layer-wise head75.779.277.7+3.5 [+2.0, +5.1]+2.0 [+0.4, +3.7]
ABC-Sim, bimanual29.443.8 (s42)45.4+9.2 [+2.1, +15.8]+16.0 [+10.8, +21.2]
LIBERO-PRO, new goals19.821.225.8+1.5 [-3.5, +5.8]+6.0 [+1.3, +11.7]
RoboCasa, reworded instructions53.361.555.9+8.1 [+3.1, +13.5]+2.6 [-2.3, +7.7]

The gain holds from 0.5B to 4B in two families and with both action heads. It is small on standard benchmarks and much larger with new instructions, new goals and on the bimanual robot. We hypothesize that the learned mask lets the backbone learn the grounding of language in the image during pretraining. A causal policy has to learn it from the demonstrations, which works for fixed instructions but not for new ones. We note that the experiments with Qwen3.5-0.8B can not be put in the same family as the others. Qwen3.5 models are hybrid attention and state-space models, with the majority of layers being linear attention layers. Thus, making these layers bidirectional is not trivial and outside of our studied setting. We will make this clearer in the main manuscript. In our current conversion, we added a backward recurrent scan instead of changing a mask. Its second seed shrinks the reversal from -3.3 to -0.7 points. In the revision, these results are in a new table in RQ1, Table 1 has ABC-Sim and LIBERO-PRO columns, and recurrent state-space models are stated as a limitation.

We hope this addresses all concerns and are happy to clarify any additional questions

Thank you for the response and new experiments. A few remaining points:

  1. Bidirectionality vs. a pretrained route. K.5 concludes that what matters is image tokens reading the instruction, learned before policy training. This differs from the main claim that bidirectional attention drives the gain. Your instruction-first P2 was pretrained image-first, so it is not a matched control. How do you separate the effect of bidirectionality from that of a route learned in pretraining?
  2. Generality. Some claims go beyond the intervals. For example, P6 − P2 with the second pretraining seed is +1.0 [−0.5, 2.6], and P6 vs. P1 with the layer-wise head is +0.1. Please tone down the wording accordingly.
  3. Clarifications. (a) The rebuttal suggests longer pretraining is necessary, but Sec. 4.2 says the short generic stage is sufficient. Which is intended? (b) Was the 4B causal baseline pretrained with the same recipe as 4B P5/P6? It is below the 2B baseline.

I will consider updating my score if the authors provide a convincing response.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.