- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how attention structure and embodied pretraining affect instruction conditioning in vision-language-action models. It compares six training configurations, including causal attention, bidirectional attention introduced during policy training, PrefixLM pretraining, and encoder–decoder conversion. The evaluation combines downstream policy performance with activation patching, closed-loop feature transplantation, attention knockouts, and object-attention measurements. The results suggest that bidirectional embodied pretraining can improve instruction-conditioned visual representations and policy performance when the action head reads the final backbone layer, with benefits depending on the backbone and action-head design.
Critical strengths
-
Closed-loop transplantation provides behavioral evidence beyond attention visualization. Transplanting image features computed under a different instruction redirects the policy toward the donor task while keeping the observed scene fixed. This shows that instruction information carried by visual representations can influence task execution, strengthening the evidence beyond correlations in attention maps.
-
The unsuccessful bidirectional configurations are informative. P3 and P4 allow images to attend to instructions, yet their object-attention lift and policy performance do not consistently improve over causal configurations. These comparisons show that enabling the connection alone is insufficient. The additional backbone and action-head experiments also expose meaningful limits to the positive findings.
Critical weaknesses
-
The resulting design guidance remains narrow relative to existing bidirectional VLAs. π0 already uses bidirectional attention between image and instruction tokens, so the main contribution here is explaining when and why this design helps. However, P3 has lower object-attention lift than P1, the Qwen3.5-0.8B conversion performs worse than its causal baseline, and the policy advantage largely disappears with a layer-wise action head. These experiments vary different factors and do not establish a unified account of when bidirectionality helps or hurts. Although the paper acknowledges several limitations, the evidence provides limited guidance beyond the principal 2B, final-layer-head setting.
-
The interventions do not establish why the changed information flow improves policy performance. Closed-loop transplantation demonstrates that instruction-conditioned image features can control behavior, while inference-time knockouts reveal dependencies of trained checkpoints. These are useful mechanistic findings, but they do not explain why this fusion route should outperform fusion through subsequent text or action tokens. The trained configuration comparisons establish performance differences under the tested recipes, but do not isolate instruction information in image tokens as the cause of those differences. Likewise, the object-attention measurements do not establish that stronger object selectivity causes higher task success.
-
P6’s gains are not fully separated from additional pretraining capacity and computation. P6 uses approximately 3.5B parameters during embodied pretraining, compared with approximately 2.1B for P2 and P5. Matching downstream backbone size and pretraining examples therefore does not match pretraining resources. The larger causal model in Table 6 reaches comparable embodied reasoning performance, but no downstream policy results are reported for it. Consequently, the contribution of additional pretraining resources to P6’s policy gains remains unresolved. This concern applies specifically to P6 and does not invalidate the P5 comparison.
Questions for authors
-
How do you explain the differing outcomes across P3, the 0.8B backbone, and the layer-wise action head? Beyond the stated restriction to final-layer heads, what conditions determine whether bidirectional embodied pretraining is beneficial?
-
Can you provide matched causal/bidirectional policy comparisons at different sizes within the same backbone family, or vary the amount of embodied pretraining data and report downstream success? Figure 5 tracks reasoning performance during conversion, but does not establish how the downstream advantage changes with model or data scale.
3 I prefer to train a downstream policy from the existing 4B causal checkpoint in Table 6, using the same downstream data, action-head type, and training protocol. Since its reasoning score matches P6, this would clarify whether comparable reasoning performance translates into comparable control performance, and what advantage P6 offers relative to enlarging the causal backbone.
4 comments
W2 (why fusion through image tokens rather than text or action tokens?). Thank you, this is exactly what our earlier tests could not show. We conduct four additional experiments. First, we retrain P5 and P6 with only the image-to-instruction route removed, so the instruction still reaches the text tokens and the head. Second, we train causal policies with the instruction before the images. Third, on ABC-Sim we block the route at inference in early or late layers. Fourth, on ABC-Sim we stop the action head from reading the instruction tokens, alone and together with the image route block.
| Route removed (pp) | P5 | P6 |
|---|---|---|
| LIBERO-Plus (training time) | -3.9 [-5.5, -2.2] | -3.2 [-4.7, -1.6] |
| LIBERO-PRO, paraphrased (training time) | -2.9 [-7.0, -0.1] | -2.1 [-4.5, -0.3] |
| LIBERO-PRO, new goals (training time) | -1.0 [-3.7, +1.9] | -5.5 [-9.0, -2.4] |
| RoboCasa, reworded (training time) | -3.7 [-7.5, -0.1] | -7.7 [-11.5, -4.1] |
| ABC-Sim, layers 0-18 at inference | -14.8 [-19.8, -9.8] | |
| ABC-Sim, layers 19-27 at inference | +2.9 [-2.1, +7.7] | |
| ABC-Sim, head cannot read the instruction | -8.8 [-15.8, -1.7] | -2.9 [-7.7, +1.9] |
| ABC-Sim, head and image route blocked | -18.3 [-25.4, -11.2] | -20.6 [-25.8, -15.4] |
| Policy (LIBERO-Plus %) | Image tokens read the instruction via | Success |
|---|---|---|
| P2 causal PT | cannot | 76.0 |
| P2, instruction first (s42) | token order | 75.0 |
| P1 plain, instruction first | token order | 76.0 |
| P5 PrefixLM | learned mask | 78.3 |
| P5, route removed | cannot | 74.4 |
Removing only the image route costs most or all of the gain on every benchmark, although the text tokens and the head still see the instruction. On ABC-Sim, blocking the early route removes the whole 16-point gain of P6. So fusion through later text tokens is not an equivalent substitute. The head test shows this directly. When the action head of P6 cannot read the instruction tokens, P6 loses only 2.9 points and stays 12.7 points above P2, while the same restriction costs the causal P2 16.7 points. Only when the image route is blocked as well does P6 fall to the level of P2. So in P6 the instruction reaches the actions through the image tokens. A new probe shows what the route adds. In P5 and P6 the image tokens of the target object change with the instruction from about layer 4 on, so they encode which object the task names. We hypothesize that the early route produces text-conditioned image features, mainly for the wrist camera, so the head receives tokens that already encode which object to approach. A causal policy has to learn this binding from demonstrations, which is slower and does not transfer to new wordings. In the revision, the route ablation is the causal test of RQ4 in the main text, and the instruction-order controls and the probe are in Appendix K. We further hypothesize that the route emerges through the modality the policy relies on most. In our policies this is the wrist camera (blocking the route only for the wrist camera costs most of the gain), which is why other features are merged into it.
W3 / Q3 (P6 capacity, policy from the 4B causal checkpoint). Thank you for the precise suggestion. We conduct two additional experiments. First, we train policies from the existing 4B causal checkpoint with the same data, head and protocol (2 seeds). Second, we pretrain PrefixLM and conversion variants of Qwen3-VL-4B on the same data and budget and train policies from them.
| Backbone (LIBERO-Plus %) | Causal PT | PrefixLM | Conversion |
|---|---|---|---|
| Qwen3-VL-2B | 76.0 | 78.3 | 78.2 |
| Qwen3-VL-4B | 73.9 | 79.5 | 80.3 |
The 4B causal policy is 4.3 points below 2B P6, although its embodied VQA score matches P6. More causal capacity does not reproduce the gain. P5 has the same parameters as P2 and gains as much as P6, and at 4B both learned masks gain more than at 2B. Over 12 backbones, embodied VQA scores do not predict policy success (r = -0.32). We hypothesize that P6 offers an attention pattern, not capacity: image tokens that read the instruction, which a causal model with images first cannot provide. In the revision, the 4B results are in the new RQ1 table, and P5 is stated as the parameter-matched control.
Q2 (model size and pretraining data). We ran both. First, the 4B comparison above. Second, we train policies from intermediate pretraining checkpoints after 1M, 2M and 4M of 6.5M samples (s42).
| LIBERO-Plus (pp) | P5 vs P2 | P6 vs P2 |
|---|---|---|
| Qwen3-VL-2B, 6.5M samples | +2.4 [+1.0, +3.7] | +2.2 [+0.8, +3.7] |
| Qwen3-VL-4B, 6.5M samples | +5.5 [+3.6, +7.5] | +6.4 [+4.8, +8.0] |
| Qwen3-VL-2B, 1M samples | +2.8 [+0.6, +5.0] | +4.0 [+1.9, +6.2] |
| Qwen3-VL-2B, 2M samples | +2.5 [+0.9, +4.2] | +0.7 [-1.5, +2.9] |
| Qwen3-VL-2B, 4M samples | +1.4 [-0.8, +3.7] | -1.0 [-3.5, +1.6] |
The gain is larger at 4B than at 2B. P5 is ahead from the first million samples on, which fits the result that a short generic PrefixLM stage recovers most of the gain. P6 is unstable in the middle of pretraining. We hypothesize that this is because conversion is a larger architectural change, and the policy has to adapt to the new encoder-decoder architecture. In the revision, both results are in Appendix K.
Thank you for the additional experiments. The training-time ablation somehow addresses my mechanistic concern. However, I still have three remaining reservations:
-
Token order. The findings in K.6 motivate a causal alternative: placing the instruction before the images so that image tokens can attend to it. K.5 tests this order during policy training, but leaves open whether using it throughout both pretraining and policy training could match bidirectional attention.
-
Practical significance. In the main 2B setting, the bidirectional models improve LIBERO-Plus success by about 2–2.5 percentage points over causal pretraining. I find this gain modest as a basis for a broader training recommendation.
-
Scaling. The larger gap at 4B partly reflects a lower causal baseline (73.9% at 4B versus 76.0% at 2B). This makes it difficult to conclude that bidirectional attention becomes more beneficial with model scale.
Therefore, I keep my original rating.
Thank you for the thoughtful review, and in particular for asking for one account of the negative cases. Changes in the revised PDF are purple, and green where they address several reviewers.
W1 / Q1 (one account of when bidirectionality helps). We agree that the paper presented these cases as separate caveats. We conduct one additional experiment per case. First, for the flip P3 we measure where in depth its image tokens read the instruction, train it for 60k instead of 30k steps, and apply the same kind of flip on a bimanual robot (ABC-Sim). Second, for Qwen3.5-0.8B we train a second seed and repeat the comparison on SmolVLM2-500M, a full-attention model of similar size, as well as a larger model (Qwen3-VL-4B). Third, for the layer-wise head we add a matched control. Our initial experiments compared against the non-pretrained model. Fourth, we block image-to-instruction attention in pi0.5 in early or late layers.
All cases follow one rule. Bidirectional attention helps when image tokens read the instruction in the early layers, through a mask on full-attention layers that the backbone learned before policy training. P3 learns the mask only at policy time, reads the instruction late and needs twice the budget to catch up, and the same flip collapses on a different embodiment. Qwen3.5-0.8B is mostly linear attention, where conversion is not a mask change. On every full-attention backbone we tested, the gains replicate. The layer-wise head follows the rule once it has the matched control. pi0 and pi0.5 also rely on the early layers. We hypothesize that the learned mask lets the backbone compute task-conditioned image features early, and that this is learned best before the policy sees few demonstrations with fixed instructions. In the revision, the Discussion states this one rule instead of separate caveats, and the cases are in Appendix K.