- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how the placement of future prediction supervision affects the representations learned by vision-language-action policies. It compares prediction from learned special tokens with prediction directly from visual patch tokens, and uses controlled gradient routing interventions to examine how forecast errors shape visual representations. The study separates forecast quality from the accessibility of spatial, dynamics, and action information. Experiments include controlled representation learning, π0.5 and SmolVLA policies on LIBERO and LIBERO-PRO, and physical robot evaluation. The results suggest that spatially matched predictive supervision can improve representations useful for action and robustness under distribution shifts.
Critical strengths
-
The paper addresses an interesting representation-learning question with informative controlled experiments. Separating forecast quality from the information retained in the policy’s visual stream helps explain why a successful auxiliary predictor may not improve control. Figure 2 evaluates both forecast content and representation quality, while Figure 3 examines gradient delivery through interventions that preserve forward computation. These analyses provide useful evidence about how predictive supervision shapes visual features and offer a practical perspective on designing auxiliary objectives.
-
The evaluation examines the practical relevance of the findings across simulation and physical manipulation. Table 1 reports results over three training seeds, four LIBERO suites, and five perturbation families. Vision-token supervision achieves 66.3% mean LIBERO-PRO success, compared with 59.5% for the baseline and 63.5% for the anchored 256-token control. The shuffled-future variant falls to 59.2%, supporting the importance of scene-matched predictive content. The blinded, paired physical evaluation provides complementary evidence beyond simulation.
Critical weaknesses
-
The causal explanation is less firmly established at the policy scale than in the controlled representation study. Section 5 isolates gradient routing in prediction-only encoders, whereas Section 6 connects policy representations to robustness through interface comparisons and representation probes. For example, the reported correlation between displacement readout and position-shift success, r = 0.82, supports the proposed explanation but does not establish that improved dynamics representations cause the robustness gains. A comparable routing intervention within a jointly trained VLA would strengthen the connection between the controlled findings and closed-loop performance.
-
The physical results provide limited isolation of the proposed spatial mechanism. Figure 4 shows the largest advantage for vision-token supervision on the unseen-pot condition, but Section 6 acknowledges that this condition changes both the layout and the required grasp. The interface ordering also varies with the shift: under camera blur, the 256-token special-token variant has a slightly higher success point estimate than the vision-token variant, at 83% versus 80%. These results demonstrate promising physical performance, but leave unclear which aspects of the changed task account for the benefit of direct spatial supervision.
Questions for authors
-
Could the authors apply a gradient-routing intervention that preserves forward computation to a jointly trained VLA, to test whether altered gradient delivery causes the observed robustness gains?
-
Would the vision-token advantage over the 256-token special-token controls on the unseen-pot task persist under a layout-only shift with the grasp requirement held fixed?
2 comments
The responses partially address the concerns raised in this review. The reviewer therefore maintains the original rating.
Response to x7LS: Gradient Routing and Physical Diagnostics
Thank you for proposing a backward-only routing intervention and for asking whether the physical result persists when layout changes without changing grasp difficulty. We address both questions directly below.
Question 1. Can gradient routing be changed while preserving forward computation in a jointly trained VLA?
Yes. The requested policy-scale Pool-16 intervention lowers PRO from 66.3 to 62.3, or 60.5 on matched hardware, and position-shift success from 60.9 to 52.6, or 48.2 on matched hardware, while object performance is not reduced (77.6 vs. 78.9). Pool-16 averages the forecast gradient within 4×4 regions before returning it to visual patches. The forward pass, target, prediction head, losses, and action-gradient path remain unchanged. We report it in Section 6, Figure 5(c), and Appendix G.1.
The original runs lower PRO and position-shift success in all three seeds, with nominal Welch p = 0.024 and 0.018. For matched hardware, the two Pool-16 seeds first trained on a different GPU type from the vision-token reference were retrained on the same type. These reruns preserve the direction, with differences of 5.8 PRO points and 12.7 position points and p = 0.060 and 0.027. A one-seed shuffled-gradient diagnostic, rerun on the same GPU type, moves in the opposite direction, giving no evidence of a consistent hardware bias. Fine spatial gradient delivery therefore affects position-shift and PRO robustness in this setup, but does not explain the full interface ordering or identify the mediating feature.
Question 2. Does the vision-token advantage persist under a layout-only shift with grasp difficulty held fixed?
Our available diagnostic does not establish that claim. We added the 15-point grasp-grid result to Appendix E.2 and the pot-stage analysis to Figure 4 and Appendix G.5.
The modest grasp-matched grid does not distinguish the policies. Separately, successful grasps in the bowl tasks are followed by placement in 94–100% of trials. In the unseen-pot task, vision tokens lead both M = N = 256 controls on all eight discordant lid-removal blocks (p = 0.008). Placement after lid removal is conditional and does not establish equivalence. These rates are conditioned on different, self-selected trial sets (28 vs. 12 and 15 trials). We did not obtain a condition that fully isolates layout while holding grasp difficulty fixed. We therefore do not claim a general layout-only advantage and limit the physical finding to a shift that couples position with an unfamiliar precision grasp.
Other points raised in the review
In the physical blur condition, vision tokens win one block and lose four against the variable-index M = N = 256 arm (p = 0.375), confirming that the ordering is shift dependent. The corrected probe correlation is r = 0.74 and falls to 0.15 without the vision-token arm. We now treat it as descriptive rather than mediation evidence.
Revised claim and contribution
Pool-16 provides a forward-preserving intervention on policy-scale gradient routing, while the stage-level physical diagnostics and corrected correlation delimit its interpretation. The results show that fine spatial gradient delivery affects position-shift and PRO robustness while object performance follows a different pattern. This establishes supervision routing as a measurable VLA design variable in the tested setting, with the object, blur, and physical results defining the current boundary.