acceptodds
Under review as a conference paper at ICLR 2027

Where Predictive Supervision Goes Shapes What VLA Policies Learn

Abstract

Future prediction is increasingly used to improve vision-language-action (VLA) policies, based on the premise that anticipating scene evolution encourages representations useful for control. However, forecast quality alone does not establish that a policy has learned a better representation for action. This distinction matters under distribution shift, where successful control depends on preserving spatial state and likely scene change beyond familiar configurations. We study what determines whether predictive supervision improves the visual representation used by a VLA policy. Through controlled comparisons with matched target constructions, prediction horizons, and training conditions, we find that different prediction interfaces produce markedly different forecasts and visual representations, including in the spatial, dynamics, and action information that transfers beyond familiar scenes. We trace these differences to how predictive errors shape the policy's visual stream. Consistent with this controlled finding, VLA policies trained with more direct, scene-matched future supervision show stronger robustness under simulated and physical distribution shifts. Together, our results frame future prediction as a representation-learning design problem whose value for control depends on whether its supervision reaches the representations through which the policy acts.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Future Feature Prediction, Spatial Credit Assignment, Visual Representation Learning, Robust Robot Manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_x7LSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how the placement of future prediction supervision affects the representations learned by vision-language-action policies. It compares prediction from learned special tokens with prediction directly from visual patch tokens, and uses controlled gradient routing interventions to examine how forecast errors shape visual representations. The study separates forecast quality from the accessibility of spatial, dynamics, and action information. Experiments include controlled representation learning, π0.5 and SmolVLA policies on LIBERO and LIBERO-PRO, and physical robot evaluation. The results suggest that spatially matched predictive supervision can improve representations useful for action and robustness under distribution shifts.

Critical strengths
  1. The paper addresses an interesting representation-learning question with informative controlled experiments. Separating forecast quality from the information retained in the policy’s visual stream helps explain why a successful auxiliary predictor may not improve control. Figure 2 evaluates both forecast content and representation quality, while Figure 3 examines gradient delivery through interventions that preserve forward computation. These analyses provide useful evidence about how predictive supervision shapes visual features and offer a practical perspective on designing auxiliary objectives.

  2. The evaluation examines the practical relevance of the findings across simulation and physical manipulation. Table 1 reports results over three training seeds, four LIBERO suites, and five perturbation families. Vision-token supervision achieves 66.3% mean LIBERO-PRO success, compared with 59.5% for the baseline and 63.5% for the anchored 256-token control. The shuffled-future variant falls to 59.2%, supporting the importance of scene-matched predictive content. The blinded, paired physical evaluation provides complementary evidence beyond simulation.

Critical weaknesses
  1. The causal explanation is less firmly established at the policy scale than in the controlled representation study. Section 5 isolates gradient routing in prediction-only encoders, whereas Section 6 connects policy representations to robustness through interface comparisons and representation probes. For example, the reported correlation between displacement readout and position-shift success, r = 0.82, supports the proposed explanation but does not establish that improved dynamics representations cause the robustness gains. A comparable routing intervention within a jointly trained VLA would strengthen the connection between the controlled findings and closed-loop performance.

  2. The physical results provide limited isolation of the proposed spatial mechanism. Figure 4 shows the largest advantage for vision-token supervision on the unseen-pot condition, but Section 6 acknowledges that this condition changes both the layout and the required grasp. The interface ordering also varies with the shift: under camera blur, the 256-token special-token variant has a slightly higher success point estimate than the vision-token variant, at 83% versus 80%. These results demonstrate promising physical performance, but leave unclear which aspects of the changed task account for the benefit of direct spatial supervision.

Questions for authors
  1. Could the authors apply a gradient-routing intervention that preserves forward computation to a jointly trained VLA, to test whether altered gradient delivery causes the observed robustness gains?

  2. Would the vision-token advantage over the 256-token special-token controls on the unseen-pot task persist under a layout-only shift with the grasp requirement held fixed?

2 comments
AuthorsOpenReview ↗

Response to x7LS: Gradient Routing and Physical Diagnostics

Thank you for proposing a backward-only routing intervention and for asking whether the physical result persists when layout changes without changing grasp difficulty. We address both questions directly below.

Question 1. Can gradient routing be changed while preserving forward computation in a jointly trained VLA?

Yes. The requested policy-scale Pool-16 intervention lowers PRO from 66.3 to 62.3, or 60.5 on matched hardware, and position-shift success from 60.9 to 52.6, or 48.2 on matched hardware, while object performance is not reduced (77.6 vs. 78.9). Pool-16 averages the forecast gradient within 4×4 regions before returning it to visual patches. The forward pass, target, prediction head, losses, and action-gradient path remain unchanged. We report it in Section 6, Figure 5(c), and Appendix G.1.

ConditionPROPositionEnvironmentObject
Vision-token reference66.360.958.877.6
Pool-16, original runs62.352.652.878.9
Pool-16, matched hardware60.548.249.178.9

The original runs lower PRO and position-shift success in all three seeds, with nominal Welch p = 0.024 and 0.018. For matched hardware, the two Pool-16 seeds first trained on a different GPU type from the vision-token reference were retrained on the same type. These reruns preserve the direction, with differences of 5.8 PRO points and 12.7 position points and p = 0.060 and 0.027. A one-seed shuffled-gradient diagnostic, rerun on the same GPU type, moves in the opposite direction, giving no evidence of a consistent hardware bias. Fine spatial gradient delivery therefore affects position-shift and PRO robustness in this setup, but does not explain the full interface ordering or identify the mediating feature.

Question 2. Does the vision-token advantage persist under a layout-only shift with grasp difficulty held fixed?

Our available diagnostic does not establish that claim. We added the 15-point grasp-grid result to Appendix E.2 and the pot-stage analysis to Figure 4 and Appendix G.5.

DiagnosticBaselineM = 16M = N = 256 (variable index)M = N = 256 (anchored index)Vision
Bowl-free 15-point grasp grid13/1511/1513/1512/1513/15
Unseen-pot lid removal--36%45%85%
Placement given that the lid is off--11/1215/1523/28

The modest grasp-matched grid does not distinguish the policies. Separately, successful grasps in the bowl tasks are followed by placement in 94–100% of trials. In the unseen-pot task, vision tokens lead both M = N = 256 controls on all eight discordant lid-removal blocks (p = 0.008). Placement after lid removal is conditional and does not establish equivalence. These rates are conditioned on different, self-selected trial sets (28 vs. 12 and 15 trials). We did not obtain a condition that fully isolates layout while holding grasp difficulty fixed. We therefore do not claim a general layout-only advantage and limit the physical finding to a shift that couples position with an unfamiliar precision grasp.

Other points raised in the review

In the physical blur condition, vision tokens win one block and lose four against the variable-index M = N = 256 arm (p = 0.375), confirming that the ordering is shift dependent. The corrected probe correlation is r = 0.74 and falls to 0.15 without the vision-token arm. We now treat it as descriptive rather than mediation evidence.

Revised claim and contribution

Pool-16 provides a forward-preserving intervention on policy-scale gradient routing, while the stage-level physical diagnostics and corrected correlation delimit its interpretation. The results show that fine spatial gradient delivery affects position-shift and PRO robustness while object performance follows a different pattern. This establishes supervision routing as a measurable VLA design variable in the tested setting, with the object, blur, and physical results defining the current boundary.

The responses partially address the concerns raised in this review. The reviewer therefore maintains the original rating.

Reviewer_uEFHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper works on the future prediction problems on a VLA policy, specifically, what determines the impact of future prediction on the visual representation used by a VLA policy. The authors study how different prediction interfaces will influence the robustness of the VLA policy, and point out that VLA policies trained with more direct, scene-matched future supervision show stronger robustness.

Critical strengths
  1. The paper argues that the matched future prediction can produce different visual representations and different prediction interfaces will determine whether spatial and temporal information remains accessible in its visual representation. The paper give an analysis on special-token and vision-token interfaces under a matched learning problem.
  2. The paper also conducts experiments on LIBERO and LIBERO-PRO benchmark to show the representation-level distinction will influence VLA robostness.
Critical weaknesses
  1. The paper only give results on LIBERO and LIBERO-PRO simulation benchmarks, which is not enough to support their arguments.
  2. The innovation is not impressive. The real-world VLA experiments are too simple to verify the conclusion.
Questions for authors
  1. The authors need to answer the above questions and give enough simulated and real-world tests to further analyze and verify their arguments.
1 comment
AuthorsOpenReview ↗

Response to uEFH: Evidence Scope and Representation-Learning Analysis

Thank you for raising concerns about evaluation breadth and methodological novelty. We agree that the evidence supports a bounded robustness claim rather than broad generalization across simulators and physical platforms.

Question for the authors. Can the paper provide sufficient simulated and real-world evidence for the robustness claim?

The original submission evaluated π0.5 on four LIBERO suites and five LIBERO-PRO shift families, included a three-seed SmolVLA replication, and reported a blinded physical study. LIBERO-PRO is useful for this analysis because its factorized shifts let us distinguish position, object, and environment robustness rather than report only aggregate success. PRO denotes mean LIBERO-PRO success. We did not add another simulation benchmark during the discussion. SmolVLA tests transfer to a second backbone, not benchmark breadth. Its prediction heads are not capacity matched, so we use it only as a replication of the interface ordering.

Future-prediction supervision is not defined by its target alone. Scene-matched content, spatial address, and the route by which forecast gradients enter the visual stream affect which robustness gains are learned and retained in the policy representation.

What the evidence supportsInterventionResult
Scene-matched future content contributes to the gainsSubmitted shuffled-future arm, 3 seedsPosition falls from 60.9 to 42.8 and PRO from 66.3 to 59.2, while object changes little
Fine spatial gradient delivery affects the gainsPool-16, 3 seeds. It pools only backward forecast gradients within 4×4 regions and leaves the forward pass unchangedPosition falls to 52.6, or 48.2 on matched hardware, and PRO to 62.3, or 60.5, while object remains unchanged
The gain remains available in the representationA fresh action expert is trained on frozen backbones, 3 matched seedsThe vision-minus-baseline PRO gain remains 7.3 points, compared with 6.8 under joint training

Together, these interventions separate target content, gradient delivery, and representation retention. The position and PRO gains are sensitive to scene-matched future information and fine spatial gradient delivery, and the resulting advantage remains available to a newly trained action expert. Object robustness follows a different pattern and remains outside this account.

The physical study serves a different role. It is a controlled transfer check rather than broad deployment validation. Its three tasks require selecting a specified object from clutter, picking two objects in order, and opening a pot before placing an object inside. Five policies are compared over 28 blinded matched blocks and 420 trials. Across 21 shifted blocks, vision tokens beat the baseline (11 wins, no losses, p = 0.001) and M = 16 (11 wins, 2 losses, p = 0.022), while comparisons with both M = N = 256 controls remain unresolved. In the unseen pot layout, the stage analysis places the difference at the unfamiliar lid interaction. Each variant has one independently trained policy, so the intervals measure evaluation uncertainty rather than policy-training variation. We therefore limit the physical claim to one platform, three tasks, and one training run per policy.

Novelty and significance of the analysis

Our contribution is methodological rather than a more complex architecture. Existing work establishes that predictive and geometric supervision can improve VLA policies, but typically changes targets, carriers, modules, or training stages together. FLARE adds a few tokens and aligns policy features with future latents, FoMoVLA combines foresight tokens with point-trajectory supervision, GLaD distills geometry into visual-token states, and AHEAD forecasts patch features around a frozen policy. Our controlled encoder and π0.5 comparisons instead use the same target definition, horizon, head capacity, backbone, and optimization while varying spatial address and forecast-gradient delivery.

This connects predictive VLA training to a broader representation-learning finding. DeepMIM shows that intermediate supervision changes layerwise representation quality, while MIM-Refiner uses intermediate-layer heads to improve transferable features. Our results bring this question to future-prediction supervision in a trainable policy and identify target content, spatial address, and gradient delivery as measurable design variables. Broader benchmarks, physical platforms, and training seeds remain the next validation scale.

Relevant references

FLARE, FoMoVLA, GLaD, AHEAD, DeepMIM, and MIM-Refiner.

Reviewer_NRSGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how the location of future prediction supervision affects the visual representation learned by a vision-language-action (VLA) policy. Through controlled comparisons of prediction interfaces with specific and visual labels, the authors demonstrate that different supervision paths lead to different prediction behaviors and learned representations, highlighting the importance of how prediction gradients reach the visual flow. Experimental results show that direct, spatially aligned, and scene-matched future supervision makes spatial state, future dynamics, and action-related information more readily accessible in the visual representation and improves robustness to distribution shifts in both simulated and real robot manipulation.

Critical strengths

This paper explores an interesting and important question: how future prediction supervision influences representation learning in VLA policies. Through systematic experimental comparisons, representation probing, and gradient routing analysis, this paper gains insights into the differences between various prediction interfaces. Evaluations using simulations and real robot operations further reinforce the experimental results and demonstrate their practical application value.

Critical weaknesses
  1. The performance gap narrows substantially when special tokens use stable positional indexing and a frozen SigLIP target (Appendix B.4), raising concerns about whether the observed advantage can be attributed to direct supervision alone rather than differences in spatial alignment and target design.
  2. The causal link between supervision routing and action robustness is not fully established. In Table C.4, correctly routed attention-mediated gradients improve forecasting and spatial representations but do not consistently improve action decoding. Moreover, the link between learned future-dynamics representations and policy robustness remains correlational.
  3. The special-token and vision-token heads differ substantially in parameter count (8.4M vs. 1.1M), introducing a potential confounding factor when isolating the effects of supervision placement from prediction-head capacity and optimization.
Questions for authors
  1. How much of the interface advantage remains when spatial indexing and prediction targets are properly controlled?
  2. What evidence establishes a causal link between the learned future-dynamics representations and improved action robustness?
  3. A comparison with matched prediction-head capacities would help clarify whether this difference affects the results.
1 comment
AuthorsOpenReview ↗

Response to NRSG: Target, Addressing, and Gradient Controls

Thank you for identifying target evolution, positional addressing, gradient delivery, and head capacity as possible confounds. We agree that these factors must be separated before interpreting the interface comparison.

Question 1. How much of the interface advantage remains after controlling positional addressing and target evolution?

In the controlled encoder study, RoPE-based addressing closes most of the gap under frozen SigLIP. Forecast gap closure, effective rank, and spatial-tail action R² nearly match, while vision tokens remain higher on spatial-tail position R² (0.56 vs. 0.47) and identity accuracy (0.55 vs. 0.33). Under the momentum target, vision tokens have higher mean spatial-tail action R² (0.69 vs. 0.58). We report this expanded interpretation in Appendix B.4.

At policy scale, we added a shared frozen-target comparison in Appendix G.3. Vision minus the anchored M = N = 256 special-token arm is +2.3 PRO, +7.0 position, and +2.7 object points. However, PRO and position are unresolved with three seeds (Welch p = 0.11 and 0.17), and the ordering changes across shift families. In the one run whose intermediate target checkpoints were saved, 75–77% of target weights remained bitwise identical to pretrained SigLIP because small bfloat16 updates rounded away. The frozen-target result does not support target co-evolution as a complete explanation, but it also does not establish a target-independent interface advantage.

Question 2. What evidence links the supervision route to policy robustness?

We added gradient-delivery and frozen-backbone interventions and re-analyzed the submitted shuffled-future arm by shift family.

ComparisonPRO gapPosition gapObject gap
Matched-future vision minus shuffled-future vision+7.1+18.1+1.0
Vision minus policy-scale Pool-16+4.0+8.3-1.3
Vision minus policy-scale Pool-16 on matched hardware+5.8+12.7-1.3
Vision minus baseline after freezing the backbone+7.3+11.7+5.8

Policy-scale Pool-16 pools only the forecast gradient within 4×4 regions while preserving forward computation. It reduces position-shift and PRO performance while leaving object performance unchanged. The original runs give nominal Welch p = 0.024 and 0.018. In the matched-hardware reruns, the two Pool-16 seeds first trained on a different GPU type from the vision-token reference were retrained on the same type, giving p = 0.060 and 0.027. The submitted shuffled-future arm removes the position-shift and PRO gains, while frozen-backbone retraining preserves the gain with a fresh action expert. Table C.4 gives a complementary boundary. Correctly addressed attention-returned gradients improve forecast gap closure and spatial-tail position R², but not spatial-tail action R² (0.579 vs. 0.580). These results show that content and gradient delivery affect robustness, but they do not identify the mediating feature.

We also corrected the probe correlation from r = 0.82 to r = 0.74 for the variants defined in the paper. It falls to 0.15 without the vision-token arm, so we now treat it as descriptive rather than mediation evidence.

Question 3. Are the forecast heads capacity matched?

Yes for π0.5. All three forecast heads use the same 8.4M-parameter LayerNorm–MLP. The 1.1M value in the submitted Appendix D.1 was a reporting error. SmolVLA uses different head structures and is now presented only as a second-backbone replication, not a capacity-matched ablation. Appendices D.1 and D.2 contain the correction.

Revised claim and contribution

These controls distinguish spatial addressing from gradient delivery, bound the role of target evolution, and correct the head-capacity report. Stable addressing recovers much of the special-token deficit, while scene-matched targets and fine spatial gradient delivery affect position-shift and PRO robustness at policy scale. The resulting contribution is a controlled representation-learning analysis of future-prediction supervision, with object robustness marking the boundary of the current explanation.

Reviewer_Z3FROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether it matters where future-prediction supervision attaches inside a VLA. With the prediction target held fixed, it compares forecasting from special tokens with forecasting directly from vision tokens, and attributes the large difference in learned representations to how spatially aligned the prediction gradients are when they reach the policy's vision tokens. Policy-scale experiments on LIBERO, LIBERO-PRO, and a physical robot then test whether these differences carry over to robustness under distribution shift. The controlled analysis is the strongest part of the paper. The physical evidence is suggestive but rests on one training run per variant.

Critical strengths
  1. The paper separates two questions that work on predictive policies often mixes together: what is predicted, and which representation receives the prediction gradient. This distinction is kept throughout, and the progression from the controlled comparison in Sections 4-5 to the policy-scale experiments in Section 6 is easy to follow.

  2. The controlled experiments isolate a mechanism. Section 4 fixes the forecasting target, horizon, backbone, and optimization and changes only the readout interface, and Table B.4 shows large differences in probes for spatial-tail position, future displacement, action, and geometry. Section 5 then tests the explanation directly. The aligned-path intervention (Figure 3(a), Table C.2) recovers representation breadth and transfer, and the backward-only routing intervention (Figure 3(b)) separates correctly routed attention gradients from direct coupling.

  3. The policy-scale controls in Table 1 are well chosen. Matching the number of special tokens to the patch granularity gives little benefit, anchoring their spatial indices recovers part of the object-position gap, and shuffling the future targets removes the gain while keeping the direct route. Together, these results support the paper's account in which both spatially structured routing and scene-matched future content matter. Appendix D.2 adds that the key simulation comparisons hold within each matched training seed.

Critical weaknesses
  1. The methodological novelty is limited. The main contribution is a systematic study of where prediction supervision should be applied. Direct supervision of vision tokens is a simple design. The paper’s value lies mainly in its analysis and empirical findings.

  2. The findings may not generalize to other designs. Better positional encoding or prediction targets also improve special-token methods. The advantage of direct vision-token supervision may therefore depend on the setup. It remains unclear whether it holds for other prediction targets or spatially grounded object- and region-level tokens.

  3. The link between better representations and greater robustness is not fully established. The controlled experiments show how supervision affects representations. In large VLAs, representation quality and OOD success improve together. However, the experiments do not isolate how much of the robustness gain comes from better spatial and action information versus other effects on training.

Questions for authors
  1. Were any independently initialized robot policies trained but left out of Table E.4? If not, would the authors revise the wording in Section 6 to distinguish uncertainty over evaluation trials from uncertainty over training?

  2. Would a sparse but explicitly addressed object- or region-centric carrier keep the transfer benefits of direct patchwise coupling? This would separate the role of dense patch coverage from the role of stable, direct spatial credit.

3 comments
AuthorsOpenReview ↗

Response to Z3FR: Physical Evaluation, Sparse Carriers, and Representation Retention

Thank you for asking whether all physical policies were reported, whether sparse addressed carriers can work, and whether the policy gain remains in the representation. These questions helped us clarify both the evidence and the scope of the contribution.

Question 1. Were independently initialized physical policies omitted from Table E.4?

No. Table E.4 reports all five policies trained on the physical dataset. Each policy was trained once with the same seed and split. The intervals resample matched evaluation blocks and therefore measure evaluation uncertainty, not variation across policy-training runs. We revised Section 6 and Appendix E to state this directly.

Question 2. Can a sparse but explicitly addressed region- or object-centric carrier recover the benefit?

We added the region-carrier results from the controlled encoder study with N = 64 patches to Appendix B.5. Each cell reports forecast gap closure, spatial-tail position R², and spatial-tail action R². Entries are five-seed means. The vision-token column repeats the reference, which has no region index. The dash marks one unavailable value.

Target and addressM = 16M = 32Vision tokens
Frozen SigLIP, native address37.3 / 0.54 / 0.5747.1 / 0.54 / 0.6248.9 / 0.56 / 0.45
Frozen SigLIP, region index37.2 / 0.55 / 0.6446.1 / 0.57 / 0.6048.9 / 0.56 / 0.45
Momentum, native address3.5 / 0.35 / 0.428.8 / 0.31 / 0.3034.5 / 0.50 / 0.69
Momentum, region index8.3 / -- / 0.2623.4 / 0.46 / 0.5134.5 / 0.50 / 0.69

A single carrier loses spatial-tail position information (R² = 0.25), so sparsity is not cost-free. With frozen SigLIP, M = 16 and M = 32 retain spatial-tail position and action R² despite much lower effective rank than vision tokens. Under the momentum target, a stable region index raises M=32 gap closure from 8.8% to 23.4% and improves both transfer measures, although it does not match the vision-token reference. Dense coverage is therefore not always necessary. Object-centric carriers remain untested.

Weakness 3. How much of the policy gain is retained in the representation?

We added Pool-16 and frozen-backbone retraining in Appendices G.1 and G.2 and a shift-wise analysis of the submitted shuffled-future arm in Appendix G.4.

ComparisonPRO gapPosition gapObject gap
Matched-future vision minus shuffled-future vision+7.1+18.1+1.0
Vision minus policy-scale Pool-16, original runs+4.0+8.3-1.3
Vision minus policy-scale Pool-16, matched hardware+5.8+12.7-1.3
Vision minus baseline after freezing the backbone+7.3+11.7+5.8

The submitted shuffled-future arm shows that scene-matched content is required for the position-shift and PRO gains. Policy-scale Pool-16 pools only the forecast gradient within 4×4 regions and affects the same outcomes without explaining object robustness. Frozen-backbone retraining preserves the gain after training a fresh action expert, showing that it remains in the representation. These experiments do not yet identify whether spatial, action, or future-dynamics information mediates the gain.

Revised claim and contribution

The added evidence clarifies the physical comparison, shows that sparse explicitly addressed carriers can retain spatial-tail position and action R², and separates representation retention from the still-open question of which feature mediates the gain. Shuffled targets, Pool-16, and frozen-backbone retraining show that scene-matched content and gradient route affect what transfers to the policy representation. Unlike AHEAD, FoMoVLA, and GLaD, our contribution is a controlled representation-learning analysis that holds the target and training setup fixed and intervenes on the supervision route, with fixed-horizon visual prediction on two backbones defining the current scope.

Thanks for the additional experiments and the clearer wording. Your explanation of how you ran the real-world experiments answers Q1 for me. The frozen-backbone and Pool-16 results also show that the gains in simulation come from the visual backbone. Overall, I’m keeping my score at 2.

  1. The new Q2 table leaves both questions open. With SigLIP frozen, sparse carriers (M=16,32M = 16, 32) match vision tokens on position R2R^2 and outperform them on action R2R^2 (0.57–0.64 versus 0.45). In the momentum setting at M=16M = 16, adding region indexing brings action R2R^2 down from 0.42 to 0.26. The table doesn’t report position R2R^2 for that comparison, and object-centric carriers are still untested.

  2. My concern in W3 also remains open. The probes show which features can be decoded, but it’s still unclear whether those specific features make the policy more robust.

Thank you for the follow-up and for confirming that Q1 is resolved and that the simulation gains are retained in the visual backbone.

Q2. We recovered the missing momentum-target position value for the region-indexed M=16M=16 carrier. Each entry below reports forecast gap closure (%) / spatial-tail position R2R^2 / spatial-tail action R2R^2, averaged over five seeds.

Target and addressM=1M=1M=16M=16M=32M=32Vision tokens
Frozen SigLIP, native13.2 / .25 / .4437.3 / .54 / .5747.1 / .54 / .6248.9 / .56 / .45
Frozen SigLIP, region index13.2 / .25 / .4437.2 / .55 / .6446.1 / .57 / .6048.9 / .56 / .45
Momentum, native0.9 / .12 / .173.5 / .35 / .428.8 / .31 / .3034.5 / .50 / .69
Momentum, region index0.9 / .12 / .178.3 / .40 / .2623.4 / .46 / .5134.5 / .50 / .69

For M=1M=1, the region and native indices are identical. The vision-token column repeats the shared reference.

Under frozen SigLIP, sparse carriers match vision tokens on position and exceed them on action, although their representations remain substantially lower-rank. Under the momentum target, all sparse variants remain below vision tokens. Region indexing has no resolved benefit at M=16M=16, but improves forecasting and position transfer at M=32M=32.

Under full coverage, the momentum-target runs have higher action readout than their frozen-target counterparts. Action R2R^2 rises from 0.45 to 0.69 for vision tokens and from 0.41 to 0.58 for the stably addressed M=NM=N special-token interface. This is consistent with additional action-relevant learning when carrier coverage and gradient delivery are sufficient. It does not establish target evolution alone as the cause of the vision-token advantage. Object-centric carriers remain untested.

W3. We agree that decodability alone does not identify the feature that mediates robustness. The table compares final-layer grid readouts on position-shift frames with LIBERO-PRO position-shift success. Values are three-seed means from Table D.4 and Appendix G.1.

ConditionInterventionPosition R2R^2Displacement R2R^2Position success (%)
Vision tokensReference.545.50460.9
Shuffled futureBreaks scene-future pairing.539.41942.8
Pool-16Pools only the forecast gradient.455.54552.6

Shuffled future nearly preserves position decoding while success falls. Pool-16 raises displacement decoding while success falls. Success is lower than the vision-token reference in every seed of both interventions. Neither readout alone is therefore sufficient to explain the robustness gain.

Frozen-backbone retraining shows that the interface gain and Pool-16 deficit remain available to a newly trained action expert. We therefore make an intervention-level claim. Scene-matched target content and fine spatial gradient delivery affect closed-loop robustness, and the resulting difference is retained in the visual backbone. This conclusion rests on interventions and closed-loop success, not on forecast quality or a single probe. Feature-level mediation remains open.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Revision overview and main findings

We are grateful to all reviewers for their careful reading and constructive feedback. Their questions helped us disentangle several effects that the original submission treated together and refine the scope of our claims.

To address these questions, we added targeted controls at both the model and policy scales. Policy-scale Pool-16, the π0.5 counterpart of the controlled gradient-pooling condition in Table C.5, isolates fine spatial gradient delivery by pooling only forecast gradients within 4×4 regions while leaving the forward pass unchanged. For the matched-hardware reruns, two Pool-16 seeds first trained on a different GPU type from the vision-token reference were retrained on the same GPU type. Frozen-backbone retraining tests whether the gain remains in the representation, and a shared frozen target tests dependence on target co-evolution. We also added sparse-carrier results, physical-grid and task-stage diagnostics, and a shift-wise analysis of the submitted shuffled-future arm. These changes appear in Section 6 and Appendices B, D, E, and G.

The revised claim is narrower. Stable addressing recovers much of the special-token deficit, while scene-matched content and fine spatial gradient delivery affect position shift and PRO (mean LIBERO-PRO success). Object shift follows a different pattern. Our contribution is a representation-learning analysis of how the target, spatial address, and gradient route shape future-prediction supervision in VLAs, with object robustness and broader cross-platform generality defining the current boundaries.

Dear reviewers,

Thank you for your reviews, and thank you to the authors for their responses. Please read the authors’ rebuttal and discuss whether it addresses your main concerns. Please update your review and rating if warranted, and briefly explain your current assessment, including any concerns that remain unresolved. If your assessment is unchanged, please explain why.

Thank you for your continued engagement.

Your AC

AuthorsOpenReview ↗

Summary of the Discussion

We thank the reviewers and the Area Chair for the discussion. Three main conclusions emerged from the questions and added controls.

Questions from Reviewer NRSG (Q1/Q3) and Reviewer Z3FR (Q2) focused on whether the interface gap could be explained by target evolution, positional indexing, or head capacity rather than supervision placement. All π0.5\pi_{0.5} forecast heads are matched at 8.4M parameters. In the controlled study, replacing the EMA teacher with a frozen forecast-trained teacher leaves the unindexed special-token representation narrow (effective rank 3.9 versus 32.3 for vision tokens). The gap therefore persists without EMA teacher updates. The target representation space still matters.

Vision-token features trained against the frozen forecast teacher retain action transfer close to the momentum target (0.57 versus 0.61), whereas frozen SigLIP produces higher-rank features with weaker action transfer (0.46). With frozen SigLIP, a stable spatial address substantially narrows the controlled interface gap. At policy scale, anchoring the special-token indices raises mean PRO from 60.2 to 63.5, closing roughly half the gap to vision tokens at 66.3. A shared frozen target then leaves mean PRO nearly unchanged for both vision tokens (66.3 to 65.5) and the anchored M=NM=N interface (63.5 to 63.2). Sparse addressed carriers further show that dense patch coverage is not always necessary. Together, these results show that the effect of supervision placement depends on both the target feature space and the spatial address.

Reviewers NRSG (Q2), Z3FR (W3), and x7LS (Q1) also asked whether the representation difference matters for policy robustness. Shuffling the scene-future pairing removes the position-shift and mean PRO gains. Pool-16 changes only the backward forecast gradient and reduces the same gains without changing the forward computation. Training a fresh action expert on frozen backbones preserves the interface gain and the Pool-16 deficit. These results connect the training interventions to closed-loop behavior and show that the difference remains in the visual backbone. The probes help describe that difference, but neither position nor displacement decoding alone explains it.

Questions from Reviewers Z3FR (Q1), x7LS (Q2), and uEFH clarified both the coverage and scope of the physical evidence. We reported all five physical policies, with one training run per variant. The additional diagnostics place the unseen-pot difference at the unfamiliar lid interaction rather than showing a general layout-only advantage. SmolVLA extends the finding to a second backbone, providing cross-model rather than broader benchmark evidence.

Overall, we see the main contribution as a controlled account of future prediction as a representation-learning interface. Forecast quality alone is not enough to understand its effect on a VLA. The target feature space, spatial address, carrier coverage, and gradient route jointly shape what enters the policy representation and what transfers to closed-loop control.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.