acceptodds
Under review as a conference paper at ICLR 2027

ForeSense: Learning Latent Visual Intuition for Vision-Language-Action Models via Future Experience Distillation

Abstract

The success of reasoning in large language models has inspired growing efforts to equip vision-language-action (VLA) models with similar reasoning capabilities. Existing approaches, however, typically externalize such reasoning either through verbal chains of thought or through explicit predictive representations of future evolution. We argue that reasoning for action generation should take the form of an abstract intuition about future dynamics, without being constrained to explicit representational formats or predefined semantic or temporal roles for intermediate reasoning states. To this end, we introduce ForeSense, a VLA framework that learns to sense how a scene may evolve by distilling privileged future experience into latent visual intuition. ForeSense is built around three key designs. 1) A Future-Conditioned Attention Bottleneck channels information from future observations through an autoregressive latent chain, which is then distilled into a current-only reasoner to form latent visual intuition without predefined semantic or temporal roles. 2) To support progressive refinement within this latent reasoning process, Detached Latent Curriculum (DeLaC) gradually extends the chain while applying stop-gradient to the existing latent prefix, encouraging each new step to refine previously learned reasoning states while mitigating latent-chain collapse. 3) To translate latent visual intuition into actions, Reasoning-to-Action Fusion (ReAct-Fuse) integrates the frozen reasoner’s latent key-value memory into each action-expert layer alongside the native VLA attention pathway, enabling fusion across heterogeneous reasoner and VLA backbone architectures. Extensive experiments on simulated and real environments demonstrate the strong and consistent performance of ForeSense, with success rates of 97.3% on LIBERO, 85.9% on LIBERO-Plus, and 79.2% on real-world tasks. Notably, the largest robustness gain appears under language perturbations, where success rises by 4.9 points from 86.3% to 91.2% over the π0.5-LIBERO base policy.

Primary area:
applications to robotics, autonomy, planning
Keywords:
VLA, Latent Reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Jo2pOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • This paper introduces a method called ForeSense, which attempts to equip vision-language-action models with better capabilities for reasoning about future dynamics. They do so by distilling a future-conditioned teacher's latent reasoning chain into a current-only reasoner to learn a form of "latent visual intuition".
Critical strengths
  1. The paper is written in an acceptably clear manner, and their motivation is good. They provided good reasoning for latent future intuitive reasoning: for simple/automatic tasks, VLAs should not be producing text chains of thought. The proposed concept and idea of latent visual intuition seems useful.
Critical weaknesses
  1. For the real-world evaluations, it would be useful to compare this method against baselines that use other forms of reasoning, e.g., those that use latent reasoning or action-centric traces (MolmoAct) or 3D point tracks. I am not saying that the proposed method must outperform those alternatives, but it would be good for scientific understanding to know how this method compares to others. This is especially important because the simulation results on LIBERO are somewhat saturated and the proposed method's performance is pretty close to alternatives there.
  2. In my view, an important benefit of reasoning in text or action-centric traces / 3D point tracks is interpretability. This may be useful in future when it is important to understand how embodied robots are reasoning and acting in the world. It would be good to enable this latent visual intuition to operate in a kind of interpretable manner.
Questions for authors

None.

4 comments

Thank you for your thoughtful review and encouraging comments on the motivation for latent visual intuition. We especially appreciate your attention to interpretability and are glad to share an initial exploration of this question below.

W1: Real-world comparison with other reasoning-based policies

Table H. Real-world task success rates (%).

TaskMolmoActVLA-JEPAForeSense
T1 Paper disposal759090
T2 Cup filling507575
T3 Semantic sorting205565
T4 Bucket stacking55585
T5 Towel-wrapped transfer156085
T6 Cylinder–arch assembly305575
Average32.565.079.2

MolmoAct and VLA-JEPA were adapted using the same real-world demonstrations as ForeSense.

Thank you for suggesting a comparison with policies that use different forms of reasoning. We evaluated MolmoAct and VLA-JEPA on the same six real-world tasks as ForeSense. Among the two baselines, VLA-JEPA is closest to ForeSense overall and matches it on paper disposal and cup filling. ForeSense’s largest gains over VLA-JEPA occur on bucket stacking, towel-wrapped transfer, and cylinder–arch assembly. The task-level comparison therefore shows where the two policies perform similarly and where their performance diverges on the robot.

W2: Interpreting the latent predictions

Thank you for highlighting the interpretability benefits of explicit reasoning. The ForeSense reasoner can continue generation after producing its latent chain to describe the expected future interaction in natural language. During routine action inference, the reasoner stops at the end of the latent chain, and the policy uses the chain’s K/V memory to generate actions. For analysis, we let the reasoner continue generating so that we can inspect what it expects to happen next.

Table F shows these forecasts alongside reference descriptions and observed policy actions. In F2, the forecast correctly describes the gripper approaching and pausing above the gray bowl, and the policy completes the task. In F3, the reasoner predicts an approach toward the gray bowl, and the policy approaches that same bowl even though the intended target is the plate. In F1, the forecast captures the bowl’s leftward movement but misses the lifting and lowering, and the rollout fails after imprecise placement. Continuing generation lets us inspect which interactions the reasoner anticipates, which details it misses, and whether its predicted target matches the object the policy actually approaches.

Table F. Future interaction predictions and observed policy behavior.

IDDecoded predictionReference descriptionAgreementObserved actionTask Success
F1A robotic arm moves over a black platform on a wooden table, rotating its gripper and shifting slightly to the left while holding a small metal bowl-like object. A red-rimmed plate, a small metal cup, and a colorful rectangular packet remain stationary on the table.A robotic arm grips a small metal bowl on a black platform, lifts it, and carries it leftward over the tabletop. It lowers the bowl near a small round container, but the clip ends before it releases it. A second bowl, a circular ring, and a small rectangular packet remain stationary on the table.Partial match. The prediction captures the leftward motion while holding the bowl but misses the lifting and the subsequent lowering near the container.Lifts the metal bowl and moves it leftward, but positions it imprecisely, causing the subsequent step to fail.✗
F2A robotic arm moves slowly downward and to the right over a kitchen countertop, bringing its gripper close to a small bowl behind a dark bottle. Nearby are a plate, a stovetop burner, a small rectangular device, and a black appliance. The arm does not visibly pick up or move anything during the clip.A robotic arm lowers and slightly rotates its gripper toward a small gray bowl on a wooden countertop, then pauses just above it. Nearby are a stovetop, a dark bottle, a red-rimmed plate, and a black appliance. No object is visibly picked up or moved.Match on key events. The prediction captures the downward approach toward the bowl and the absence of object pickup or movement.Lowers the gripper toward the gray bowl and pauses just above it.✓
F3A robotic arm moves above a ktchen countertop, lowering and rotating its gripper toward a small gray bowl behind a red-rimmed place. It pauses just above the bowl without visibly picking anything up. A stovetop, a small blue rectangular object, a knife block, and a black appliance remain stationary nearby.A robotic arm lowers its two-finger gripper toward a small plate with a pink patterned rim on a wooden tabletop. It makes slight sideways adjustments until the fingers are positioned over the plate’s edges, then pauses without visibly lifting it. Nearby are a small bowl, a blue rectangular object, a hot plate, and a black cabinet-like unit.Incorrect target object. The prediction captures the gripper lowering and pausing without a pickup, but identifies the gray bowl as the target. The reference shows the gripper aligning with the edges of the pink-rimmed plate.Approaches the gray bowl instead of the intended plate and continues the task with the wrong object, resulting in failure.✗

Agreement assesses how well the decoded prediction matches the reference clip. Task Success records the outcome of the full rollout (✓ for success and ✗ for failure).

Thanks. The authors addressed my concerns. I support the paper being accepted.

I do have comments below, though.

On (1) -- thanks for running the real-world evals. Their approach's results are strong. Real-world evals are always hard to do adequate controls, but nevertheless some attempt was made.

On (2) -- This is interesting. It seems like a glass half-full; sometimes it works, but other times it's challenging to interpret. I do think a greater attempt can be made -- methodology-wise -- at interpreting the latent visual intuition (but this is difficult, I understand).

For instance, the model produces the latent chain, followed by the natural language reasoning trace. But nothing actually guarantees (or even encourages) them to be aligned. That is, in the current interpretability setup, it might be that the two are completely independent, and the text trace tells us nothing about the latent chain.

AuthorsOpenReview ↗

Thank you for your follow-Up

Thank you for your thoughtful and positive feedback. We truly appreciate your support and constructive suggestions.

Reviewer_XeV9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ForeSense adds a latent “future-dynamics” reasoner to a VLA policy. The reasoner is a Qwen3-VL-4B model trained in three stages. First, it learns to predict text descriptions of future dynamics (GPT-6 Astra captions plus gripper-point tracks) from current and future frames. Second, it is forced to route that prediction through an 8-step autoregressive latent chain, trained with a curriculum that detaches earlier latents (DeLaC). Third, the chain is distilled into a student that sees only the current frame. Finally, the frozen reasoner’s latent key-value memory is injected into each action-expert layer through an added cross-attention pathway (ReAct-Fuse). Compared with the π0.5-LIBERO checkpoint, ForeSense reports 96.9→97.3% on LIBERO, 84.3→85.9% on LIBERO-Plus (+4.9 on language perturbations), and 60.0→79.2% on six real-world tasks.

Critical strengths

Real-world improvement: A +19-point average gain over π0.5 across six bimanual tasks, including deformable and assembly tasks, is the most practically significant result, if it holds up.

Well-motivated components with mechanistic evidence: DeLaC comes with a clear collapse analysis (effective rank of about 34–37 vs 4.5–10.5 under direct training). ReAct-Fuse is a clean way to attach a different reasoning backbone to an existing VLA without changing its native pathway.

Thorough reporting: The appendix documents data, schedules, annotation prompts, evaluation seeds, and latency in detail, which makes the work reproducible.

Critical weaknesses

Simulation gains are within noise: +0.4 on LIBERO and +1.6 on LIBERO-Plus come from single runs with unfixed diffusion noise. The ablations differ by only about 1 point (84.47–85.92), so neither the headline gains nor the component contributions are statistically established.

Baseline confound: ForeSense fully retrains π0.5 for 30k steps, while the LIBERO comparison uses the officially reported π0.5 number. The “KV w/o reasoning” control already recovers part of the gap (84.78), and it is missing from the real-world experiments, so added capacity from a 4B VLM may explain much of the improvement.

The “future intuition” claim is weakly supported: Token accuracy (99.95%) is measured teacher-forced on the training set, which reflects fitting, not prediction. Nothing shows the student’s latents actually encode future dynamics on held-out data, and the large language-perturbation gain may simply come from Qwen’s stronger language encoder.

Compute-mismatched ablations: DeLaC trains 8 curriculum stages × 430 steps, but it is unclear whether direct 8-token training received matched compute. The reasoner also adds about 73 ms of latency, and π0.5’s own latency is not reported for comparison.

Overclaimed conclusions: Claims of “substantial gains” and of learning from internet video without action labels are not supported by marginal simulation gains and no video-pretraining experiment. The future-dynamics targets also come from an unvalidated one-line captioning prompt.

Questions for authors

Can you report mean ± std over several seeds for π0.5 (retrained under your Stage-4 schedule), KV w/o reasoning, and ForeSense on LIBERO and LIBERO-Plus?

Can you add the KV w/o reasoning control to the real-world experiments, so that the +19-point gain can be attributed to future-dynamics reasoning rather than extra model capacity?

How well does the current-only student predict future dynamics on held-out episodes (e.g., trajectory error of the decoded gripper points)? Did direct 8-token training receive the same total optimization steps as the full DeLaC curriculum?

5 comments

Thank you for reading our paper so carefully and for your constructive suggestions. We especially appreciate your recommendations for additional ablations and controls, which helped us strengthen the evaluation and clarify what drives the observed gains.

W1 & Q1: Results on several seeds

Table D. Multi-seed evaluation on LIBERO and LIBERO-Plus under matched Stage 4 policy training.

IDVariantLIBERO Overall SR (%)LIBERO-Plus Overall SR (%)LIBERO-Plus Language SR (%)
D1π0.5 (fine-tuned for 30k steps)96.87 ± 0.0684.37 ± 0.0486.17 ± 0.06
D2π0.5 + pretrained Qwen K/V96.93 ± 0.0684.75 ± 0.0488.77 ± 0.06
D3ForeSense (π0.5 + learned latent K/V)97.33 ± 0.0685.99 ± 0.0991.23 ± 0.06

We evaluated three variants on LIBERO and LIBERO-Plus under matched Stage 4 policy training, including a π0.5 baseline fine-tuned for 30,000 steps. Table D reports the mean ± sample standard deviation across three evaluation seeds for each benchmark.

The overall improvement persists on both benchmarks across all three evaluation seeds, including against π0.5 fine-tuned under the same Stage 4 schedule. Compared with fine-tuned π0.5 (D1) and the pretrained Qwen K/V control (D2), ForeSense improves mean overall success by 0.46 and 0.40 percentage points on LIBERO and by 1.62 and 1.24 percentage points on LIBERO-Plus, respectively. On the LIBERO-Plus Language subset, the corresponding gains are 5.06 and 2.46 percentage points.

W2 & Q2: Real-world comparison with pretrained Qwen K/V

Table E. Real-world comparison with pretrained Qwen K/V and ablation of future-to-current distillation.

IDVariantPaper disposalCup fillingSemantic sortingBucket stackingTowel-wrapped transferCylinder–arch assemblyOverall SR (%)
E1π0.517/2010/2011/2015/2010/209/2060.0
E2π0.5 + frozen pretrained Qwen K/V17/2010/2013/2015/2013/2011/2065.8
E3ForeSense w/o future-to-current distillation17/2010/2013/2015/2015/2012/2068.3
E4ForeSense (full model)18/2015/2013/2017/2017/2015/2079.2

To test whether the real-world gains can be explained by the additional 4B Qwen backbone, we compare full ForeSense (E4) with a frozen pretrained Qwen K/V control (E2) on the same six tasks, with 20 trials per task. Both variants use the same Qwen backbone architecture, K/V memory dimensions, and ReAct-Fuse interface. Their action policies are trained on the same demonstrations using the same Stage 4 schedule. Adding pretrained Qwen K/V improves success from 72/120 to 79/120. With the same backbone architecture and fusion interface, ForeSense succeeds in 95/120 trials, or 16 more than E2 (+13.3 percentage points). The gain over E2 therefore persists with the added Qwen capacity held fixed.

To isolate the effect of latent distillation, we initialize the Stage 3 students in E3 and E4 from the same Stage 2 checkpoint and train them for the same number of steps using the same current observations, task instructions, and future-dynamics prediction targets. Both students use the same next-token prediction loss. The only difference is that E4 also matches its latent chain to that of the future-conditioned teacher. Adding this loss increases real-world task success from 82/120 to 95/120 (+10.8 percentage points), with gains on five of the six tasks and a tie on semantic sorting.

W3 & Q3: Held-out future-dynamics prediction

Table C(b). Held-out future prediction from current observations.

MethodDescription Token Accuracy (%)Gripper Trajectory MSE (px²)
Current-only student without latent distillation82.85121.1
ForeSense current-only student with latent distillation90.1389.9

Latent distillation improves the student’s prediction of future gripper motion from the current observation and task instruction. On the episode-disjoint held-out split, it reduces the MSE of freely decoded trajectories against the reference trajectories by 25.8% (121.1 to 89.9 px²) and raises description-token accuracy from 82.85% to 90.13% (Table C(b)). Both Stage 3 students use the same Stage 2 checkpoint for initialization, prediction targets, and training schedule. The latent distillation loss is the only training difference.

The language-perturbation gain also persists with the Qwen backbone architecture held fixed. ForeSense reaches 91.23% mean success on the LIBERO-Plus Language subset, compared with 88.77% for the pretrained Qwen K/V control (+2.46 percentage points, Table D). Both policies use the same ReAct-Fuse interface and Stage 4 training schedule.

W4 & Q3: Comparable Training cost in DeLaC

Table B. Effects of the latent-length curriculum and detached-prefix training.

IDStage 2 variantChain-length scheduleDetach existing prefix during expansionOptimizer stepsPrefix drift (1 − cosine)Inter-position similarityHeld-out Stage2 Token Acc. (%)LIBERO-Plus Overall SR (%)
B1Direct K=8Fixed at K=8N/A3,440N/A0.98278.3584.47
B3Curriculum onlyK=1 → 2 → … → 8No3,4400.880.96880.4185.07
B4Full DeLaCK=1 → 2 → … → 8Yes3,4400.270.68390.2085.92

For each expansion from K−1K-1 to KK, we use the same held-out Stage 2 inputs to compare the projected latent embeddings fed back into the backbone before expansion and after 430 training steps at length KK. We compute cosine distance (1−cos⁡1-\cos) between the two embeddings for each example and each existing position i<Ki<K, average across examples and positions within each expansion, and then average equally across the seven expansions. B3 and B4 use the same held-out inputs and measurement protocol.

The direct 8-latent baseline (B1) and full DeLaC (B4) both use 3,440 Stage 2 optimizer steps. B1 uses a chain length of 8 from the first step, while B4 increases the chain length from 1 to 8, with 430 steps at each length. The two variants are matched in optimizer steps, and per-step computation varies with chain length. At the same step budget, DeLaC improves held-out token accuracy from 78.35% to 90.20% and overall LIBERO-Plus task success from 84.47% to 85.92%.

For inference, standalone π0.5 and ForeSense take 61 ms and 119.13 ms per action chunk at P50 on a single H200, respectively. As shown in Figure 7(b) of the submitted manuscript (Appendix B.5, p. 17), the reasoner’s latent K/V and the native π0.5 prefix cache can be computed independently, allowing 29.08 ms of prefix computation to overlap with reasoner execution on two H200s. With all eight latents retained, parallel execution reduces ForeSense’s end-to-end latency to 91.60 ms, a 23.1% reduction relative to single-GPU sequential execution. We will explicitly report the standalone π0.5 latency in the revision.

W5: Explanation of conclusion

Thank you for pointing out that the placement of this discussion could make its scope unclear. The limitations paragraph in Section 5 of the submitted manuscript (p. 9) states:

Whether such pretraining improves the learned intuition and its transfer to downstream control has yet to be established.

The sentence beginning with “More broadly” was intended to draw out an implication for future research. The first three stages of ForeSense derive supervision from future observations in robot demonstrations without action labels, offering a starting point for investigating whether physical reasoners could be pretrained on large-scale internet video. We will move this discussion to the limitations and future-work paragraph so that its scope is clear. We will revise the conclusion to distinguish the modest gains in simulation from the larger gains observed across the six real-world tasks.

The one-line prompt in Appendix B.2 is used for the semantic annotation only. We selected GPT-6 Astra to describe the interactions observed in each clip and manually reviewed the generated descriptions. As described in Section 3.2 and Appendices B.2–B.3 of the submitted manuscript, the full supervision target combines these descriptions with gripper keypoint trajectories derived from the observed motion. The trajectories come from simulator projections for simulation data and keypoint tracking in recorded videos for real-world data. They provide geometric supervision alongside the captions. We will clarify the human review procedure in the revision.

Reviewer_HZ5EOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper distills future interaction experience into latent representations to guide VLA action generation, separating latent reasoning from the policy and fusing them via ReAct-Fuse . the authors did experiments on LIBERO and LIBERO-PLUS, and real world robots, analysis on each component is conducted

Critical strengths

A meaningful research direction. Learning predictive representations from future experience without requiring explicit reasoning traces at inference time is an interesting direction for improving VLA models.

A modular design. The separation between latent reasoning and action generation, together with the proposed ReAct-Fuse mechanism, provides a flexible way to integrate a reasoning model with an existing VLA policy.

Critical weaknesses
  1. The contribution of Stage 1 to Stage 2 is unclear.

Stage 2 appears to require substantial additional learning to establish the attention bottleneck and latent reasoning chain. The role and necessity of Stage 1 are therefore insufficiently justified. It is unclear how much Stage 1 contributes to the final representation, rather than merely providing an initialization for subsequent training.

  1. The motivation and mechanism of DeLaC are insufficiently established.

The proposed curriculum and detached-prefix strategy are intended to mitigate latent representation collapse, but the underlying mechanism remains unclear. In particular, stop-gradient only prevents gradients from propagating through the detached embeddings during a given backward pass; it does not freeze the shared backbone or guarantee that previously generated latent representations remain stable across optimization steps. The paper does not sufficiently explain why this intervention should prevent collapse, nor does it isolate the contributions of curriculum learning and detached-prefix training.

  1. The source of the performance gains remains unclear.

Although the results are encouraging, the improvements over the baseline are modest on some benchmarks. It remains unclear how much of the gain comes specifically from the learned latent reasoning representations, rather than the additional parameters, model capacity, or computational overhead introduced by the reasoning-to-action fusion module. More controlled experiments would help establish the contribution of the proposed mechanism.

Questions for authors
  1. Can the authors provide ablations comparing Stage 2 initialized from Stage 1 with training Stage 2 directly from the pretrained backbone, or removing Stage 1 entirely? How much does Stage 1 contribute to downstream performance?

  2. What is the precise mechanism by which detached-prefix training mitigates latent collapse when the shared backbone remains trainable? Can the authors provide separate ablations for curriculum learning and stop-gradient, along with analyses of latent representation stability across training steps and representation diversity across positions?

  3. Can the authors better isolate the contribution of learned reasoning from the additional capacity and computation introduced by ReAct-Fuse? For example, could they compare against parameter- or compute-matched controls and provide stronger evidence that the learned latent memory captures useful predictive information beyond the original VLA representations?

6 comments

Thank you for your thoughtful and constructive review. We especially appreciate your question about how stop-gradient works when the backbone remains trainable. It gave us an opportunity to clarify the motivation and mechanism behind DeLaC.

W1 & Q1: Contribution of Stage 1

Table A. Effect of Stage 1 initialization on Stage 2 token accuracy and downstream performance.

IDVariantStage 1 stepsStage 2 stepsTotal (S1 + S2)Held-out Stage2 Token Accuracy (%)LIBERO-Plus Overall SR (%)
A1ForeSense4303,4403,87090.2085.92
A2Skip Stage 103,4403,44074.3983.07
A3Skip Stage 1 (matched total steps)03,8703,87073.5283.22

Stage 1 adapts the reasoner to future interaction prediction through explicit supervision. Stage 2 then learns to encode the information needed for prediction in a latent chain. To quantify the contribution of Stage 1, we added two controls that train Stage 2 directly from the pretrained Qwen backbone. A2 matches the number of Stage 2 steps, while A3 reallocates all 430 Stage 1 steps to Stage 2 to match the total number of optimizer steps.

As shown in Table A, with Stage 2 steps matched, including Stage 1 improves held-out token accuracy by 15.81 percentage points and overall LIBERO-Plus task success by 2.85 points. With total steps matched, the gains are 16.68 and 2.70 points, respectively. These results show that, under the same step budget, training the reasoner on explicit future interaction prediction before introducing the latent chain improves both Stage 2 prediction accuracy and downstream task performance.

W2 & Q2: Motivation and mechanism of DeLaC

Table B. Effects of the latent-length curriculum and detached-prefix training.

IDStage 2 variantChain-length scheduleDetach existing prefix during expansionOptimizer stepsPrefix drift (1 − cosine)Inter-position similarityHeld-out Stage2 Token Acc. (%)LIBERO-Plus Overall SR (%)
B1Direct K=8Fixed at K=8N/A3,440N/A0.98278.3584.47
B3Curriculum onlyK=1 → 2 → … → 8No3,4400.880.96880.4185.07
B4Full DeLaCK=1 → 2 → … → 8Yes3,4400.270.68390.2085.92

For each expansion from K−1K-1 to KK, we use the same held-out Stage 2 inputs to compare the projected latent embeddings fed back into the backbone before expansion and after 430 training steps at length KK. We compute cosine distance (1−cos⁡1-\cos) between the two embeddings for each example and each existing position i<Ki < K, average across examples and positions within each expansion, and then average equally across the seven expansions. B3 and B4 use the same held-out inputs and measurement protocol.

We agree that the shared backbone continues to update after prefix detachment. DeLaC preserves these updates so that the model can adapt as the chain grows from K−1K-1 to KK. Let θ\theta denote the shared backbone parameters, hi−1h_{i-1} the hidden state before projection, and zi=Bϕ(hi−1)z_i=B_\phi(h_{i-1}) the projected latent embedding. We write z~i\tilde z_i for the embedding fed back into the backbone. To isolate the autoregressive latent path, we write L=L(zK,θ)\mathcal{L}=\mathcal{L}(z_K,\theta) and zK=zK(z~K−1,θ)z_K=z_K(\tilde z_{K-1},\theta), leaving the K/V-cache paths implicit. Without detachment, z~i=zi\tilde z_i=z_i, and the chain rule gives

gfull=∂L∂θ∣zK+∂L∂zK(∂zK∂θ∣z~K−1+∂zK∂z~K−1dz~K−1dθ)g_{\mathrm{full}} = \left.\frac{\partial\mathcal{L}}{\partial\theta}\right|_{z_K} + \frac{\partial\mathcal{L}}{\partial z_K} \left( \left.\frac{\partial z_K}{\partial\theta}\right|_{\tilde z_{K-1}} + \frac{\partial z_K}{\partial\tilde z_{K-1}} \frac{d\tilde z_{K-1}}{d\theta} \right)

The last term in parentheses carries gradients back through z~K−1=zK−1\tilde z_{K-1}=z_{K-1}. Expanding it recursively exposes the contributions through zK−1z_{K-1}, zK−2z_{K-2}, and earlier latents along this path. With detachment, z~i=sg⁡(zi)\tilde z_i=\operatorname{sg}(z_i) for i<Ki < K. Stop-gradient preserves the forward values and sets dz~K−1/dθ=0d\tilde z_{K-1}/d\theta=0 in backpropagation, giving

gdetach=∂L∂θ∣zK+∂L∂zK∂zK∂θ∣z~K−1g_{\mathrm{detach}} = \left.\frac{\partial\mathcal{L}}{\partial\theta}\right|_{z_K} + \frac{\partial\mathcal{L}}{\partial z_K} \left.\frac{\partial z_K}{\partial\theta}\right|_{\tilde z_{K-1}}

The final input z~K=zK\tilde z_K=z_K remains differentiable. Even with z~K−1\tilde z_{K-1} held fixed during backpropagation, generating zKz_K still uses the shared backbone, so the future-dynamics next-token loss can update θ\theta through zKz_K. These updates can change the earlier latents on subsequent forward passes. We therefore measure prefix drift before and after each expansion to test whether detachment improves stability in practice.

This intervention applies to the prefix learned at the preceding curriculum stage. Direct (K=8) trains all eight positions jointly from the first step, so there is no previously learned shorter prefix to detach. In Table B, B1 → B3 compares fixed-length training with the latent-length curriculum under the same optimizer-step budget, and B3 → B4 isolates the effect of prefix detachment under the same curriculum.

The curriculum alone improves held-out Stage 2 token accuracy and downstream LIBERO-Plus task success by 2.06 and 0.60 percentage points, respectively, while inter-position similarity remains high (0.968). Adding prefix detachment reduces mean prefix drift across the seven expansions from 0.88 to 0.27 and the final mean absolute cosine similarity between the eight latent positions from 0.968 to 0.683. Token accuracy and task success improve by a further 9.79 and 0.85 percentage points, respectively. With the shared backbone still trainable, these results show that prefix detachment reduces drift during expansion and yields more distinct representations across latent positions.

W3 & Q3: Source of the performance gains

Table C(a). LIBERO-Plus performance with different K/V sources

IDVariantOverall SR (%)Language SR (%)
C1π0.5 + frozen pretrained PaliGemma K/V84.4186.3
C2π0.5 + frozen pretrained Qwen K/V84.7888.8
C3ForeSense (π0.5 + learned latent K/V)85.9291.2

To test whether the gains come from adding a pretrained VLM, we compare three K/V sources for π0.5 in Table C(a). C1 provides a reference using frozen pretrained PaliGemma K/V. Our main comparison is between C2 and C3, which share the same Qwen backbone architecture, K/V memory length and dimensions, and ReAct-Fuse interface. C2 uses frozen pretrained Qwen K/V, while C3 uses K/V from ForeSense’s learned latent chain generated from the current observation and task instruction.

Compared with C2, C3 improves LIBERO-Plus overall success from 84.78% to 85.92% (+1.14 percentage points) and Language success from 88.8% to 91.2% (+2.4 percentage points). These gains show that ForeSense’s learned latent K/V improves policy performance beyond what we obtain by adding pretrained Qwen K/V through the same fusion interface.

Table C(b). Held-out future prediction from current observations.

MethodDescription Token Accuracy (%)Gripper Trajectory MSE (px²)
Current-only student without latent distillation82.85121.1
ForeSense current-only student with latent distillation90.1389.9

We also tested whether the latent chain supports future prediction. We evaluated both Stage 3 students on the same episode-disjoint held-out split, using only the current observation and task instruction. Both students start from the same Stage 2 checkpoint and use the same prediction targets and training schedule. As shown in Table C(b), adding latent distillation improves description token accuracy by 7.28 percentage points and reduces the MSE of freely decoded gripper trajectories by 25.8%.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.