acceptodds
Withdrawn from ICLR 2027

PRIVILEGED FORESIGHT DISTILLATION: AMORTIZED FUTURE CORRECTION FOR WORLD ACTION MODELS

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

World action models jointly predict future video and action during training, raising an open question about what role the future-prediction branch actually plays. A recent finding shows that this branch can be removed at inference with little to no loss on common manipulation benchmarks, suggesting that future information may act merely as a regularizer on the shared visual backbone. We propose instead that joint training induces an action-conditioned correction that privileged future observations impose on action denoising, and that current-only policies capture this correction only partially. Making the account precise, we formulate privileged foresight as a residual in the action-denoising direction—the difference between what a model predicts given the true future and what it predicts given only the current frame—and introduce Privileged Foresight Distillation (PFD), which transfers this residual from a training-time teacher into a small adapter on a current-only student. The teacher and student share the same backbone and differ only in the attention mask over video tokens; future video is never generated at inference. Controlled experiments support that this gain reflects a future-conditioned correction rather than added capacity: with the total teacher weight held fixed, routing it directly rather than as a residual drops the policy below the undistilled baseline. Empirically, PFD improves over Fast-WAM on LIBERO and RoboTwin manipulation benchmarks while preserving the current- only inference interface with only a slight adapter-induced latency overhead. This view reframes the role of future information in world action models: not as a target to predict, nor as a branch to discard, but as a compressible correction to be distilled—one that the privileged path carries even when it is the worse predictor.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world action model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fMjEOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper addresses a new viewpoint to understand the role of video generation in action prediction. Based on this, it develops a framework called privileged foresight distillation (PFD) that trains an adapter to predict the residuals between the student and future-conditioned teacher to improve the precision of action prediction.

Critical strengths
  1. The paper addresses a new viewpoint to understand the role of video generation in action prediction.

  2. The PFD surpasses Fast-WAM and other methods on benchmarks

Critical weaknesses
  1. The evidence in Appendix B that the cosine similarity is negative does not support the paper's central "foresight correction" interpretation.

  2. There is an inconsistency in the reported RoboTwin score improvements. Table 2 reports Fast-WAM = 91.88 / 91.78 / 91.83 for Clean / Randomized / Average, whereas PFD reports 94.23 / 93.59 / 93.91. These numbers correspond to improvements of approximately +2.35, +1.81, and +2.08 percentage points. However, Section 4.2 states that the improvements are +1.23, +0.91, and +1.07, respectively.

Questions for authors

1.How do you explain the negative cosine similarity between residuals and the ground-truth correction direction, which does not support the central interpretation of the paper?

2.Can the authors explain the RoboTwin numerical inconsistency?

Reviewer_hDeiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper discusses the role that future-prediction branch plays in predicting actions. Discussing about prior work which showed that removing future-video prediction branch at inference led to little performance degradation, this paper raises the question of whether future observations provide meaningful action-specific information or merely act as a regularizer during training. It explores how future video observations can be used to improve action prediction in World Action Models (WAMs), without requiring expensive future-video generation at inference time.

For this, the authors propose Privileged Foresight Distillation (PFD) which is a teacher–student framework for transferring future-conditioned action information into the current policy. The teacher and student share the same backbone and differ only in their attention masks: the student attends to the current video frame, while the privileged teacher additionally attends to ground-truth future frames during training. The paper then uses a residual which is the difference between the teacher's and student's action-denoising velocity predictions. A lightweight residual adapter is trained to predict this correction using only the student's velocity prediction and the denoising timestep.

The training loss function combines matching video flow-matching targets to preserve video training, loss with ground truth to predict correct actions, a residual loss to learn the future-conditioned residual, and a teacher loss. At inference, the teacher is discarded, and the student uses the learned adapter to correct its action-velocity predictions without generating future videos. Experiments are conducted on the LIBERO and RoboTwin 2.0 robotic manipulation benchmarks and it demonstrates improvements over Fast-WAM.

Overall, the paper's main contribution is discussing that the future observations can serve as a useful training-time correction signal rather than requiring explicit future prediction at inference.

Critical strengths
  1. The proposed framework is relatively simple and it does not require significant architecture changes. The adapter for predicting the residual is lightweight with 1.2M-parameter and the teacher-student share the same backbone, differing only in their attention masks. This has practical value for robot manipulation tasks where computational efficiency is important.

  2. The paper explores an interesting question of how much the future-video prediction is contributing towards action prediction and in what simpler ways can it be used to improve action prediction, without much performance degradation. The paper provides a simple approach off computing the residuals to avoid the expensive future video prediction.

Critical weaknesses

It is somewhat unclear whether adapter is learning from the residual or it is solely learning from the ground-truth supervision. In Appendix, Table 5, the cosine similarity with of the learned adapter with the residual (difference between the teacher and student predictions) is -0.2 whereas the cosine similarity with the ground truth action is +0.3. The learned adapter is much better aligned with the ground truth than the raw residual but it raises a question whether the adapter is improving because it learned some useful information from future observations, or because it was directly trained to correct its predictions using the ground-truth actions. The text in Appendix does discuss "why the raw residual can anti-align while the learned correction aligns" but it is not very clear from the text and intuitive to understand what is helping the adapter to learn. This concern somewhat weakens the claim of the paper that future observations contain useful information when the learned adapter also has access to ground-truth action supervision during training.

Questions for authors

Is there a potential distribution mismatch between the teacher and student distribution? Since the teacher and student attend to different tokens and have different masks, can there be attention-distribution mismatch? The privileged teacher is constructed by modifying the attention mask of the shared backbone to allow action tokens to attend to future video tokens. However, the teacher backbone is not explicitly trained to predict the actions using this additional future information.

Reviewer_5uobOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper challenges the world action model setup in which an action model observes the imagined future visual modality. The setup in this work re-interprets the input frame processing in two branches: one that observes the present, and one that receives privileged information about future frames. This is achieved through different attention masks in the same vision backbone. The action is composed of a prediction over the present branch, and a residual from an anticipative module, with the inference time final prediction being the sum of the two.

Critical strengths

The research question is important, well motivated, and starts to gain traction by the research community. The paper is well written, interesting, and relatively easy to follow.

Critical weaknesses

My main concern about this paper is the experimental setup on LIBERO. It is well known that LIBERO has critical drawbacks [1,2], therefore the results presented on LIBERO might skew otherwise important insights. I encourage the authors to revisit the experimental setup, and possibly include validation/ablation experiments using a different dataset (e.g. RoboTwin2.0 or CALVIN). I would highlight that LIBERO can induce stochastic initial setups that, combined with the flow matching starting from random noise, might result in significant result variance. Therefore, I am not convinced by the Table 2 results.

Some results in the Appendix suggest that the teacher might lead to result degradation. Since the teacher branch does not receive a gradient update, the model does not adapt to the teacher mask pattern in training. This is important to clarify, since most conclusions rest on this potential issue.

[1] Fei, Senyu, et al. "Libero-plus: A progressive robustness benchmark for visual-language-action models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026. [2] Zhou, Xueyang, et al. "Libero-pro: Towards robust and fair evaluation of vision-language-action models beyond memorization." arXiv preprint arXiv:2510.03827 (2025).

Questions for authors

Do you have validation results on a different dataset? Can you clarify the potential masking issue I highlighted?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.