acceptodds
Under review as a conference paper at ICLR 2027

iWAM: Efficient Action Prediction from World Models without Explicit Future Prediction

21 views

Abstract

World-action models (WAMs) equip a policy with a pretrained video generation model, on the premise that predicting how a scene should evolve helps decide how to act. While this simplifies action prediction, imagining future scenes is expensive for real-time control. This motivates efficient proxies for future imagination, but it remains unclear what such a proxy should preserve. Our key insight is that an action policy does not require a high-resolution future; it only needs enough information to specify a plausible future with respect to which an action should be produced. This leads to a simple design principle for *i*WAM: rather than denoising an entire future frame, we condition the action policy on intermediate activations from a single video-model evaluation at a latent noise vector that would otherwise initialize denoising. The key challenge is ensuring these activations correspond to the demonstrated future whose actions supervise the policy. We address this by training the video model to approximately represent an invertible flow map: integrating a demonstrated future in the clean-to-noise direction recovers a latent endpoint corresponding to that future, yielding activations aligned with the demonstrated actions. At inference, we sample this endpoint from the prior, and require only a single video-backbone forward pass for conditioning. Across simulated benchmarks and real robots, *i*WAM recovers the generalization benefits of future-conditioning at the inference cost of methods that discard futures entirely.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Action Models (WAMs), Robot Learning, Efficient Inference, Flow Maps

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UpPuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper demonstrates through experiments that FastWAM-style efficiency can come at the cost of weaker generalization. Motivated by this, it proposes iWAM, which uses a bidirectional flow map to extract future-specific intermediate representations with a single video-model evaluation, avoiding explicit future generation. Experiments show that iWAM achieves performance close to IDM with substantially lower inference cost.

Critical strengths
  1. The paper demonstrates that FastWAM-style efficiency can hurt generalization through experiments.

  2. The paper addresses iWAM, which offers a good performance-latency trade-off.

  3. The paper includes real-robot experiments, which strengthens the empirical validation.

Critical weaknesses
  1. Train-test mismatch. During training, the action policy is conditioned on noise endpoints recovered from GT future observations, while at inference the endpoint is sampled directly from the prior. Since the clean-to-noise map is only approximate, it is unclear whether these two endpoint distributions match.

  2. The one-step distillation method could be better explored. Recent works[1,2,3,4] provide stronger one-step causal video distillation techniques, while direct one-step generation videos show bad performance. It would be useful to try these alternatives. And the represention of the laywise forward activations could be explored, which helps understand why the method is useful.

  3. The evaluation benchmarks show substantial saturation. On the selected DROID simulation tasks, strong methods already achieve over 90% average success, with many settings reaching 95–100%. Similarly, on the unperturbed LIBERO setting, most methods achieve around 97–99% success. These ceiling effects limit the ability of the current benchmarks to distinguish strong methods. Evaluating on harder tasks or more challenging benchmarks would strengthen the empirical conclusions.

  4. Presentation. Figure 1 could be improved; in particular, the connections across the video-model blocks are somewhat confusing. Besides, Table 1 could also explicitly state that it reports results on the DROID simulation benchmark.

[1] AAD-1: Asymmetric Adversarial Distillation for One-Step Autoregressive Video Generation

[2] One-Forcing: Towards Stable One-Step Autoregressive Video Generation

[3] Towards One-Step Causal Video Generation via Adversarial Self-Distillation

[4] Causal-rCM: A Unified Teacher-Forcing and Self-Forcing Open Recipe for Autoregressive Diffusion Distillation in Streaming Video Generation and Interactive World Models

Questions for authors

Do the authors plan to release the training and evaluation code, as well as model checkpoints, to facilitate reproducibility?

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors - I

Thank you for your feedback! To address your concerns, we (i) quantify the distribution mismatch between recovered noise endpoints and explain why it does not prevent strong test-time performance, (ii) clarify that unperturbed LIBERO serves to highlight saturation rather than evaluate generalization, while also highlighting our evaluations on real-world manipulation tasks where success requires recovering from failures and handling distractors.

We have also revised the manuscript to improve clarity and strengthen the analysis: we added Figure 2 to highlight the generalization gap under LIBERO-PRO perturbations, expanded Section 4 and Appendix C.4 with qualitative analysis of FastWAM's failure modes and representative film strip comparisons of iWAM and FastWAM rollouts, and improved Figure 1. We have highlighted all the edits in blue.

Re: Distribution Mismatch

Train-test mismatch. During training, the action policy is conditioned on noise endpoints recovered from GT future observations, while at inference the endpoint is sampled directly from the prior. Since the clean-to-noise map is only approximate, it is unclear whether these two endpoint distributions match.

Thanks for pointing out the potential issue regarding the distribution mismatch. First, we would like to highlight that iWAM doesn't require the inversion endpoint distribution used during training and the prior distribution to match or even be close, for strong test-time performance. Crucially, as long as the video model is trained well, it can generate consistent futures under both these distributions. This allows the action policy to learn from representations aligned with the demonstrated future during training and use representations associated with a specific plausible future at inference.

We quantify the discrepancy between recovered inversion endpoints and the standard Gaussian prior using squared maximum mean discrepancy (MMD²), using techniques from [5], and also show that the inverted endpoints can be distinguished from the prior with high accuracy.

Setup: We evaluate 57.6k distinct DROID episodes, randomly selecting an observation–future window per episode, and show results where ground-truth VAE latents are inverted using 1, 2, and 4 shortcut inversion steps. Each recovered future endpoint consists of four VAE latent frames, which are flattened into a d=76,800d = 76{,}800-dimensional vector. We use an equally weighted mixture of Gaussian RBF kernels with squared bandwidths {d/2,d,2d}\lbrace d/2, d, 2d \rbrace. For each inversion step count, we estimate MMD² by dividing the 57.6k recovered endpoints into 225 blocks of 256 samples and averaging the unbiased estimates across blocks.

MMD² results: We observe that MMD² decreases from 0.005022 to 0.004846 to 0.004767 as the number of inversion steps increases from 1 to 2 to 4. (For scale, the MMD² between N(0,0.92I)\mathcal{N}(0, 0.9^2 I) and N(0,I)\mathcal{N}(0, I) is approximately 0.003672 under the same kernel.) These values indicate a measurable distribution mismatch, slightly larger than that caused by a 10% reduction in the Gaussian prior's standard deviation, which decreases only modestly with additional inversion steps.

We additionally assess distinguishability using a simple threshold classifier based on the normalized latent radius r(z)=∥z∥2/dr(z) = \lVert z \rVert_2 / \sqrt{d}, which is approximately 1 for samples from the standard Gaussian prior. For each inversion setting, we select the threshold for the classifier using a mixture of 28.8k recovered endpoints and 28.8k independent Gaussian samples. We then evaluate the fixed classifier on a disjoint, balanced test set of 57.6k samples (28.8k held-out recovered endpoints and 28.8k Gaussian samples). Held-out accuracy is 99.85%, 99.18%, and 97.06% for 1-, 2-, and 4-step inversion, respectively, compared with 49.74% for an independent Gaussian-versus-Gaussian control.

Thus, additional inversion steps reduce both MMD² and radius-based distinguishability, although the recovered endpoints are easily distinguishable from the prior.

We have also added Figure 4, which demonstrates that the video model can generate futures effectively from both distributions while maintaining strong inversion consistency. (The reconstructed future is obtained using 2 shortcut inversion steps, and both futures are generated using 8 shortcut denoising steps.)

Takeaway: These results confirm a measurable mismatch between the demonstration-inverted endpoint distribution used during training and the Gaussian prior sampled at inference. Despite this mismatch, iWAM achieves strong test-time performance, supporting our claim that matching the training and inference endpoint distributions is not necessary for effective action prediction.

[5] Rustamov, Raif M. "Closed‐form expressions for maximum mean discrepancy with applications to Wasserstein auto‐encoders."

AuthorsOpenReview ↗

Rebuttal by Authors - II

Re: One-Step Distillation

The one-step distillation method could be better explored. Recent works[1,2,3,4] provide stronger one-step causal video distillation techniques, while direct one-step generation videos show bad performance. It would be useful to try these alternatives. And the represention of the laywise forward activations could be explored, which helps understand why the method is useful.

We thank the reviewer for highlighting recent advances in one-step video distillation and suggesting layer-wise representation analysis, and we will cite and discuss them in our paper. These works offer promising extensions to our bidirectional shortcut objective: (i) incorporating auxiliary GAN loss as in One-Forcing [2], potentially using the bidirectionally attentive video discriminator from AAD-1 [1]; (ii) complementing our existing shortcut bootstrapping loss with adversarial self-distillation across different denoising-step counts [3]; and (iii) exploring DMD-based refinement, potentially adopting the self-forcing strategy of Causal-rCM [4]. These techniques could improve generation quality while retaining our bidirectional consistency objective. However, our work primarily focuses on identifying the distinct requirements for efficient future condition and addressing them through a principled use of simple bidirectional flow maps. We therefore leave incorporating these advanced distillation techniques as promising directions for future work.

We also thank the reviewer for suggesting layer-wise activation analysis. Section 5.1 already provides empirical evidence that iWAM's forward activations capture the relevant mode of the future-video distribution. Specifically, a single forward evaluation of a standard flow model from randomly sampled noise yields high action-training loss. We now further explicitly mention in Section 5.1 that even conditioning on a fully generated, plausible future does not reduce this loss when the future is sampled independently of the demonstration.

In contrast, forward activations evaluated at the noise endpoint obtained by inverting the demonstrated future achieve action-training loss close to IDM, which directly conditions on that future (Figure 3). Together, these controlled experiments provide strong evidence that our activations encode the demonstration-aligned future mode relevant to the supervised actions during training, rather than merely representing an arbitrary plausible future. While layer-wise visualization could further reveal how this information is distributed across network layers and remains an interesting direction for future work, our experiments directly establish what information the activations preserve and why it matters.

Re: Potential Saturation of Eval Benchmarks

The evaluation benchmarks show substantial saturation. On the selected DROID simulation tasks, strong methods already achieve over 90% average success, with many settings reaching 95–100%. Similarly, on the unperturbed LIBERO setting, most methods achieve around 97–99% success. These ceiling effects limit the ability of the current benchmarks to distinguish strong methods. Evaluating on harder tasks or more challenging benchmarks would strengthen the empirical conclusions.

We would like to note that we do not rely on unperturbed LIBERO to evaluate generalization; rather, we use its strong in-distribution performance to highlight benchmark saturation (Lines 73–78, Lines 258–266). Although FastWAM achieves 96.9% success in-distribution, its performance degrades significantly under the out-of-distribution perturbations in LIBERO-PRO, where the gap to IDM becomes substantially larger. To make this takeaway clearer, we add Figure 2 to illustrate this. In particular, the relative performance gap increases from approximately 1% in-distribution to 16% across OOD perturbations under joint training. This widening gap is precisely the result we highlight and analyze in the paper to motivate our approach.

We also evaluate challenging physical manipulation tasks for DROID policies, including towel folding and pen transfer in cluttered environments. Successful execution often requires multiple attempts, allowing us to observe how policies recover from failed grasps and handle distractors. These evaluations reveal important differences: while IDM and iWAM can recover and complete the tasks, FastWAM often abandons the target after a failed grasp or manipulates unrelated objects instead of following the instruction. These failures contribute to its substantially lower real-world success rate (13.3%, compared with 80.0% for IDM and 85.0% for iWAM (MIP)).

To make these differences clearer, we have added qualitative rollout illustrations, including representative film strips comparing iWAM and FastWAM, along with a discussion of FastWAM's failure modes in Section 4 and Appendix C.4.

AuthorsOpenReview ↗

Rebuttal by Authors - III

Re: Presentation

Presentation. Figure 1 could be improved; in particular, the connections across the video-model blocks are somewhat confusing. Besides, Table 1 could also explicitly state that it reports results on the DROID simulation benchmark.

Thank you for your constructive suggestions. We have revised Figure 1 to improve its clarity and also updated the caption of the table accordingly.

Re: Code and Checkpoint Release

Do the authors plan to release the training and evaluation code, as well as model checkpoints, to facilitate reproducibility?

Yes, we plan to publicly release our training and evaluation code, along with model checkpoints, upon acceptance as part of the camera-ready version, to facilitate reproducibility.

AuthorsOpenReview ↗

Did our Rebuttal Address your Concerns?

Dear Reviewer,

Given that there is just over a day left in the discussion period, we wanted to reach out to see if our rebuttal above addressed all your concerns or if any others still remained? Please let us know if any further clarification would be helpful. If all your concerns are addressed, we would be really grateful if you could acknowledge that soon.

Thanks,

Authors

Reviewer_NVjkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes iWAM, which conditions the action policy on intermediate activations obtained from a single forward pass of the video model at a latent noise vector, rather than denoising an entire future frame.

Critical strengths

The paper addresses an important and practically relevant problem: reducing the inference cost of world-action models for real-time robot control.

Critical weaknesses
  1. The introduction is overly verbose and difficult to follow.
  2. The method feels more like a careful engineering combination of existing components rather than a novel algorithmic insight. The core technique of extracting intermediate activations from a pretrained video model is not fundamentally new — it builds directly on existing ideas from FastWAM, UVA, and IDM.
  3. The paper uses dense notation that is introduced without sufficient motivation (e.g., the bidirectional flow map parameterization, the shortcut bootstrap loss). Some equations are presented without clear intuition about what they achieve.
Questions for authors

please see weaknesses.

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors - I

Thank you for your feedback! To address your concerns, we (i) streamline the Introduction to clarify the motivation and central research question, and (ii) clarify our contributions beyond extracting intermediate activations, particularly identifying the fundamental limitations of existing efficient WAMs and establishing the requirements for efficient future conditioning.

We have also revised the manuscript to improve clarity and strengthen the presentation: we reorganized Section 5.1 into two subsections to establish the requirements for efficient future conditioning before explaining how bidirectional flow maps address them. We have also added Figure 4 to illustrate the importance of alignment with demonstration during training, introduced a notation table (Table 1), and also expanded Section 4 and Appendix C.4 with qualitative analysis of FastWAM's failure modes and representative film strip comparisons of iWAM and FastWAM rollouts. We have highlighted all the edits in blue.

Re: Verbose Intro

The introduction is overly verbose and difficult to follow.

We thank the reviewer for the feedback. We have revised the Introduction to reduce verbosity and improve its logical flow. In particular, we streamlined the discussion of existing WAM approaches and clarified the motivation for retaining information about a particular future. We also more directly motivated the central question of our work and simplified the explanation of how our bidirectional flow map addresses the efficiency–generalization trade-off. Please let us know if this version is better and if you have more thoughts on how we can improve it further; we are very happy to edit the paper in the remainder of the discussion period.

Re: Explaining Novelty (1/2)

The method feels more like a careful engineering combination of existing components rather than a novel algorithmic insight. The core technique of extracting intermediate activations from a pretrained video model is not fundamentally new — it builds directly on existing ideas from FastWAM, UVA, and IDM.

While existing WAMs, including FastWAM, UVA, and IDM, condition action policies on intermediate-layer activations of video models, the key distinction lies in the inputs used to compute these activations and what information they capture. Our analysis in Section 5.1 identifies fundamental limitations of WAMs designed for fast inference that compute these activations solely from current observations/history (e.g., FastWAM and UVA). These limitations constrain their generalization and performance on realistic datasets, as demonstrated through analysis and experimentation (Section 4).

First, Fast-WAM computes activations solely from current observations (i.e., with no actual future prediction). These activations do not correspond to any particular future, as similar observations can be associated with diverse futures. In Section 4, we empirically demonstrate that while not using activations from future imagination does not hurt in-distribution performance, Fast-WAM performs significantly worse than IDM and iWAM under out-of-distribution settings. We have also added a discussion of these qualitative rollouts and FastWAM's failure modes in Section 4 and have included representative film strips in Appendix C.4 that visually illustrate these behaviors and provide a frame-by-frame comparison of FastWAM and iWAM.

Furthermore, naively using activations from future imaginations is also insufficient. For example, using activations from a single forward step of a flow model starting from pure noise (e.g., iWAM (prior noise)) leads to significantly worse performance compared to IDM and iWAM (Section 5.1). The activations from this forward pass do not correspond to any particular imagined future, resulting in high action-training loss. Proposition A.1 provides additional theoretical intuition: for an optimally trained flow model, the one-step denoising estimate collapses to the conditional mean, independent of the sampled noise. Nevertheless, as illustrated in Figure 3, an additional problem is that the conditioning features and action targets will not be aligned during training, making the training objective harder to fit.

Addressing this limitation while maintaining fast inference requires more than combining existing methods, and engineering alone cannot resolve the underlying flaw. In Section 5.1, we systematically investigate what is needed to retain the benefits of future-conditioned action prediction without iterative generation, revealing two critical requirements: (1) future specificity, where a single model evaluation must expose information about a particular future, and (2) alignment between the represented future and the actions used for supervision during policy training.

AuthorsOpenReview ↗

Rebuttal by Authors - II

Re: Explaining Novelty (2/2)

Implicit WAM addresses both these requirements. We retain the advantage of a single forward pass while ensuring that the conditioning features correspond to a particular imagined future and the noise latent is aligned with the action used for supervision during training. The key technical mechanism we use to ensure both future specificity and alignment is the bidirectional flow map.

Thus, the novelty of iWAM lies not in using intermediate activations, but in three key contributions:

  • Identifying the limitation: We empirically demonstrate the limitations of existing WAMs that condition actions on activations from a single video-model forward pass and uncover fundamental flaws in their conditioning mechanisms.
  • Design requirements: We systematically establish the distinct training and inference requirements for retaining the benefits of future conditioning without iterative generation.
  • Proposed mechanism: We address both requirements through a principled use of bidirectional flow maps, enabling demonstration-aligned conditioning during training and future-specific representations in a single evaluation at inference.

Our analysis demonstrates that these requirements are not merely incidental engineering choices: violating these requirements substantially increases action-training loss and degrades downstream performance.

To make this more clear, we split the original Section 5.1 into two subsections: the first explicitly identifies and summarizes the requirements for efficient future conditioning, while the second motivates how bidirectional flow maps satisfy them. We have also added a short paragraph in Section 5.1 motivating the alignment requirement, along with Figure 4 illustrating its importance.

Re: Explaining Notation and Meaning of Equations

The paper uses dense notation that is introduced without sufficient motivation (e.g., the bidirectional flow map parameterization, the shortcut bootstrap loss). Some equations are presented without clear intuition about what they achieve.

We thank the reviewer for the feedback. We would like to clarify that our flow-matching and flow-map formulations follow the conventions of MeanFlow, and we have now also added a notation table (Table 1) summarizing these quantities for easier reference.

To improve clarity, we split the original Section 5.1 into two subsections: the first explicitly identifies and summarizes the requirements for efficient future conditioning, while the second motivates how bidirectional flow maps satisfy them. We also moved the bidirectional flow-map parameterization immediately after its motivation, so the equations follow naturally from the underlying intuition. Finally, we also clarified the intuition behind the shortcut bootstrap objective stated in Section 5.4.

AuthorsOpenReview ↗

Did our Rebuttal Address your Concerns?

Dear Reviewer,

Given that there is just over a day left in the discussion period, we wanted to reach out to see if our rebuttal above addressed all your concerns or if any others still remained? Please let us know if any further clarification would be helpful. If all your concerns are addressed, we would be really grateful if you could acknowledge that soon.

Thanks,

Authors

Reviewer_TPy1OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents iWAM, a world-action model that predicts actions using intermediate features from a single video-backbone evaluation, without explicitly generating future video. Its main mechanism is a bidirectional flow map: during training, demonstrated futures are mapped to corresponding noise endpoints, from which the model extracts features intended to align with the demonstrated actions. At inference, noise is sampled from the prior and processed once to condition an action expert. The full method is evaluated using DROID-trained policies in simulation and on three physical-robot tasks. A simplified prior-noise variant is also evaluated on LIBERO-PRO.

Critical strengths
  1. The method addresses a concrete conditioning problem. Aligning the future represented by the video-model features with the actions used for supervision is a reasonable objective. Recovering a demonstration-associated noise endpoint provides a specific mechanism for addressing this mismatch while retaining a single video-backbone evaluation at deployment.

  2. The evaluation includes useful controls and physical-robot experiments. Sharing the video checkpoint between iWAM and IDM, and including an IDM-flow-map baseline, helps distinguish feature extraction from explicit future generation. The simulation and robot results provide encouraging evidence that the method can retain competitive task performance with fewer video-backbone evaluations.

Critical weaknesses
  1. The incremental contribution over existing efficient video-based policies is not sufficiently established. Both Fast-WAM and VPP already avoid iterative future-video generation by conditioning actions on features obtained through a single video-model pass. The contribution therefore rests primarily on the demonstration-aligned conditioning enabled by the bidirectional flow map. However, evidence isolating this mechanism is limited. The LIBERO-PRO experiments use the simplified prior-noise variant and consequently do not validate the full alignment procedure. On DROID, the training-loss comparison and reported failure of prior-noise conditioning are suggestive, but quantitative task-level ablations under matched video checkpoints and training conditions are needed to establish the mechanism’s independent benefit. The exceptionally large gap to FastWAM also makes the conclusions particularly dependent on the baseline implementation and training recipe.

  2. The efficiency evaluation does not consistently pair performance and latency from the same configuration, and the physical validation remains limited. The performance–latency plots combine success rates obtained with eight action-denoising steps for flow-based policies with latency measured using two steps. These points therefore do not represent the performance and latency of a single evaluated configuration. Although the reduction in video-backbone evaluations is clear, the practical trade-off should be measured using consistent settings. The real-world evaluation covers only three tasks with twenty trials each, providing limited resolution for interpreting the difference between 85% and 80% success. Furthermore, the execution budget excludes inference waiting time, so it does not establish improved wall-clock task completion. In the provided PDF, the robot figures show initial configurations rather than execution sequences, and I could not locate an execution-video link. This limits assessment of control continuity, replanning delays, and failure behavior.

Questions for authors
  1. What are the inference latencies under the exact action-denoising configurations used to obtain the reported success rates? How does the comparison change when task completion time includes inference waiting?

  2. Under a shared video checkpoint and matched action-policy training, what are the task-level success rates for demonstration-inverted endpoints versus independently sampled endpoints? Can the authors also quantify inversion consistency and the distribution mismatch between recovered endpoints and inference-time prior samples?

  3. How does the full bidirectional iWAM perform on LIBERO-PRO? Why is only the simplified prior-noise variant reported there?

  4. Can the authors provide further reproducibility details for the very low FastWAM results, including the omitted two-stage baseline’s task-level scores? Are physical execution videos available in the supplementary materials?

8 comments
AuthorsOpenReview ↗

Rebuttal by Authors - I

Thank you for your detailed review! To address your concerns, we (i) clarify our contributions beyond current WAMs relying on single-pass conditioning, (ii) provide controlled ablations demonstrating the importance of demonstration alignment, along with quantitative analyses showing strong inversion consistency and that close matching between training and inference noise-endpoint distributions is not necessary for iWAM's strong performance, (iii) report inference latencies under the exact evaluation configurations and wall-clock task completion times, (iv) evaluate and report the full bidirectional iWAM on LIBERO-PRO, and (v) provide additional FastWAM implementation details, two-stage baseline results, and representative qualitative rollouts with real-world execution videos.

We have also revised the manuscript to improve clarity and strengthen the presentation: we reorganized Section 5.1 into two subsections separating the requirements for efficient future conditioning from how bidirectional flow maps address them, added Figure 4 to illustrate the importance of alignment with demonstration during training, and expanded Section 4 and Appendix C.4 with qualitative analysis of FastWAM's failure modes and representative film strip comparisons of iWAM and FastWAM rollouts. We have highlighted all the edits in blue. Please let us know if your concerns are addressed.

Re: Novelty over FastWAM and VPP (1/2)

The incremental contribution over existing efficient video-based policies is not sufficiently established. Both Fast-WAM and VPP already avoid iterative future-video generation by conditioning actions on features obtained through a single video-model pass. The contribution therefore rests primarily on the demonstration-aligned conditioning enabled by the bidirectional flow map.

We agree with the reviewer that both Fast-WAM and VPP already take the activations from a single video-model forward pass. However, our analysis in Section 5.1 identifies fundamental limitations in the way these two methods obtain those single-pass features, which constrain their generalization and performance on realistic datasets, as demonstrated through analysis and experimentation (Section 4).

First, Fast-WAM computes activations solely from current observations (i.e., with no actual future prediction). These activations do not correspond to any particular future, as similar observations can be associated with diverse futures. In Section 4, we empirically demonstrate that while not using activations from future imagination does not hurt in-distribution performance, Fast-WAM performs significantly worse than IDM and iWAM under out-of-distribution settings.

Second, while VPP does use activations coming from future imaginations, we show in Section 5.1 and the table below that this alone is not sufficient, and in fact VPP severely underperforms when compared to IDM and our method. VPP uses the activations from a single forward step starting from pure noise (similarly to iWAM (prior noise) from the paper and also iWAM (w/o GT inversion) from below). The activations from this forward pass do not correspond to any particular imagined future, resulting in high action-training loss. Proposition A.1 provides additional theoretical intuition: for an optimally trained flow model, the one-step denoising estimate collapses to the conditional mean, independent of the sampled noise. Nevertheless, as illustrated in Figure 3, an additional problem is that the conditioning features and action targets will not be aligned during training, making the training objective harder to fit.

MethodCube → BowlCan → MugBanana → BinAvg.
162416241624
IDM95%100%95%90%100%100%96.7%
iWAM80%85%100%95%100%100%93.3%
iWAM (w/o GT inversion)0%0%0%0%5%5%1.7%

In Section 5.1, we systematically investigate what is needed for efficient future conditioning, revealing two critical requirements: (1) future specificity, where a single model evaluation must expose information about a particular future, and (2) alignment between the represented future and the actions used for supervision during policy training.

Implicit WAM addresses both these requirements. We retain the advantage of a single forward pass while ensuring that the conditioning features correspond to a particular imagined future and the noise latent is aligned with the action used for supervision during training. The key technical mechanism we use to ensure both future specificity and alignment is the bidirectional flow map.

AuthorsOpenReview ↗

Rebuttal by Authors - II

Re: Novelty over FastWAM and VPP (2/2)

Thus, our novelty extends beyond demonstration alignment alone:

  • Identifying the limitation: We empirically demonstrate the limitations of existing WAMs that condition actions on activations from a single video-model forward pass and uncover fundamental flaws in their conditioning mechanisms.
  • Design requirements: We systematically establish the distinct training and inference requirements for retaining the benefits of future conditioning without iterative generation.
  • Proposed mechanism: We address both requirements through a principled use of bidirectional flow maps, enabling demonstration-aligned conditioning during training and future-specific representations in a single evaluation at inference.

To make this more clear, we split the original Section 5.1 into two subsections: the first explicitly identifies and summarizes the requirements for efficient future conditioning, while the second motivates how bidirectional flow maps satisfy them.

Re: Inference Latency and Wall-Clock Time

What are the inference latencies under the exact action-denoising configurations used to obtain the reported success rates? How does the comparison change when task completion time includes inference waiting?

For latency measurements, we used two action-denoising steps for all methods, matching iWAM (MIP), to enable a better comparison. This ensures that iWAM's latency improvements over IDM arise only from the reduction in the number of video model forward passes. Below, we report the inference latencies under the exact action-denoising configurations used to obtain the reported success rates.

MethodSuccess (%)Latency (ms)
FastWAM9.272.03
IDM-flow map92.5138.69
IDM96.7433.37
iWAM93.397.88
iWAM (MIP)91.762.86

Below, we report the environment execution duration to success (no. of executed environment steps divided by the 15 Hz control frequency), cumulative inference latency (summed across all replanning steps), and total wall-clock time (sum of the two), averaged over successful trials on DROID simulation tasks with a replanning horizon of 16 steps. Across all three tasks, iWAM achieves comparable environment execution duration to IDM, indicating that faster inference does not come at the cost of additional environment interaction. Its substantially reduced inference latency translates into lower average wall-clock time to success across the three tasks.

MethodEnv. execution (s)Cumulative inference latency (s)Total wall-clock time (s)
Cube → Bowl
IDM12.655.3818.04
iWAM12.731.2213.95
iWAM (MIP)13.000.8013.80
Can → Mug
IDM12.755.3518.10
iWAM10.611.0211.63
iWAM (MIP)10.310.6410.95
Banana → Bin
FastWAM6.280.496.77
IDM12.535.3717.90
iWAM12.851.2414.09
iWAM (MIP)12.730.7913.52

FastWAM is reported only on Banana → Bin, as it has no successful trials on Cube → Bowl and Can → Mug (Table 2).

Re: Demonstration-Inverted vs. Independently Sampled Endpoints

Under a shared video checkpoint and matched action-policy training, what are the task-level success rates for demonstration-inverted endpoints versus independently sampled endpoints?

Below, we report the task-level success rates of iWAM using the same bidirectional shortcut video checkpoint and matched action-policy training, comparing demonstration-inverted endpoints against independently sampled endpoints from the prior (iWAM (w/o GT inversion)). Although both approaches expose information about a particular future, these results show that aligning the conditioning with the demonstrated actions through inversion substantially improves task success.

MethodCube → BowlCan → MugBanana → BinAvg.
162416241624
iWAM (w/o GT inversion)0%0%0%0%5%5%1.7%
iWAM80%85%100%95%100%100%93.3%
AuthorsOpenReview ↗

Rebuttal by Authors - III

Re: Inversion Consistency and Distribution Mismatch (1/2)

Can the authors also quantify inversion consistency and the distribution mismatch between recovered endpoints and inference-time prior samples?

Inversion Consistency

We evaluate inversion consistency on 5.8k DROID video samples, where each sample contains one current observation and 16 future frames drawn from a different episode. We show results where ground-truth VAE latents are inverted using 1, 2, and 4 shortcut inversion steps and reconstructed using 1, 2, 4, and 8 shortcut denoising steps for guidance strengths w=7w = 7 and w=1w = 1. We report the MSE between the decoded reconstruction and the VAE-decoded ground truth, averaged over future-frame RGB pixels and all samples, excluding the current observation. Both videos are in the [0, 1] RGB range when computing the MSE. MSE values are reported in units of 10−310^{-3}, and values in parentheses indicate the percentage reduction in MSE relative to Gaussian initialization with the same guidance strength and number of denoising steps.

Guidance strength w=7w = 7

Initialization1-step denoising2-step denoising4-step denoising8-step denoising
Gaussian reference27.710 (0.00%)30.801 (0.00%)32.898 (0.00%)33.853 (0.00%)
1-step inversion4.522 (83.68%)4.838 (84.29%)5.337 (83.78%)5.633 (83.36%)
2-step inversion3.034 (89.05%)2.972 (90.35%)3.156 (90.41%)3.218 (90.49%)
4-step inversion2.933 (89.42%)2.365 (92.32%)2.225 (93.24%)2.141 (93.68%)

Guidance strength w=1w = 1

Initialization1-step denoising2-step denoising4-step denoising8-step denoising
Gaussian reference27.851 (0.00%)30.450 (0.00%)32.036 (0.00%)32.772 (0.00%)
1-step inversion4.463 (83.98%)4.609 (84.86%)4.457 (86.09%)4.370 (86.67%)
2-step inversion3.331 (88.04%)3.175 (89.57%)2.961 (90.76%)2.834 (91.35%)
4-step inversion4.109 (85.25%)3.554 (88.33%)3.101 (90.32%)2.722 (91.69%)

Even 1-step inversion significantly reduces MSE compared to Gaussian initialization, achieving over 83% reduction across all denoising steps counts. Using 2 inversion steps further reduces MSE consistently across all denoising step counts.

Distribution Mismatch

Thanks for pointing out the potential issue regarding the distribution mismatch. First, we would like to highlight that iWAM doesn't require the inversion endpoint distribution used during training and the prior distribution to match or even be close, for strong test-time performance. Crucially, as long as the video model is trained well, it can generate consistent futures under both these distributions. This allows the action policy to learn from representations aligned with the demonstrated future during training and use representations associated with a specific plausible future at inference.

We quantify the discrepancy between recovered inversion endpoints and the standard Gaussian prior using squared maximum mean discrepancy (MMD²), using techniques from [1], and also show that the inverted endpoints can be distinguished from the prior with high accuracy.

Setup: We evaluate 57.6k distinct DROID episodes, randomly selecting an observation–future window per episode, and show results where ground-truth VAE latents are inverted using 1, 2, and 4 shortcut inversion steps. Each recovered future endpoint consists of four VAE latent frames, which are flattened into a d=76,800d = 76{,}800-dimensional vector. We use an equally weighted mixture of Gaussian RBF kernels with squared bandwidths {d/2,d,2d}\lbrace d/2, d, 2d \rbrace. For each inversion step count, we estimate MMD² by dividing the 57.6k recovered endpoints into 225 blocks of 256 samples and averaging the unbiased estimates across blocks.

MMD² results: We observe that MMD² decreases from 0.005022 to 0.004846 to 0.004767 as the number of inversion steps increases from 1 to 2 to 4. (For scale, the MMD² between N(0,0.92I)\mathcal{N}(0, 0.9^2 I) and N(0,I)\mathcal{N}(0, I) is approximately 0.003672 under the same kernel.) These values indicate a measurable distribution mismatch, slightly larger than that caused by a 10% reduction in the Gaussian prior's standard deviation, which decreases only modestly with additional inversion steps.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.