- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper demonstrates through experiments that FastWAM-style efficiency can come at the cost of weaker generalization. Motivated by this, it proposes iWAM, which uses a bidirectional flow map to extract future-specific intermediate representations with a single video-model evaluation, avoiding explicit future generation. Experiments show that iWAM achieves performance close to IDM with substantially lower inference cost.
Critical strengths
-
The paper demonstrates that FastWAM-style efficiency can hurt generalization through experiments.
-
The paper addresses iWAM, which offers a good performance-latency trade-off.
-
The paper includes real-robot experiments, which strengthens the empirical validation.
Critical weaknesses
-
Train-test mismatch. During training, the action policy is conditioned on noise endpoints recovered from GT future observations, while at inference the endpoint is sampled directly from the prior. Since the clean-to-noise map is only approximate, it is unclear whether these two endpoint distributions match.
-
The one-step distillation method could be better explored. Recent works[1,2,3,4] provide stronger one-step causal video distillation techniques, while direct one-step generation videos show bad performance. It would be useful to try these alternatives. And the represention of the laywise forward activations could be explored, which helps understand why the method is useful.
-
The evaluation benchmarks show substantial saturation. On the selected DROID simulation tasks, strong methods already achieve over 90% average success, with many settings reaching 95–100%. Similarly, on the unperturbed LIBERO setting, most methods achieve around 97–99% success. These ceiling effects limit the ability of the current benchmarks to distinguish strong methods. Evaluating on harder tasks or more challenging benchmarks would strengthen the empirical conclusions.
-
Presentation. Figure 1 could be improved; in particular, the connections across the video-model blocks are somewhat confusing. Besides, Table 1 could also explicitly state that it reports results on the DROID simulation benchmark.
[1] AAD-1: Asymmetric Adversarial Distillation for One-Step Autoregressive Video Generation
[2] One-Forcing: Towards Stable One-Step Autoregressive Video Generation
[3] Towards One-Step Causal Video Generation via Adversarial Self-Distillation
[4] Causal-rCM: A Unified Teacher-Forcing and Self-Forcing Open Recipe for Autoregressive Diffusion Distillation in Streaming Video Generation and Interactive World Models
Questions for authors
Do the authors plan to release the training and evaluation code, as well as model checkpoints, to facilitate reproducibility?
4 comments
Rebuttal by Authors - II
Re: One-Step Distillation
The one-step distillation method could be better explored. Recent works[1,2,3,4] provide stronger one-step causal video distillation techniques, while direct one-step generation videos show bad performance. It would be useful to try these alternatives. And the represention of the laywise forward activations could be explored, which helps understand why the method is useful.
We thank the reviewer for highlighting recent advances in one-step video distillation and suggesting layer-wise representation analysis, and we will cite and discuss them in our paper. These works offer promising extensions to our bidirectional shortcut objective: (i) incorporating auxiliary GAN loss as in One-Forcing [2], potentially using the bidirectionally attentive video discriminator from AAD-1 [1]; (ii) complementing our existing shortcut bootstrapping loss with adversarial self-distillation across different denoising-step counts [3]; and (iii) exploring DMD-based refinement, potentially adopting the self-forcing strategy of Causal-rCM [4]. These techniques could improve generation quality while retaining our bidirectional consistency objective. However, our work primarily focuses on identifying the distinct requirements for efficient future condition and addressing them through a principled use of simple bidirectional flow maps. We therefore leave incorporating these advanced distillation techniques as promising directions for future work.
We also thank the reviewer for suggesting layer-wise activation analysis. Section 5.1 already provides empirical evidence that iWAM's forward activations capture the relevant mode of the future-video distribution. Specifically, a single forward evaluation of a standard flow model from randomly sampled noise yields high action-training loss. We now further explicitly mention in Section 5.1 that even conditioning on a fully generated, plausible future does not reduce this loss when the future is sampled independently of the demonstration.
In contrast, forward activations evaluated at the noise endpoint obtained by inverting the demonstrated future achieve action-training loss close to IDM, which directly conditions on that future (Figure 3). Together, these controlled experiments provide strong evidence that our activations encode the demonstration-aligned future mode relevant to the supervised actions during training, rather than merely representing an arbitrary plausible future. While layer-wise visualization could further reveal how this information is distributed across network layers and remains an interesting direction for future work, our experiments directly establish what information the activations preserve and why it matters.
Re: Potential Saturation of Eval Benchmarks
The evaluation benchmarks show substantial saturation. On the selected DROID simulation tasks, strong methods already achieve over 90% average success, with many settings reaching 95–100%. Similarly, on the unperturbed LIBERO setting, most methods achieve around 97–99% success. These ceiling effects limit the ability of the current benchmarks to distinguish strong methods. Evaluating on harder tasks or more challenging benchmarks would strengthen the empirical conclusions.
We would like to note that we do not rely on unperturbed LIBERO to evaluate generalization; rather, we use its strong in-distribution performance to highlight benchmark saturation (Lines 73–78, Lines 258–266). Although FastWAM achieves 96.9% success in-distribution, its performance degrades significantly under the out-of-distribution perturbations in LIBERO-PRO, where the gap to IDM becomes substantially larger. To make this takeaway clearer, we add Figure 2 to illustrate this. In particular, the relative performance gap increases from approximately 1% in-distribution to 16% across OOD perturbations under joint training. This widening gap is precisely the result we highlight and analyze in the paper to motivate our approach.
We also evaluate challenging physical manipulation tasks for DROID policies, including towel folding and pen transfer in cluttered environments. Successful execution often requires multiple attempts, allowing us to observe how policies recover from failed grasps and handle distractors. These evaluations reveal important differences: while IDM and iWAM can recover and complete the tasks, FastWAM often abandons the target after a failed grasp or manipulates unrelated objects instead of following the instruction. These failures contribute to its substantially lower real-world success rate (13.3%, compared with 80.0% for IDM and 85.0% for iWAM (MIP)).
To make these differences clearer, we have added qualitative rollout illustrations, including representative film strips comparing iWAM and FastWAM, along with a discussion of FastWAM's failure modes in Section 4 and Appendix C.4.
Rebuttal by Authors - III
Re: Presentation
Presentation. Figure 1 could be improved; in particular, the connections across the video-model blocks are somewhat confusing. Besides, Table 1 could also explicitly state that it reports results on the DROID simulation benchmark.
Thank you for your constructive suggestions. We have revised Figure 1 to improve its clarity and also updated the caption of the table accordingly.
Re: Code and Checkpoint Release
Do the authors plan to release the training and evaluation code, as well as model checkpoints, to facilitate reproducibility?
Yes, we plan to publicly release our training and evaluation code, along with model checkpoints, upon acceptance as part of the camera-ready version, to facilitate reproducibility.
Did our Rebuttal Address your Concerns?
Dear Reviewer,
Given that there is just over a day left in the discussion period, we wanted to reach out to see if our rebuttal above addressed all your concerns or if any others still remained? Please let us know if any further clarification would be helpful. If all your concerns are addressed, we would be really grateful if you could acknowledge that soon.
Thanks,
Authors
Rebuttal by Authors - I
Thank you for your feedback! To address your concerns, we (i) quantify the distribution mismatch between recovered noise endpoints and explain why it does not prevent strong test-time performance, (ii) clarify that unperturbed LIBERO serves to highlight saturation rather than evaluate generalization, while also highlighting our evaluations on real-world manipulation tasks where success requires recovering from failures and handling distractors.
We have also revised the manuscript to improve clarity and strengthen the analysis: we added Figure 2 to highlight the generalization gap under LIBERO-PRO perturbations, expanded Section 4 and Appendix C.4 with qualitative analysis of FastWAM's failure modes and representative film strip comparisons of iWAM and FastWAM rollouts, and improved Figure 1. We have highlighted all the edits in blue.
Re: Distribution Mismatch
Thanks for pointing out the potential issue regarding the distribution mismatch. First, we would like to highlight that iWAM doesn't require the inversion endpoint distribution used during training and the prior distribution to match or even be close, for strong test-time performance. Crucially, as long as the video model is trained well, it can generate consistent futures under both these distributions. This allows the action policy to learn from representations aligned with the demonstrated future during training and use representations associated with a specific plausible future at inference.
We quantify the discrepancy between recovered inversion endpoints and the standard Gaussian prior using squared maximum mean discrepancy (MMD²), using techniques from [5], and also show that the inverted endpoints can be distinguished from the prior with high accuracy.
Setup: We evaluate 57.6k distinct DROID episodes, randomly selecting an observation–future window per episode, and show results where ground-truth VAE latents are inverted using 1, 2, and 4 shortcut inversion steps. Each recovered future endpoint consists of four VAE latent frames, which are flattened into a d=76,800-dimensional vector. We use an equally weighted mixture of Gaussian RBF kernels with squared bandwidths {d/2,d,2d}. For each inversion step count, we estimate MMD² by dividing the 57.6k recovered endpoints into 225 blocks of 256 samples and averaging the unbiased estimates across blocks.
MMD² results: We observe that MMD² decreases from 0.005022 to 0.004846 to 0.004767 as the number of inversion steps increases from 1 to 2 to 4. (For scale, the MMD² between N(0,0.92I) and N(0,I) is approximately 0.003672 under the same kernel.) These values indicate a measurable distribution mismatch, slightly larger than that caused by a 10% reduction in the Gaussian prior's standard deviation, which decreases only modestly with additional inversion steps.
We additionally assess distinguishability using a simple threshold classifier based on the normalized latent radius r(z)=∥z∥2/d, which is approximately 1 for samples from the standard Gaussian prior. For each inversion setting, we select the threshold for the classifier using a mixture of 28.8k recovered endpoints and 28.8k independent Gaussian samples. We then evaluate the fixed classifier on a disjoint, balanced test set of 57.6k samples (28.8k held-out recovered endpoints and 28.8k Gaussian samples). Held-out accuracy is 99.85%, 99.18%, and 97.06% for 1-, 2-, and 4-step inversion, respectively, compared with 49.74% for an independent Gaussian-versus-Gaussian control.
Thus, additional inversion steps reduce both MMD² and radius-based distinguishability, although the recovered endpoints are easily distinguishable from the prior.
We have also added Figure 4, which demonstrates that the video model can generate futures effectively from both distributions while maintaining strong inversion consistency. (The reconstructed future is obtained using 2 shortcut inversion steps, and both futures are generated using 8 shortcut denoising steps.)
Takeaway: These results confirm a measurable mismatch between the demonstration-inverted endpoint distribution used during training and the Gaussian prior sampled at inference. Despite this mismatch, iWAM achieves strong test-time performance, supporting our claim that matching the training and inference endpoint distributions is not necessary for effective action prediction.
[5] Rustamov, Raif M. "Closed‐form expressions for maximum mean discrepancy with applications to Wasserstein auto‐encoders."