- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how autoencoder representations affect video world models. The authors find that reconstruction-only training can produce latents where dynamics are difficult to predict. They combine temporal learning objectives with a decoder conditioned on a past frame to improve latent predictability while preserving reconstruction ability. Experiments on robotic manipulation data show improved OOD prediction and long-horizon rollouts, fewer sampling steps at comparable quality, and better agreement between simulated and real-world policy evaluations.
Critical strengths
The paper is well-written and their idea is easy to follow. The paper provides a controlled study of temporal objectives for autoencoder training. Comparisons at matched reconstruction quality, together with controls for decoder conditioning, show that these objectives improve downstream world-model prediction beyond reconstruction quality alone. This provides useful empirical guidance for learning latent representations that better support dynamics modeling.
Critical weaknesses
- Limited validation on modern video VAEs. The method is evaluated on framewise image encoders, without testing temporally compressed video VAEs such as Wan-VAE. These models already learn spatiotemporal structure and reduce downstream sequence length. Consequently, it remains unclear whether the proposed objectives provide additional benefits for modern video tokenizers. After response, I still hold my concern for Video VAE. A temporally compressed video VAE can represent the same rollout with fewer tokens than a framewise encoder, reducing latency in robotics applications (which is the main data domain studied in this paper). Without a direct comparison, it remains unclear whether the proposed method offers better performance. I therefore maintain my current score.
- Insufficient evidence for end-to-end efficiency. The reported speedups measure world-model training updates and sampling steps, without establishing total savings after accounting for autoencoder fine-tuning and conditional decoding. Without measured training costs and full-rollout latency, the practical efficiency advantage remains unclear.
Questions for authors
- I think some video VAEs should be involved to prove the effectiveness of proposed method. Can the authors evaluate their objectives on such a tokenizer while matching reconstruction quality, rollout duration, and control frequency?
- Beyond Tab. 6, can the authors complete the matched-budget comparison with temporal objectives on the ordinary decoder, and report full-rollout quality versus measured latency across sampling budgets?
2 comments
Response to Reviewer opdB [2/2]
(2) Ordinary decoder at the matched budget
The reviewer asks us to complete the matched-budget comparison with the temporal objectives on the ordinary decoder.
We provide additional experiments for the matched-budget comparison with both temporal objectives on the ordinary decoder (full results in Tables 6 and 8; the rows of the submitted Table 6 are now in Tables 2 and 6).
We fine-tuned each autoencoder for the same 80,000 updates as the other pipelines, trained the same world model on its latents, and evaluated it in the same conditions, policy evaluation included. For latent prediction:
| Decoder, objective | LPIPS ↓, out of distribution | LPIPS ↓, long horizon | LPIPS ↓, 2 steps | Pearson r with real-world policy success ↑, 2 steps |
|---|---|---|---|---|
| Ordinary, reconstruction (baseline) | 0.302 | 0.394 | 0.232 | 0.21 |
| Past-conditioned, reconstruction | 0.270 | 0.366 | 0.198 | 0.15 |
| Ordinary, latent prediction | 0.287 | 0.342 | 0.191 | 0.62 |
| Past-conditioned, latent prediction | 0.257 | 0.331 | 0.177 | 0.67 |
Both components contribute. The decoder alone improves the offline metrics but not policy evaluation. The objective alone improves both, and carries the gain in policy evaluation. Together, they are best on all four metrics. The same picture mostly holds for SSIM and pixel MSE, and for inverse dynamics (Tables 6 and 8).
(3) Video autoencoders such as the Wan VAE
The reviewer argues that video autoencoders should also be considered.
The main contribution of our study is a careful understanding of the role of temporal structure in autoencoders for video world models. It rests on controlled comparisons and a characterization of the main trade-offs, which provide valuable insights and practical guidance. Our goal is therefore not to build the best possible video world model.
We agree that autoencoders that compress along the time axis are an important extension for future work. However, in these autoencoders, time mostly serves as an additional compression axis and enters the training objective, if at all, through pixel-space terms (Section 2), not through a predictive task in latent space like ours. We therefore believe that they would not fundamentally change the sequence-modeling task at hand, and would offer limited new insight for our goals. In fact, single frames are already a compressed, though not learned, representation of an extended temporal chunk of the underlying continuous video stream. The relationships between frame representations that we study are therefore not very different from those between the chunk representations of a video autoencoder.
Response to Reviewer opdB [1/2]
We thank the reviewer for recognizing the clarity of the paper and the value of our controlled study, which gives useful empirical guidance for learning latents that support dynamics modeling. We refer the reviewer to the general response for a summary of contributions and changes to the manuscript. Here, we address their specific concerns.
(1) End-to-end efficiency
The reviewer asks whether the speed-ups hold after accounting for autoencoder fine-tuning and the past-conditioned decoder, with measured training cost and rollout latency.
We provide additional end-to-end measurements of training time and rollout latency to show that the speed-ups hold once autoencoder fine-tuning and the past-conditioned decoder are accounted for (details in Appendix H, new). The reported numbers of world model updates and sampling steps remain useful implementation-independent measures of efficiency.
For training, we fine-tuned the FLUX VAE with latent prediction and the past-conditioned decoder, as in Section 5.3, for only five epochs (5,540 updates, 3.0 hours on one H200 GPU), trained the same world model on its latents, and measured the total training time, fine-tuning included, after which its validation metrics first reach the baseline's final values. For inference, we timed complete nine-frame rollouts on one H200 GPU at batch 1, from the first frame and the actions to the decoded frames, with 1 to 10 sampling steps.
Tables 17 and 18 and Figure 29 give the training speed and the rollout time against the baseline, which takes 46.7 hours of total training and 2.58 seconds per rollout with 10 steps:
Against the pretrained autoencoder, which needs no fine-tuning and takes 15.4 hours, the five-epoch pipeline reaches the final LPIPS and SSIM 3.6 and 3.9 times sooner. After full training, the five-epoch pipeline is better than the baseline on all nine out-of-distribution, long-horizon and two-step metrics (Table 16). With 59 frames generated in chunks of 8, as in the long-rollout evaluation, the saving in rollout time is smaller: 1.5 times for inverse dynamics with 1 step and 1.4 times for latent prediction with 2 steps (Appendix H).
So the savings in world-model updates and sampling steps carry over to wall-clock time: the five-epoch fine-tuning costs 20% of one world-model training run, but is enough to recover most of the gains of the full 80k-step fine-tuning; and the past-conditioned decoder adds only about 0.12 seconds to a nine-frame rollout, small next to the time saved by fewer sampling steps.