- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates how autoencoder design affects the performance of interactive world models. Specifically, it evaluates how spatial and temporal compression impact downstream world modelling when the compression is handled by either the VAE or patchification within the DiT world model. The paper's investigation spans both navigation and manipulation and reports mixed results between the two settings, which I found particularly interesting.
Critical strengths
Overall, the paper is well written and easy to follow. The paper investigates an interesting and important direction, that is, how best to design video autoencoders when using the latent space for world modelling with a DiT.
I already knew that compression via the VAE is better than compression via patchification. However, it was interesting to see the comparison between navigation and manipulation and how they're impacted differently by temporal and spatial compression.
Critical weaknesses
One of my largest concerns is regarding the evaluation of the baselines, which were not fine-tuned on the same data. A fairer comparison would be to fine-tune Wan/Cosmos, etc on the same data. At least for Wan, I know that all details are in the paper and it's easy to reproduce training. I would not expect these pre-trained encoders to perform well without being fine-tuned on the same data. The easiest approach would be to leave the encoder frozen and just fine-tune the decoder. That way the world models don't need to be retrained. Just fine-tune the decoder and then re-run inference and evals.
With regard to the issue at chunk boundaries, this seems like a significant failing of the VAE if it's to be used for world modelling in a streaming decode setup. The Wan VAE is causal and I have not observed the visible jumps between chunks. Why did you opt for a DiT-style causal decoder using a sliding window, instead of the causal style used by WAN, which uses a causal state? Furthermore, can you elaborate on how the inference is done in the VAE? Did the VAE decoder use a full KV cache, a sliding window with KV cache, or a sliding window with KV exactly recomputed for each window?
Also, the method's spatial and temporal compression investigation is within a VAE setup I would call non-standard. It would be interesting to take the other VAEs, e.g. Wan, and modify the architecture to run at the different compression rates in the paper. They can then be trained (fine-tuned) on the same data. I suspect the VAE architecture also plays a significant role in how well the different compression rates impact world model performance.
Questions for authors
- Were all the baselines fine-tuned on your data? If not, why not?
- How is inference performed with the decoder? Does it maintain a full KV, or use a sliding window, etc.?
- In video world models, it is common to use full attention over space and time instead of factorised space-time attention. I suspect this decision can impact how the model performs under different VAE comparisons, etc.
- In many video VAEs, the first frame is not temporally compressed. In CADRE, is the first frame temporally compressed?
3 comments
Response to Reviewer hihA (2/2): Compression within CADRE, attention and first frame
W3. It would be interesting to take the other VAEs, e.g. Wan, and modify the architecture to run at the different compression rates in the paper.
We agree that the architecture matters, which is why Section 4.2 compares CADRE with five autoencoders of different designs. To study compression itself, we vary it within one family, so that nothing else changes: only the bottleneck, the lift and the temporal patch size depend on the compression (Figure 3). Rebuilding Wan 2.2 at each compression would need a training codebase it does not release, seven new autoencoders and 28 world models. We will state in the paper that the compression trends are measured within the CADRE family and are left open for other architectures like Wan’s.
Q3. In video world models, it is common to use full attention over space and time instead of factorised space-time attention. I suspect this decision can impact how the model performs under different VAE comparisons, etc.
We agree: factorized attention is an inductive bias that keeps the world model fast enough for real-time interaction, but it limits what the world model can express, and it may change how much each autoencoder benefits. Our world model follows MIRA, designed for interactive world modeling, as our study is about the autoencoder; we keep its factorized attention because it keeps the world model real-time. With full attention over the 40-frame context, each world-model step would need up to 2.6× more compute: 1.8× for CADRE-2×16×16, which already runs at the edge of real time, and 2.6× for CADRE-1×16×16. The same world model serves every autoencoder, so the comparison is controlled, though only for this architecture, which we will state as a limitation.
Q4. In many video VAEs, the first frame is not temporally compressed. In CADRE, is the first frame temporally compressed?
Yes, the temporal bottleneck is a convolution whose kernel and stride both equal τ, without padding, so every latent covers exactly τ frames, the first chunk included (Appendix B). This suits streaming, where every world-model step produces τ frames, and a single context frame could be repeated τ times to fill the first latent.
We hope these answers address the reviewer's concerns, and are happy to clarify any further point.
I thank the authors for their rebuttal.
I have taken the time to read through all reviews and rebuttals. On the most part, the response has answered my questions.
Response to Reviewer hihA (1/2): Baselines and streaming decoder
We thank the reviewer for the careful review, and are glad they found the paper clear and the contrast between navigation and manipulation interesting. In short, we use the off-the-shelf autoencoders as released, since Wan 2.2 and LTX-2 do not release the training code of their autoencoders, and decoding does not seem to be what limits Wan 2.2, which reconstructs held-out video better than any other autoencoder yet gives weaker world models (W1, Q1).Our decoder recomputes its 16-frame window exactly at each step, and seams, a cost of long latents, are smallest at two frames per latent, the setting of our real-time models (W2, Q2). Our compression trends come from varying compression within CADRE, where nothing else changes, as other autoencoders cannot be rebuilt at each compression without their training code (W3). We keep the world model's factorized attention, since full attention would cost up to 2.6× more compute per step and push our fine-grid models out of real time (Q3). CADRE compresses the first frame with the next ones, and a single context frame could be repeated to fill a chunk if no context video is available (Q4).
We use the off-the-shelf autoencoders as released, as a practitioner would. Video autoencoders we compare to were trained on far more video than CADRE, likely including many of the datasets we use (Section 4). Wan 2.2 and LTX-2 do not release the training code of their video autoencoders, so fine-tuning them would mean re-implementing their training from its description, and we would no longer evaluate the released models. More importantly, decoding does not seem to be what holds Wan 2.2 back: on held-out general video (OpenVid-HD, Table 6), it reconstructs best of all twelve autoencoders (PSNR 35.0 dB, against 31.4 for Cosmos, the next best), yet ranks fifth on rollout FVD in robot manipulation and last indoors (Appendix F). Fine-tuning its decoder could still help on the latents the world model generates, which differ from encoded ones, but it would not change those latents. The comparison that controls data and budget is the MIRA recipe, retrained on our data, where CADRE lowers FVD by 1.4 to 2.5×.
CADRE is a representation autoencoder: its encoder is a frozen DINOv3 with a small spatio-temporal bottleneck, and the decoder has to turn semantic features back into pixels. RAE (Zheng et al., 2026) does this with a ViT decoder, and MIRA showed that a causal version of it works for an interactive world model, so we started from MIRA's decoder and trained it as a diffusion model. We did not test whether Wan's convolutional decoder can decode such features because if it had failed, it would have been hard to tell why (bad re-implementation because the original code is not available, or a fundamental incompatibility of the decoder with an RAE-like setup).
In a transformer, the counterpart of Wan's causal state is a key-value cache. At inference, we instead recompute the 16-frame window the decoder was trained on for each new latent and keep the new frames. As part of this rebuttal, we also tried a cache: it is 1.4 to 1.7× faster for τ ≤ 2, but it lowers PSNR by 1.6 to 5 dB, likely because the cached activations still depend on frames that have left the window, which never happens in training.
Seams are a cost of long latents, which is one reason the paper recommends few frames per latent when the camera moves (Section 5.2). They are smallest at two frames per latent, the setting of our real-time models: 8 to 47% more change at chunk boundaries than real video, against 58 to 83% at eight (Appendix K).