acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Video Autoencoders for Interactive World Models

32 views

Abstract

Video world models predict future frames from past frames and actions. They serve as simulators in which agents train, plan or are evaluated, and that humans can control interactively. They generate in the latent space of a video autoencoder, whose compression sets their cost per frame and how long each action waits to take effect. Yet world models often reuse the autoencoders of text-to-video generators, built for offline generation with no agent waiting and, without their training data and code, not retrainable at another compression. We therefore ask how, where and how much a world model should compress. We build CADRE, an open family of causal diffusion representation autoencoders spanning multiple spatio-temporal compressions. Its members differ only in a small bottleneck on frozen features and share a diffusion decoder self-distilled to one or a few steps. We train the same world model on each autoencoder in two domains, manipulation and navigation, with two environments each. CADRE forms most of the latency-quality frontier, especially with a moving camera. Our study also leads to four findings. At strong compression, a diffusion decoder is sharper than perceptual regression. In robot manipulation, temporal compression is more effective in the autoencoder than in the world model. Manipulation and navigation call for opposite compressions: temporal compression costs little when the camera is still, and spatial compression when it moves. Surprisingly, with a still camera, coarser spatial tokens hurt motion far more than image quality. We will release code and models; rollouts are at https://wmoptimal.github.io

Primary area:
generative models
Keywords:
world models, video autoencoders, latent compression, interactive video generation, diffusion decoders, action-conditioned video prediction, real-time generation, representation autoencoders, few-step diffusion

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_hihAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how autoencoder design affects the performance of interactive world models. Specifically, it evaluates how spatial and temporal compression impact downstream world modelling when the compression is handled by either the VAE or patchification within the DiT world model. The paper's investigation spans both navigation and manipulation and reports mixed results between the two settings, which I found particularly interesting.

Critical strengths

Overall, the paper is well written and easy to follow. The paper investigates an interesting and important direction, that is, how best to design video autoencoders when using the latent space for world modelling with a DiT.

I already knew that compression via the VAE is better than compression via patchification. However, it was interesting to see the comparison between navigation and manipulation and how they're impacted differently by temporal and spatial compression.

Critical weaknesses

One of my largest concerns is regarding the evaluation of the baselines, which were not fine-tuned on the same data. A fairer comparison would be to fine-tune Wan/Cosmos, etc on the same data. At least for Wan, I know that all details are in the paper and it's easy to reproduce training. I would not expect these pre-trained encoders to perform well without being fine-tuned on the same data. The easiest approach would be to leave the encoder frozen and just fine-tune the decoder. That way the world models don't need to be retrained. Just fine-tune the decoder and then re-run inference and evals.

With regard to the issue at chunk boundaries, this seems like a significant failing of the VAE if it's to be used for world modelling in a streaming decode setup. The Wan VAE is causal and I have not observed the visible jumps between chunks. Why did you opt for a DiT-style causal decoder using a sliding window, instead of the causal style used by WAN, which uses a causal state? Furthermore, can you elaborate on how the inference is done in the VAE? Did the VAE decoder use a full KV cache, a sliding window with KV cache, or a sliding window with KV exactly recomputed for each window?

Also, the method's spatial and temporal compression investigation is within a VAE setup I would call non-standard. It would be interesting to take the other VAEs, e.g. Wan, and modify the architecture to run at the different compression rates in the paper. They can then be trained (fine-tuned) on the same data. I suspect the VAE architecture also plays a significant role in how well the different compression rates impact world model performance.

Questions for authors
  1. Were all the baselines fine-tuned on your data? If not, why not?
  2. How is inference performed with the decoder? Does it maintain a full KV, or use a sliding window, etc.?
  3. In video world models, it is common to use full attention over space and time instead of factorised space-time attention. I suspect this decision can impact how the model performs under different VAE comparisons, etc.
  4. In many video VAEs, the first frame is not temporally compressed. In CADRE, is the first frame temporally compressed?
3 comments
AuthorsOpenReview ↗

Response to Reviewer hihA (1/2): Baselines and streaming decoder

We thank the reviewer for the careful review, and are glad they found the paper clear and the contrast between navigation and manipulation interesting. In short, we use the off-the-shelf autoencoders as released, since Wan 2.2 and LTX-2 do not release the training code of their autoencoders, and decoding does not seem to be what limits Wan 2.2, which reconstructs held-out video better than any other autoencoder yet gives weaker world models (W1, Q1).Our decoder recomputes its 16-frame window exactly at each step, and seams, a cost of long latents, are smallest at two frames per latent, the setting of our real-time models (W2, Q2). Our compression trends come from varying compression within CADRE, where nothing else changes, as other autoencoders cannot be rebuilt at each compression without their training code (W3). We keep the world model's factorized attention, since full attention would cost up to 2.6× more compute per step and push our fine-grid models out of real time (Q3). CADRE compresses the first frame with the next ones, and a single context frame could be repeated to fill a chunk if no context video is available (Q4).

W1 / Q1. A fairer comparison would be to fine-tune Wan/Cosmos, etc on the same data. [...] Were all the baselines fine-tuned on your data? If not, why not?

We use the off-the-shelf autoencoders as released, as a practitioner would. Video autoencoders we compare to were trained on far more video than CADRE, likely including many of the datasets we use (Section 4). Wan 2.2 and LTX-2 do not release the training code of their video autoencoders, so fine-tuning them would mean re-implementing their training from its description, and we would no longer evaluate the released models. More importantly, decoding does not seem to be what holds Wan 2.2 back: on held-out general video (OpenVid-HD, Table 6), it reconstructs best of all twelve autoencoders (PSNR 35.0 dB, against 31.4 for Cosmos, the next best), yet ranks fifth on rollout FVD in robot manipulation and last indoors (Appendix F). Fine-tuning its decoder could still help on the latents the world model generates, which differ from encoded ones, but it would not change those latents. The comparison that controls data and budget is the MIRA recipe, retrained on our data, where CADRE lowers FVD by 1.4 to 2.5×.

W2 / Q2. With regard to the issue at chunk boundaries, this seems like a significant failing of the VAE if it's to be used for world modelling in a streaming decode setup. [...] Why did you opt for a DiT-style causal decoder using a sliding window, instead of the causal style used by WAN, which uses a causal state? [...] How is inference performed with the decoder? Does it maintain a full KV, or use a sliding window, etc.?

CADRE is a representation autoencoder: its encoder is a frozen DINOv3 with a small spatio-temporal bottleneck, and the decoder has to turn semantic features back into pixels. RAE (Zheng et al., 2026) does this with a ViT decoder, and MIRA showed that a causal version of it works for an interactive world model, so we started from MIRA's decoder and trained it as a diffusion model. We did not test whether Wan's convolutional decoder can decode such features because if it had failed, it would have been hard to tell why (bad re-implementation because the original code is not available, or a fundamental incompatibility of the decoder with an RAE-like setup).

In a transformer, the counterpart of Wan's causal state is a key-value cache. At inference, we instead recompute the 16-frame window the decoder was trained on for each new latent and keep the new frames. As part of this rebuttal, we also tried a cache: it is 1.4 to 1.7× faster for τ ≤ 2, but it lowers PSNR by 1.6 to 5 dB, likely because the cached activations still depend on frames that have left the window, which never happens in training.

Seams are a cost of long latents, which is one reason the paper recommends few frames per latent when the camera moves (Section 5.2). They are smallest at two frames per latent, the setting of our real-time models: 8 to 47% more change at chunk boundaries than real video, against 58 to 83% at eight (Appendix K).

AuthorsOpenReview ↗

Response to Reviewer hihA (2/2): Compression within CADRE, attention and first frame

W3. It would be interesting to take the other VAEs, e.g. Wan, and modify the architecture to run at the different compression rates in the paper.

We agree that the architecture matters, which is why Section 4.2 compares CADRE with five autoencoders of different designs. To study compression itself, we vary it within one family, so that nothing else changes: only the bottleneck, the lift and the temporal patch size depend on the compression (Figure 3). Rebuilding Wan 2.2 at each compression would need a training codebase it does not release, seven new autoencoders and 28 world models. We will state in the paper that the compression trends are measured within the CADRE family and are left open for other architectures like Wan’s.

Q3. In video world models, it is common to use full attention over space and time instead of factorised space-time attention. I suspect this decision can impact how the model performs under different VAE comparisons, etc.

We agree: factorized attention is an inductive bias that keeps the world model fast enough for real-time interaction, but it limits what the world model can express, and it may change how much each autoencoder benefits. Our world model follows MIRA, designed for interactive world modeling, as our study is about the autoencoder; we keep its factorized attention because it keeps the world model real-time. With full attention over the 40-frame context, each world-model step would need up to 2.6× more compute: 1.8× for CADRE-2×16×16, which already runs at the edge of real time, and 2.6× for CADRE-1×16×16. The same world model serves every autoencoder, so the comparison is controlled, though only for this architecture, which we will state as a limitation.

Q4. In many video VAEs, the first frame is not temporally compressed. In CADRE, is the first frame temporally compressed?

Yes, the temporal bottleneck is a convolution whose kernel and stride both equal τ, without padding, so every latent covers exactly τ frames, the first chunk included (Appendix B). This suits streaming, where every world-model step produces τ frames, and a single context frame could be repeated τ times to fill the first latent.

We hope these answers address the reviewer's concerns, and are happy to clarify any further point.

I thank the authors for their rebuttal.

I have taken the time to read through all reviews and rebuttals. On the most part, the response has answered my questions.

Reviewer_d5D9OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how video autoencoders should compress space and time for interactive world models. It introduces CADRE, a family of seven causal representation autoencoders combining frozen DINOv3 features, a learnable bottleneck, and a few-step diffusion decoder trained through progressive self-distillation. The authors train 1.2B-parameter world models with a shared recipe across robot manipulation, human manipulation, driving, and indoor navigation. The main contribution is a substantial empirical design study, supported by an autoencoder training recipe. The results identify useful, domain-dependent compression trade-offs under the tested training and evaluation settings.

Critical strengths

A consequential design question is studied through downstream generation with meaningful controls. The paper evaluates representations through the world models trained on them, rather than assuming that reconstruction quality predicts generation quality. The matched comparison with the MIRA recipe uses the same autoencoder training mixture and budget: at 2×32×32, one-step CADRE achieves robot-manipulation FVD of 105.0 versus 267.1. The matched-token-grid comparison with temporal patching inside the world model adds a useful controlled test. These comparisons support several substantial trends beyond comparisons with unrelated public checkpoints (Sections 4.1 and 5.1; Table 7).

The study makes the quality–responsiveness trade-off concrete. Equation 1 separates waiting for an action block from generation and decoding costs, and Table 4 reports both latency and sustainable throughput. The four-domain evaluation, action-recoverability controls, long robot-manipulation rollouts, and seam analysis make the study informative for practitioners choosing a representation. The paper also discloses important limits of its metrics and experimental scope (Appendices D–K).

Critical weaknesses
  1. The central quality–latency frontier has not been established using the same implemented streaming decoding path. Appendix D states that CADRE, MIRA, and Cosmos decoding times use the cost of decoding a new latent as a proxy for a decoder cache, whereas Wan uses exact streaming reuse. Table 7 evaluates quality using each decoder’s trained window. Thus, timing and quality need not describe the same execution path. This limitation is disclosed, and decoding cost is included, but it remains unclear whether an implemented deployment can jointly achieve the reported quality and latency. In particular, CADRE-2×32×32’s 185 ms latency should retain this qualification. A paired check of quality and timing through the actual streaming path would directly address this concern (PDF pp. 26–28).
  2. The precision and generality of the compression recommendations exceed what some comparisons establish. Table 7 provides approximate episode-level intervals for PSNR, LPIPS, and ARR, but not uncertainty for FID/FVD or variation across training seeds. Large differences are therefore more persuasive than close frontier rankings or the count of improvements exceeding 2%. Moreover, the approximately token-matched 4×24×24 configuration changes the overlapping kernel and channel-to-space layout and is acknowledged as a navigation outlier. Camera motion is also a plausible explanation rather than an isolated cause, since domains differ in actions and scenes. These qualifications limit fine-grained rankings and causal design advice, while leaving the broader empirical findings useful (Section 5.2; Appendices G, I, and L). I lean toward acceptance because the controlled within-family results and matched MIRA comparison provide useful knowledge despite these limitations. Evidence that the streaming quality–latency frontier disappears in practice, or that the main compression trends are unstable, would weaken my recommendation.
Questions for authors

Clarification and targeted validation: Which decoder-cache operations are actually implemented and measured, and which savings are estimated? Does exactly that streaming decoding path preserve the quality reported in Table 7? Robustness: Which decisive close FVD/frontier comparisons remain stable under available uncertainty estimates or limited repeated training? Which compression conclusions remain after excluding the architecturally distinct 4×24×24 configuration? Scope clarification: Can the camera-motion conclusions be consistently framed as an association in these datasets? If a within-domain motion-stratified analysis is already available, does it support the same explanation?

2 comments
AuthorsOpenReview ↗

Response to Reviewer d5D9 (1/2): Streaming decode and latency

We thank the reviewer for a careful review, and are glad they found our autoencoder evaluations through world models, the matched comparisons and the latency analysis informative. In short, timing the decoders exactly as they run in Table 7 makes some of them slower, but CADRE keeps the same advantage over the baselines, and we will update the latency numbers accordingly (W1, Q1). Most of CADRE's wins are also by more than 10%, so our conclusions do not hinge on close calls (W2, Q2), and we will present camera motion as a likely explanation rather than a proven cause (W2, Q3).

W1 / Q1. The central quality–latency frontier has not been established using the same implemented streaming decoding path. [...] Which decoder-cache operations are actually implemented and measured, and which savings are estimated? Does exactly that streaming decoding path preserve the quality reported in Table 7?

The world model streams with a key-value cache, and all timings include it. The decoder has no cache: for each new latent, it decodes the 16-frame window it was trained on and keeps only the new frames. Table 7 was produced this way, whereas Table 4 timed the new latent alone, as a decoder cache would.

Following the reviewer's question, we implemented and tested such a cache in the decoder. On a B200, it is 1.4 to 1.7× faster for τ ≤ 2, but it lowers PSNR by 1.6 to 5 dB on real clips, likely because the cached activations still depend on frames that have left the window, which never happens in training. For τ ≥ 4, it keeps quality but is only 1.1 to 1.3× faster. We therefore keep the windowed decode.

With this windowed decode, on an H100 and in the setting of Table 7, CADRE-2×32×32 reacts in 192 ms instead of 185 ms and runs at 21.7 fps, and CADRE-2×16×16 reaches 20 fps with seven denoising steps instead of ten; with the progressive self-distillation we take from MIRA, MIRA reports stable quality as the number of steps drops in this range of inference steps (Hu et al., 2026). For the other models, we only had B200 GPUs during the rebuttal, so we scaled Table 4 by the slowdown we measured there. Latency grows by 7 to 42 ms for CADRE with one decoding step (17 to 94 ms with two), and by less than 20 ms for MIRA and Cosmos. The frontier holds: recomputed with these latencies, CADRE is still at least as fast and better in 42 of the 48 comparisons of Section 4.2, and all CADRE models with τ ≥ 4 still run above 20 fps. We will update Table 4 and the latency claims accordingly. For reference, we also timed the CADRE models and MIRA directly on one B200, in the setting of Table 7 (ten denoising steps, windowed decode):

Modelfps (1 / 2 decoding steps)Latency (1 / 2 decoding steps)
CADRE-1×48×4823 / 1494 / 119 ms
CADRE-2×32×3255 / 36137 / 155 ms
CADRE-4×24×24109 / 76237 / 253 ms
CADRE-8×16×16183 / 140444 / 457 ms
CADRE-1×16×1617 / 12109 / 134 ms
CADRE-2×16×1640 / 29149 / 168 ms
CADRE-4×16×1689 / 66245 / 260 ms
MIRA recipe66130 ms
AuthorsOpenReview ↗

Response to Reviewer d5D9 (2/2): Margins of the comparisons and role of camera motion

W2 / Q2. Table 7 provides approximate episode-level intervals for PSNR, LPIPS, and ARR, but not uncertainty for FID/FVD or variation across training seeds. [...] Which decisive close FVD/frontier comparisons remain stable under available uncertainty estimates or limited repeated training? Which compression conclusions remain after excluding the architecturally distinct 4×24×24 configuration?

Retraining our roughly 50 world models on 16 H100 several times to get FID.FVD uncertainty is beyond our budget, and bootstrap intervals over samples would not provide meaningful uncertainty, since FID and FVD vary more between training runs than between samples (Dufour et al., 2026). Instead, we looked at the size of the margins, and most are large. With a 10% margin instead of 2%, CADRE still wins 37 of the 48 comparisons, and every FVD win is above 20%. We can flag the five wins below 10%, all on FID or ARR, as close calls. Section 5 also rests on large effects: going from two to eight frames per latent changes robot-manipulation FVD by less than 2% but raises it by 31 to 64% in navigation, and coarser spatial tokens multiply robot-manipulation FVD by 2.4.

No conclusion depends on CADRE-4×24×24. Section 5 leaves it out of its figures, and removing it from Section 4.2 leaves the 42 wins unchanged.

W2 / Q3. Camera motion is also a plausible explanation rather than an isolated cause, since domains differ in actions and scenes. [...] Can the camera-motion conclusions be consistently framed as an association in these datasets? If a within-domain motion-stratified analysis is already available, does it support the same explanation?

We agree, and will use the framing of Section 5.2 throughout: in our datasets, manipulation, where the camera is still, favors temporal compression, and navigation, where it moves, favors spatial compression. To separate camera motion from the other differences between domains, we will add a within-domain test: in each environment, we will compare the cost of eight-frame latents on low- and high-motion clips, measured by optical flow (Appendix I) and, in driving, by ego speed. A first check with the autoencoders shows that motion matters within each environment: eight-frame latents lose more against two-frame latents on the clips that move most, with reconstructions 34% worse in LPIPS on the more moving half of the clips, against 26% on the calmer half.

We hope these answers address the reviewer's concerns, and are happy to clarify any further point.

Reviewer_RTdFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that the autoencoder of an interactive world model fixes two things that are usually ignored: the cost of each generated frame and the time taken for an action to show up on screen (as in Eq. 1). It introduces CADRE, a family of seven causal diffusion autoencoders, differing only in the bottleneck that sets temporal and spatial compression dimensions. The main claim is that CADRE forms most of the latency-quality frontier. Also, the paper adds some findings about: (1) decoder type (2) where temporal compression should live (3) domain-dependent compression (4) what coarse spatial tokens cost in actual motion quality.

Critical strengths

The question is practically important, and evaluating autoencoders through downstream rollouts makes the study useful. The shared world-model recipe and within-family comparisons provide evidence about compression choices. Particularly wanted to highlight the robot-manipulation result (FVD increasing from 39.4 to 95.2 and ARR falling from 0.73 to 0.45 while FID changes little). This in my opinion sufficiently shows why image quality alone is insufficient (Table 7). The detailed timing, long-rollout and counterfactual analyses strengthen the paper.

Critical weaknesses

(1) The latency and quality results have a mismatch. Appendix D times new-latent decoding as the cost of a decoder cache, whereas Table 7 evaluates trained-window decoding. The eight-step real-time configuration also differs from the ten-step evaluation setup.

(2) Finding 1’s controlled comparison is limited only to the discriminator-free MIRA recipe at 2×32×32 which may be insifficient. The decoder capacity is unclear, and no capacity-matched perceptual+adversarial decoder is evaluated. Either scope the conclusion to the tested recipe or provide such a comparison. A longer-step reference and a fixed-encoder self-distillation ablation would also clarify the contribution of the proposed decoder training.

(3) In my opinion, the robustness of the cross-family frontier is not fully established. Training-run variability is unreported, and the episode-level intervals cover PSNR, LPIPS and ARR, but don't cover FID/FVD. While the large improvements may seem persuasive, close frontier comparisons and smaller temporal-patching differences are harder to assess from the current version.

Questions for authors
  • Were the held-out evaluation episodes of ABC, Physical AI AV and RealEstate10K excluded from autoencoder training? All three appear in the training mixture (Table 2).
  • For each baseline, what are the latent channel count, the noise-schedule shift and the tuning range you explored?
  • The latency numbers (In Appendix D - decoding only the new latent as the cost of a decoder cache, eight denoising steps in Sec. 4.2) and the quality numbers (Table 7, decoding over the trained window, ten steps) come from different configurations. Can you report both for one implemented streaming configuration, or argue the settings are equivalent?
  • MIRA Table 23 reports similar generation quality when 2× temporal compression occurs in the codec or patch embedding at spatial factor 32. How should this result, together with CADRE’s own 2×32×32 exception, inform the scope of Finding 2?
2 comments
AuthorsOpenReview ↗

Response to Reviewer RTdF (1/2): Data splits, baseline setup and latency

We thank the reviewer for the careful reading and the precise questions, and we are glad the robot-manipulation result was found convincing. In short: the evaluation episodes are excluded from autoencoder training (Q1); every autoencoder is trained with the same recipe and noise-schedule rule (Q2); latency is now timed with the same decoding as Table 7, and our conclusions hold (W1, Q3); we restate Finding 2 as a result on temporal compression in robot manipulation, consistent with MIRA's result (Q4); we scope Finding 1 to perceptual regression, against which our comparison is controlled (W2); and we clarify how our comparisons handle run-to-run variability (W3).

Q1. Were the held-out evaluation episodes of ABC, Physical AI AV and RealEstate10K excluded from autoencoder training?

Yes. For ABC and RealEstate10K, we hold out 1% of the episodes, and for Physical AI AV we use the official split and evaluate on its validation clips. The autoencoder mixture only includes the training split of each dataset, so neither CADRE nor the MIRA recipe sees an evaluation episode. The released code will contain the split lists for reproducibility.

Q2. For each baseline, what are the latent channel count, the noise-schedule shift and the tuning range you explored?

Off-the-shelf autoencoders keep their native channel count. For the MIRA recipe, we use 64 channels instead of the original 32, so that its latent has the same shape as CADRE-2×32×32 and the comparison of Section 4.1 focuses on the decoder.

AutoencoderChannelsTokens / latent frame
Wan 2.248576
Cosmos162,304
LTX-2128144
DC-AE32144
MIRA recipe64144
CADRE (all members)6466–576

Every world model uses the same recipe from MIRA (Hu et al., 2026), with latents standardized by their own statistics. The only change per autoencoder is the dimension-dependent noise shift of RAE (Zheng et al., 2026), which generalizes the resolution-dependent shift of SD3 (Esser et al., 2024). Since each latent frame is noised independently, mm counts the tokens of one latent frame times its channels, the product of the two columns above, and α=m/4096\alpha = \sqrt{m / 4096}. All flow times, in training and at inference, are warped as t′=t/(α−(α−1)t)t' = t / (\alpha - (\alpha - 1)t), with t=0t = 0 being noise.

We did not further tune α\alpha or any other hyperparameter per autoencoder because one 1.2B-parameter world model takes 100k steps on 16 H100s, and the study trains about 50 of them making a hyperparameter sweep prohibitively expensive. Every autoencoder, CADRE included, receives the same recipe and the same rule-based shift, so none benefits from model-specific tuning. We will state the setup more clearly in the paper.

W1 / Q3. The latency numbers (Appendix D ) and the quality numbers (Table 7) come from different configurations. Can you report both for one implemented streaming configuration, or argue the settings are equivalent?

We have now timed the decode as implemented for streaming, so latency and quality come from the same configuration. For CADRE, MIRA and Cosmos, Table 4 timed only the decode of the newest latent, as a decoder cache would, while Table 7 decodes over the window the decoder was trained on, which is slower.

Measured on an H100 in the setting of Table 7, with ten denoising steps and one decoding step, CADRE-2×32×32 runs at 21.7 fps and reacts in 192 ms, under the 230 ms threshold. CADRE-2×16×16 runs at 20 fps with seven denoising steps, three fewer than in Table 7. Our world models are robust to reducing flow steps because, as done in MIRA, they are trained with progressive self-distillation, which MIRA reports extremely robust in quality in the range of steps we explore (Hu et al., 2026).

For the other models, we only had B200 GPUs during the rebuttal, so their latencies are estimates. On a B200, we measured how much slower the windowed decode is than the decode timed in Table 4, and increased the H100 timings of Table 4 by the same proportion. This adds 7 to 42 ms of latency toƒ the other CADRE models with one decoding step and 17 to 94 ms with two, and less than 20 ms to MIRA and Cosmos.

With these latencies, Table 7 is unchanged, and CADRE is still at least as fast and better by more than 2% in 42 of the 48 comparisons of Section 4.2, MIRA still runs at 22 fps under 230 ms, and every CADRE model with τ≥4\tau \geq 4 still runs above 20 fps, the Figure 5 models at 24 to 52 fps instead of 29 to 58. The revision will update Table 4, the latency figures and the real-time claims, distinguishing measured and estimated timings.

AuthorsOpenReview ↗

Response to Reviewer RTdF (2/2): Scope of Findings 1 and 2, and run-to-run variability

Q4. MIRA Table 23 reports similar generation quality when 2× temporal compression occurs in the codec or patch embedding at spatial factor 32. How should this result, together with CADRE's own 2×32×32 exception, inform the scope of Finding 2?

Finding 2 is about where to compress time (in the autoencoder or in the world model), and we tested it in robot manipulation, with tokens at spatial factor 16. MIRA's Table 23 and our 2×32×32 exception involve different kinds of compression, so we discuss them separately.

MIRA's Rows 1–2 test temporal compression alone, over 2 frames. In our study, the advantage of compressing in the autoencoder is smallest at 2 frames and grows with the number of frames, from 9 FVD points at 2 frames to 28 at 8. MIRA's near-tie at 2 frames, in another domain and at a coarser spatial factor, is therefore consistent with Finding 2, although it also shows that this advantage may not carry over to every setting.

Our 2×32×32 exception, where compressing in the world model gives better FVD, does not contradict Finding 2, which is about temporal compression: it is the only comparison in which the world model also compresses space, as in MIRA's Row 3. It raises a separate question, where to compress space, on which the two works partly disagree. Both find sharper frames when the autoencoder compresses space, but on motion, the world model is better in our robot-manipulation setting, while the autoencoder is better in MIRA's Rocket League. The camera may explain this: with a still camera, as in ours, coarse spatial tokens hurt motion (Finding 4), while with a moving camera, as in MIRA, spatial compression costs less (Finding 3). We will state more explicitly that Finding 2 concerns temporal compression in robot manipulation a spatial downsampling factor of 16.

W2. Finding 1's controlled comparison is limited only to the discriminator-free MIRA recipe at 2×32×32. The decoder capacity is unclear, and no capacity-matched perceptual+adversarial decoder is evaluated. Either scope the conclusion to the tested recipe or provide such a comparison. A longer-step reference and a fixed-encoder self-distillation ablation would also clarify the contribution of the proposed decoder training.

We agree that the controlled comparison covers a single perceptual regression recipe, and we will scope Finding 1 accordingly. We chose the MIRA recipe as a comparison because it is a recent autoencoder built specifically for interactive world modeling. Both autoencoders use the same frozen feature extractor, compression and latent shape, and are trained on the same data with the same budget. MIRA found almost no difference in generation between its L and XL decoders, which suggests that capacity explains little of an FVD gap. We therefore use the Large decoder, at the same depth and width as ours (24 blocks, width 1024), with 411M parameters against 475M.

We also agree that an adversarial decoder would be a valuable comparison. Our baselines give a first, uncontrolled indication: all of them but MIRA train their decoders with an adversarial loss, and with two decoding steps, the best CADRE autoencoder beats all of them on world model rollouts FID and FVD in every environment, except for a tie with Cosmos on FVD in robot manipulation.

As for the decoder training, flow matching without self-distillation sampled in one step predicts the average of the sampled distribution, i.e. is a regression decoder, like MIRA. We report one and two decoding steps, where the second step lowers FID by up to 23%, and do not go further, since each extra step slows down decoding, against the real-time constraint.

W3. Training-run variability is unreported, and the episode-level intervals cover PSNR, LPIPS and ARR, but don't cover FID/FVD.

We agree that our own training-run variability is unmeasured. FID and FVD vary more across training runs than across resampling (Dufour et al., 2026), so proper error bars would need several retrainings of each world model, beyond our budget given there are around 50 of them, trained each for more than a day on 16 H100 GPUs . In the frontier comparisons of Section 4.2, we count a win only when CADRE is better by more than 2%, a tolerance motivated by Dufour et al. (2026), and we report close comparisons, such as CADRE and Cosmos on FVD in robot manipulation, as ties. Most wins are far from this tolerance: with a 10% margin, CADRE still wins 37 of the 48 comparisons, and every FVD win exceeds 20%. Our main conclusions rest on effects that are large or that repeat across independently trained world models: the diffusion decoder lowers FVD by 1.4 to 2.5× in every environment, in navigation all CADRE autoencoders but one beat every baseline on FID and FVD, and for Finding 2, the autoencoder wins on FID, FVD, PSNR and LPIPS at 2, 4 and 8 frames, with a growing gap.

We hope these answers address the reviewer's concerns, and we are happy to clarify any further point.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Author Response

We thank the reviewers for their careful reviews and the AC for their time. We are glad the reviewers saw the question as "practically important" (RTdF), "a consequential design question" (d5D9) and "an interesting and important direction" (hihA), and found the paper "clear and easy to follow" (RTdF, d5D9 and hihA). They valued that "evaluating autoencoders through downstream rollouts makes the study useful" (RTdF), and that the paper evaluates representations through world models "rather than assuming that reconstruction quality predicts generation quality" (d5D9). The "controlled within-family results and matched MIRA comparison provide useful knowledge" (d5D9), with one-step CADRE reaching a robot-manipulation FVD of 105.0 against 267.1. The robot-manipulation result, where FVD rises from 39.4 to 95.2 while FID changes little, "sufficiently shows why image quality alone is insufficient" (RTdF). The study also "makes the quality–responsiveness trade-off concrete" and is "informative for practitioners choosing a representation" (d5D9), and the contrast between manipulation and navigation was found "particularly interesting" (hihA). The reviews raised four shared points.

Latency. We have now timed the decode as implemented for streaming, so latency and quality come from the same configuration. In the setting of Table 7, CADRE-2×32×32 reacts in 192 ms and runs at 21.7 fps. CADRE is still at least as fast and better by more than 2% (FID significance threshold) in 42 of the 48 comparisons of Section 4.2. See Response to Reviewer d5D9 (1/2): Streaming decode and latency.

Robustness. Retraining our roughly 50 world models is beyond our budget, so we looked at the size of the margins, and most are large. With a 10% margin instead of 2%, CADRE still wins 37 of the 48 comparisons, and every FVD win is above 20%. See Response to Reviewer d5D9 (2/2): Margins of the comparisons and role of camera motion.

Scope. We agree that some claims should be narrower. We will state Finding 1 as a result on perceptual regression, Finding 2 as a result on temporal compression in robot manipulation, and camera motion as an association in our datasets. We will also state that the compression trends are measured within the CADRE family. See Response to Reviewer RTdF (2/2): Scope of Findings 1 and 2, and run-to-run variability and Response to Reviewer hihA (2/2): Compression within CADRE, attention and first frame.

Baselines. We use the off-the-shelf autoencoders as released, as a practitioner would. The text-to-video autoencoders release their weights but not their training data, and Wan 2.2 and LTX-2 do not release their training code either. They therefore cannot be retrained on the same data or at another compression. This is why we propose CADRE as an end-to-end open baseline, trained only on published data, with code and models to be released, that anyone can retrain at the compression their world model needs. The comparison that controls data and budget is the MIRA recipe, retrained on our data, and the evaluation episodes are excluded from autoencoder training. See Response to Reviewer hihA (1/2): Baselines and streaming decoder and Response to Reviewer RTdF (1/2): Data splits, baseline setup and latency.

We will update the latency results and state these scopes and limitations in the revision. We hope these answers address the reviewers' concerns, and are happy to clarify any further point.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.