acceptodds
Under review as a conference paper at ICLR 2027

Video World Models Need Predictable Latents

Abstract

Video world models trained on large-scale robot data promise a safe and scalable approach for learning and evaluating policies that act in the physical world. However, they still struggle to generate realistic long-horizon rollouts, generalize out of distribution, and they remain computationally demanding at inference time. The dominant approach, making predictions in the latent space of a reconstruction-trained autoencoder, does not incentivize a latent geometry that facilitates dynamics modeling. For this reason, we investigate whether encouraging a meaningful temporal organization in latent space can benefit downstream world model performance, using a series of controlled experiments. We find that temporal representation learning objectives can improve downstream performance of the world model at matched reconstruction quality. However, the best-performing objectives need over 20x more autoencoder updates to reach the same reconstruction performance. To address this issue, we characterize the trade-off between information retention and useful geometric organization in latent space training, and we propose a simple modification to the architecture of the decoder that improves it. We find that pairing this architecture with temporal objectives for autoencoder finetuning improves downstream performance of the world model at matched training budget. Compared to a reconstruction baseline, the gains include up to 5x faster inference, 8x faster training, 24% MSE decrease out of distribution and 18% MSE decrease in long-horizon extrapolation. This also translates into better agreement of the world model with real-world policy evaluation, with up to a 0.46 increase in Pearson correlation using 2-step inference. Common to the best temporal objectives is a large increase in one-step predictability of the latents, lower intrinsic dimensionality, and weaker high frequency components. Overall, our study indicates that temporal predictability is an important design criterion for the latent space of video world models, and provides concrete guidance on how to incorporate it into autoencoder training.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
world models, self-supervised learning, representation learning, generative models, video data

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_opdBOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how autoencoder representations affect video world models. The authors find that reconstruction-only training can produce latents where dynamics are difficult to predict. They combine temporal learning objectives with a decoder conditioned on a past frame to improve latent predictability while preserving reconstruction ability. Experiments on robotic manipulation data show improved OOD prediction and long-horizon rollouts, fewer sampling steps at comparable quality, and better agreement between simulated and real-world policy evaluations.

Critical strengths

The paper is well-written and their idea is easy to follow. The paper provides a controlled study of temporal objectives for autoencoder training. Comparisons at matched reconstruction quality, together with controls for decoder conditioning, show that these objectives improve downstream world-model prediction beyond reconstruction quality alone. This provides useful empirical guidance for learning latent representations that better support dynamics modeling.

Critical weaknesses
  1. Limited validation on modern video VAEs. The method is evaluated on framewise image encoders, without testing temporally compressed video VAEs such as Wan-VAE. These models already learn spatiotemporal structure and reduce downstream sequence length. Consequently, it remains unclear whether the proposed objectives provide additional benefits for modern video tokenizers. After response, I still hold my concern for Video VAE. A temporally compressed video VAE can represent the same rollout with fewer tokens than a framewise encoder, reducing latency in robotics applications (which is the main data domain studied in this paper). Without a direct comparison, it remains unclear whether the proposed method offers better performance. I therefore maintain my current score.
  2. Insufficient evidence for end-to-end efficiency. The reported speedups measure world-model training updates and sampling steps, without establishing total savings after accounting for autoencoder fine-tuning and conditional decoding. Without measured training costs and full-rollout latency, the practical efficiency advantage remains unclear.
Questions for authors
  1. I think some video VAEs should be involved to prove the effectiveness of proposed method. Can the authors evaluate their objectives on such a tokenizer while matching reconstruction quality, rollout duration, and control frequency?
  2. Beyond Tab. 6, can the authors complete the matched-budget comparison with temporal objectives on the ordinary decoder, and report full-rollout quality versus measured latency across sampling budgets?
2 comments
AuthorsOpenReview ↗

Response to Reviewer opdB [1/2]

We thank the reviewer for recognizing the clarity of the paper and the value of our controlled study, which gives useful empirical guidance for learning latents that support dynamics modeling. We refer the reviewer to the general response for a summary of contributions and changes to the manuscript. Here, we address their specific concerns.

(1) End-to-end efficiency

The reviewer asks whether the speed-ups hold after accounting for autoencoder fine-tuning and the past-conditioned decoder, with measured training cost and rollout latency.

We provide additional end-to-end measurements of training time and rollout latency to show that the speed-ups hold once autoencoder fine-tuning and the past-conditioned decoder are accounted for (details in Appendix H, new). The reported numbers of world model updates and sampling steps remain useful implementation-independent measures of efficiency.

For training, we fine-tuned the FLUX VAE with latent prediction and the past-conditioned decoder, as in Section 5.3, for only five epochs (5,540 updates, 3.0 hours on one H200 GPU), trained the same world model on its latents, and measured the total training time, fine-tuning included, after which its validation metrics first reach the baseline's final values. For inference, we timed complete nine-frame rollouts on one H200 GPU at batch 1, from the first frame and the actions to the decoded frames, with 1 to 10 sampling steps.

Tables 17 and 18 and Figure 29 give the training speed and the rollout time against the baseline, which takes 46.7 hours of total training and 2.58 seconds per rollout with 10 steps:

MetricTotal training time to reach the baseline's final value, five-epoch latent predictionRollout time to match the baseline with 10 steps, inverse dynamicsRollout time to match the baseline with 10 steps, latent prediction
LPIPS ↓5.8 h (8.0x less)0.43 s, 1 step (6.1x less)0.67 s, 2 steps (3.8x less)
SSIM ↑5.6 h (8.3x less)0.43 s, 1 step (6.1x less)0.42 s, 1 step (6.1x less)
Pixel MSE ↓14.9 h (3.1x less)0.69 s, 2 steps (3.8x less)1.69 s, 6 steps (1.5x less)

Against the pretrained autoencoder, which needs no fine-tuning and takes 15.4 hours, the five-epoch pipeline reaches the final LPIPS and SSIM 3.6 and 3.9 times sooner. After full training, the five-epoch pipeline is better than the baseline on all nine out-of-distribution, long-horizon and two-step metrics (Table 16). With 59 frames generated in chunks of 8, as in the long-rollout evaluation, the saving in rollout time is smaller: 1.5 times for inverse dynamics with 1 step and 1.4 times for latent prediction with 2 steps (Appendix H).

So the savings in world-model updates and sampling steps carry over to wall-clock time: the five-epoch fine-tuning costs 20% of one world-model training run, but is enough to recover most of the gains of the full 80k-step fine-tuning; and the past-conditioned decoder adds only about 0.12 seconds to a nine-frame rollout, small next to the time saved by fewer sampling steps.

AuthorsOpenReview ↗

Response to Reviewer opdB [2/2]

(2) Ordinary decoder at the matched budget

The reviewer asks us to complete the matched-budget comparison with the temporal objectives on the ordinary decoder.

We provide additional experiments for the matched-budget comparison with both temporal objectives on the ordinary decoder (full results in Tables 6 and 8; the rows of the submitted Table 6 are now in Tables 2 and 6).

We fine-tuned each autoencoder for the same 80,000 updates as the other pipelines, trained the same world model on its latents, and evaluated it in the same conditions, policy evaluation included. For latent prediction:

Decoder, objectiveLPIPS ↓, out of distributionLPIPS ↓, long horizonLPIPS ↓, 2 stepsPearson r with real-world policy success ↑, 2 steps
Ordinary, reconstruction (baseline)0.3020.3940.2320.21
Past-conditioned, reconstruction0.2700.3660.1980.15
Ordinary, latent prediction0.2870.3420.1910.62
Past-conditioned, latent prediction0.2570.3310.1770.67

Both components contribute. The decoder alone improves the offline metrics but not policy evaluation. The objective alone improves both, and carries the gain in policy evaluation. Together, they are best on all four metrics. The same picture mostly holds for SSIM and pixel MSE, and for inverse dynamics (Tables 6 and 8).

(3) Video autoencoders such as the Wan VAE

The reviewer argues that video autoencoders should also be considered.

The main contribution of our study is a careful understanding of the role of temporal structure in autoencoders for video world models. It rests on controlled comparisons and a characterization of the main trade-offs, which provide valuable insights and practical guidance. Our goal is therefore not to build the best possible video world model.

We agree that autoencoders that compress along the time axis are an important extension for future work. However, in these autoencoders, time mostly serves as an additional compression axis and enters the training objective, if at all, through pixel-space terms (Section 2), not through a predictive task in latent space like ours. We therefore believe that they would not fundamentally change the sequence-modeling task at hand, and would offer limited new insight for our goals. In fact, single frames are already a compressed, though not learned, representation of an extended temporal chunk of the underlying continuous video stream. The relationships between frame representations that we study are therefore not very different from those between the chunk representations of a video autoencoder.

Reviewer_QHHPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the training objective of the autoencoder that defines the latent space in which a video world model is trained. Several temporal objectives are compared, and it is shown that training for a better-structured latent space improves the quality of the world model’s predictions. In addition, a past-observation-conditioned decoder is introduced, which further improves predictions. The experimental section is strong: world models are compared both with visual metrics and as policy evaluators, where the correlation with real-world policy performance is measured. Qualitatively, the improvement from the reconstruction-only baseline to the proposed world model is significant.

Critical strengths
  • The objective is clear and the problem is well defined.
  • Training efficiency of the world model (latent predictor) improves several-fold with the better autoencoder.
  • The experimental section is strong: predictions are compared by visual quality as well as by correlation with real-world policy performance.
Critical weaknesses
  • Generality beyond Bridge. Bridge is becoming saturated nowadays, and it would be good to show that the same improvement can be achieved on other datasets, e.g. DROID or ABC-130k.
Questions for authors

Can you show generality beyond Bridge, as described in the weaknesses?

1 comment
AuthorsOpenReview ↗

Response to Reviewer QHHP

We thank the reviewer for recognizing the thoroughness of our experiments and the strength of our empirical results. We refer the reviewer to the general response for a summary of contributions and changes to the manuscript. Here, we address their question about the generality of our findings.

(1) Generality beyond Bridge V2

The reviewer asks whether our findings are specific to the Bridge V2 dataset, or whether they hold more broadly.

We provide additional experiment results on nine Open X-Embodiment datasets to show the generality of our findings beyond Bridge V2 (details in Appendix G.10, new).

We trained five pairs of world models of different sizes (widths 128 to 1,024) on a mixture of nine datasets recorded with different robots, including Bridge V2, RT-1, and seven smaller datasets with tasks such as cable routing and door opening. Each pair compares standard reconstruction fine-tuning with a weak version (Figure 25) of our recipe that does not use the past-conditioned decoder and uses latent prediction with only a small weight (0.2). We then measured the prediction error of the trained world models with three metrics (LPIPS, SSIM, pixel MSE), under three evaluation conditions (in-distribution, two-step, long-horizon), and on each of Bridge V2, RT-1, and a pool from the other seven datasets.

As shown in Figure 24, the improvements due to the temporal objective on RT-1 and on the pool of other datasets are comparable to those on Bridge V2. Per metric, the table gives the improvement from latent prediction averaged over the 15 combinations of size and condition, and the Pearson correlation of the per-combination improvements with those on Bridge V2:

MetricImprovement ↑, Bridge V2Improvement ↑, RT-1Improvement ↑, other sevenPearson r with Bridge V2, RT-1Pearson r with Bridge V2, other seven
LPIPS9.3%8.2%7.6%0.870.86
SSIM1.9%1.5%1.1%0.870.78
Pixel MSE5.7%3.6%3.4%0.330.19

The gains in LPIPS and SSIM correlate well with those on Bridge V2. Pixel MSE tracks less closely (this experiment has only one training seed), but on average latent prediction improves all three metrics on every test set.

Our synthetic study (Section 4 and Figure 5, new) shows the mechanism behind the gains: when observations mix predictable dynamics with unpredictable detail, reconstruction alone yields latents that are hard to predict; adding a temporal predictive objective lets the world model predict the signal far into the future, while reconstructing nearly as well. Together, these results are a strong indication that the benefit of temporal predictive objectives is not specific to Bridge V2.

For the camera-ready version of the paper, we will repeat this experiment using our final proposed recipe, which would likely lead to larger gains. We also remark that the closest peer-reviewed study to ours [1] focuses exclusively on Bridge V2.

References

[1] Nilaksh, S. Jha, A. Zholus, S. Chandar. Reconstruction or Semantics? What Makes a Latent Space Useful for Robotic World Models. NeurIPS 2026.

Reviewer_nV2POpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tries to solve an important question in video world models for robotics: while video world models offer high reconstruction fidelity, it comes at high inference cost. The argument is to build a latent space that is easily predictable without sacrificing the ability to accurately reconstruct the observations, and the key way of enabling this is through incentivizing latent geometry. Through controlled study, the authors establish interesting conclusions with temporal representation-learning objectives. It first compares objectives at matched reconstruction quality, then introduces a past-conditioned decoder to alleviate the tension between reconstruction and temporal structure, and evaluates the resulting recipe at matched training-update budgets.

Critical strengths
  1. Controlled experiment and careful analysis. The paper provides a useful and well-controlled empirical study with consistent improvements in several practically useful settings. It provides clear evidence of world models can benefit from balancing temporal predictability with reconstruction and task-information retention. It is also very nice that the authors extend experiments beyond visual metrics, showcasing usefulness in policy evaluation.
Critical weaknesses
  1. I do see the usefulness of the proposed method, but I doubt the reasoning trace attributing it to temporal predictability. Temporal objectives simultaneously change latent predictability, effective dimensionality, and spatial smoothness. A reconstruction-matched experiment using spatial regularization or dimensionality reduction, without temporal supervision, would help distinguish temporal structure from more general improvements in latent diffusability.

  2. Writing is very difficult to follow in places, especially due to a lack of methods section. I would suggest the authors refine the structure of the sections for better readability, with section names indicating the research questions answered in the section rather than "Experiments on xxxx".

Questions for authors
  1. I think that the paper's title is a bit overclaiming. While the experiment sections cover a set of world models from different families, I would like to see how broader claims about video world models are justified through experiments or theoretical results. Otherwise, mildly softening the claim would help.

  2. Could the authors provide clearer computational interpretations on efficiency claims? More specifically, I would like to see comparisions that directly establishes an equivalent end-to-end wall-clock speedup. For example, Autoencoder fine-tuning is a substantial cost, and the past-conditioned decoder introduces additional computation.

3 comments
AuthorsOpenReview ↗

Response to Reviewer nV2P [1/2]

We thank the reviewer for recognizing our controlled experiments and careful analysis, and the consistent improvements in practically useful settings. We refer the reviewer to the general response for a summary of contributions and changes to the manuscript. Here, we address their specific concerns.

(1) End-to-end efficiency

The reviewer asks for an end-to-end wall-clock comparison that counts the cost of autoencoder fine-tuning and of the past-conditioned decoder.

We provide additional end-to-end measurements of training time and rollout latency to show that the speed-ups hold once autoencoder fine-tuning and the past-conditioned decoder are accounted for (details in Appendix H, new). The reported numbers of world model updates and sampling steps remain useful implementation-independent measures of efficiency.

For training, we fine-tuned the FLUX VAE with latent prediction and the past-conditioned decoder, as in Section 5.3, for only five epochs (5,540 updates, 3.0 hours on one H200 GPU), trained the same world model on its latents, and measured the total training time, fine-tuning included, after which its validation metrics first reach the baseline's final values. For inference, we timed complete nine-frame rollouts on one H200 GPU at batch 1, from the first frame and the actions to the decoded frames, with 1 to 10 sampling steps.

Tables 17 and 18 and Figure 29 give the training speed and the rollout time against the baseline, which takes 46.7 hours of total training and 2.58 seconds per rollout with 10 steps:

MetricTotal training time to reach the baseline's final value, five-epoch latent predictionRollout time to match the baseline with 10 steps, inverse dynamicsRollout time to match the baseline with 10 steps, latent prediction
LPIPS ↓5.8 h (8.0x less)0.43 s, 1 step (6.1x less)0.67 s, 2 steps (3.8x less)
SSIM ↑5.6 h (8.3x less)0.43 s, 1 step (6.1x less)0.42 s, 1 step (6.1x less)
Pixel MSE ↓14.9 h (3.1x less)0.69 s, 2 steps (3.8x less)1.69 s, 6 steps (1.5x less)

Against the pretrained autoencoder, which needs no fine-tuning and takes 15.4 hours, the five-epoch pipeline reaches the final LPIPS and SSIM 3.6 and 3.9 times sooner. After full training, the five-epoch pipeline is better than the baseline on all nine out-of-distribution, long-horizon and two-step metrics (Table 16). With 59 frames generated in chunks of 8, as in the long-rollout evaluation, the saving in rollout time is smaller: 1.5 times for inverse dynamics with 1 step and 1.4 times for latent prediction with 2 steps (Appendix H).

So the savings in world-model updates and sampling steps carry over to wall-clock time: the five-epoch fine-tuning costs 20% of one world-model training run, but is enough to recover most of the gains of the full 80k-step fine-tuning; and the past-conditioned decoder adds only about 0.12 seconds to a nine-frame rollout, small next to the time saved by fewer sampling steps.

AuthorsOpenReview ↗

Response to Reviewer nV2P [2/2]

(2) Temporal structure or general diffusability

The reviewer asks whether the gains come from temporal structure or from lower dimension and spatial smoothness, and suggests a reconstruction-matched control with spatial regularization and no temporal supervision.

We provide the requested reconstruction-matched control, showing that the gains crucially depend on the temporal predictive objectives (details in Section 5.3 and Appendix G.9, new).

We fine-tuned the autoencoder with the scale-equivariance regularizer of Skorokhodov et al. [1], which targets diffusability, using its authors' released implementation and settings, at equal reconstruction and at equal training budget, and trained the same world model on its latents.

As shown in Tables 12 and 13, the regularizer lowers the dimension and smooths the latents, as latent prediction does. Yet its world models are worse than those of every temporal objective in all four evaluation conditions, and worse than reconstruction alone in nearly all comparisons. At equal reconstruction, with the principal components for 95% of the variance (lower means lower dimension) and the high-frequency share of the spatial spectrum (lower means smoother):

ObjectiveComponents for 95% varianceHigh-frequency shareLPIPS ↓, out of distributionLPIPS ↓, 2 stepsLPIPS ↓, long horizon
Reconstruction3880.0730.3080.2140.287
Latent prediction1770.0610.2980.1600.262
Spatial regularizer1110.0210.3750.2200.325

In policy evaluation at equal budget, with two sampling steps, its correlation with real-world success is 0.13, against 0.15 for reconstruction and 0.67 for latent prediction.

So, while the best performing temporal objectives, i.e. those that define their predictive task in latent space rather than pixel space, exhibit smoother and lower dimensional latents, these factors alone are not responsible for the gains, which require the temporal predictive objective.

(3) Readability, structure and title

The reviewer finds the paper hard to follow without a methods section, and suggests section titles that name the research questions. They also find the title a bit too assertive and suggest softening it.

We have revised the structure and adopted more informative titles for paragraphs and sections. The revised Background section (Section 3) introduces the notation and defines the latent video world model, the temporal objectives with the combined loss, and the diffusion world model, effectively serving as a methods section. We have opted for this structure because our contribution is not a completely new method, but a careful exploration and analysis of existing ideas in a new light, showing that the perceived gap between video world models built on reconstruction-trained latents and world models that learn predictable latents for planning can be fruitfully bridged.

This is supported by three results. In a synthetic environment, the combination of reconstruction and a temporal predictive objective gives latents that support both reconstruction and accurate dynamics modeling, while each individual objective fails at one of the two (Section 4). On Bridge V2, we characterize a trade-off between reconstruction and temporal predictability, and we show that both can be improved at the same time through a simple architectural modification of the decoder, achieving the best of both worlds (Section 5.2). At equal training budget, the resulting recipe gives up to 6x lower inference latency and up to 8x faster world-model training at matched perceptual quality, 15–24% lower errors out of distribution and over long horizons, and up to 0.45 higher Pearson correlation with real-world policy evaluation (Section 5.3).

Accordingly, our title is phrased to take a deliberately strong position and invite discussion in the research community. Since the abstract qualifies our claims, we believe that the reader will not be misled. Following the control of point (2), we decided to change one word of the title for the camera-ready version, from "Predictable" to "Predictive", to more sharply attribute the source of the gains.

References

[1] I. Skorokhodov, S. Girish, B. Hu, W. Menapace, Y. Li, R. Abdal, S. Tulyakov, A. Siarohin. Improving the Diffusability of Autoencoders. ICML 2025.

I thank the authors for their efforts trying to address my concerns. I find the end-to-end efficiency comparison greatly addressed my efficiency concerns, however I still maintain skeptical on the limited scope without comparison to modern encoders such as Video VAEs and the its limited applications. I will maintain my current score with increased confidence.

Reviewer_RNG6OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Video world models are typically trained to predict in the latent space of a reconstruction-optimized autoencoder — an objective that does not incentivize latent geometry that facilitates dynamics modeling. This paper investigates whether adding temporal representation learning objectives to autoencoder fine-tuning improves downstream world model performance. The key finding is that temporal objectives do improve performance at matched reconstruction quality, but the best-performing ones require over 20× more autoencoder updates to reach the same reconstruction performance. To address this cost, the paper proposes a past-conditioned decoder — a simple architectural modification that cross-attends to the previous frame when decoding the current one — which shifts the reconstruction-predictability trade-off frontier, achieving better performance on both dimensions simultaneously. The authors report significant improvement across a number of metrics.

Critical strengths
  1. Very strong empirical results.

  2. Arguably the proposed approach is pretty simple (but with strong results)

Critical weaknesses
  1. The experiments are limited by the single robotics domain of Bridge V2, which may be limiting. It is not very clear to me the results would generalize to a broader set of robotics tasks.
Questions for authors
  1. I would consider at least one other robotics domain. Can the authors showcase similarly strong results on other domains?
1 comment
AuthorsOpenReview ↗

Response to Reviewer RNG6

We thank the reviewer for recognizing the strength of our empirical results and the simplicity of our approach. We refer the reviewer to the general response for a summary of contributions and changes to the manuscript. Here, we address their question about the generality of our findings.

(1) Other robotics domains

The reviewer asks whether similarly strong results hold in at least one other robotics domain.

We provide additional experiment results on nine Open X-Embodiment datasets to show the generality of our findings beyond Bridge V2 (details in Appendix G.10, new).

We trained five pairs of world models of different sizes (widths 128 to 1,024) on a mixture of nine datasets recorded with different robots, including Bridge V2, RT-1, and seven smaller datasets with tasks such as cable routing and door opening. Each pair compares standard reconstruction fine-tuning with a weak version (Figure 25) of our recipe that does not use the past-conditioned decoder and uses latent prediction with only a small weight (0.2). We then measured the prediction error of the trained world models with three metrics (LPIPS, SSIM, pixel MSE), under three evaluation conditions (in-distribution, two-step, long-horizon), and on each of Bridge V2, RT-1, and a pool from the other seven datasets.

As shown in Figure 24, the improvements due to the temporal objective on RT-1 and on the pool of other datasets are comparable to those on Bridge V2. Per metric, the table gives the improvement from latent prediction averaged over the 15 combinations of size and condition, and the Pearson correlation of the per-combination improvements with those on Bridge V2:

MetricImprovement ↑, Bridge V2Improvement ↑, RT-1Improvement ↑, other sevenPearson r with Bridge V2, RT-1Pearson r with Bridge V2, other seven
LPIPS9.3%8.2%7.6%0.870.86
SSIM1.9%1.5%1.1%0.870.78
Pixel MSE5.7%3.6%3.4%0.330.19

The gains in LPIPS and SSIM correlate well with those on Bridge V2. Pixel MSE tracks less closely (this experiment has only one training seed), but on average latent prediction improves all three metrics on every test set.

Our synthetic study (Section 4 and Figure 5, new) shows the mechanism behind the gains: when observations mix predictable dynamics with unpredictable detail, reconstruction alone yields latents that are hard to predict; adding a temporal predictive objective lets the world model predict the signal far into the future, while reconstructing nearly as well. Together, these results are a strong indication that the benefit of temporal predictive objectives is not specific to Bridge V2.

For the camera-ready version of the paper, we will repeat this experiment using our final proposed recipe, which would likely lead to larger gains. We also remark that the closest peer-reviewed study to ours [1] focuses exclusively on Bridge V2.

References

[1] Nilaksh, S. Jha, A. Zholus, S. Chandar. Reconstruction or Semantics? What Makes a Latent Space Useful for Robotic World Models. NeurIPS 2026.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers [1/2]

We thank the reviewers for recognizing our careful controlled study (opdB, nV2P), the strength of the experiments and results (QHHP, RNG6), the thoroughness of the evaluations (QHHP, nV2P), the simplicity of the approach (RNG6), and the clarity of the exposition (opdB).

[1] Revisiting the key contributions. The paper presents a controlled study of temporal predictive objectives and the trade-off between structure and reconstruction in autoencoder training, and how they affect latent video world models. Its key contributions are:

  1. Temporal structure is useful for world models. On Bridge V2, temporal objectives improve the world model at equal reconstruction error. The best ones define the predictive task in latent space and give smoother, lower-dimensional latents; however, a spatial regularizer that induces the same geometric changes performs much worse: the temporal predictive character of the objective is the key element.
  2. The trade-off between information and temporal structure. We find a tension between minimizing a temporal predictive objective and retaining the information needed for reconstruction, and we trace the frontier between the two by varying the objective's weight. Conditioning the decoder on a past frame gives a Pareto improvement of this frontier, which gives better reconstructions with more predictable latents than the ordinary decoder.
  3. A practical recipe. With temporal objectives and the past-conditioned decoder, at equal training budget, world models reach the reconstruction-only baseline's perceptual quality with up to 8x faster training and up to 6x lower inference latency, have 15–24% lower errors out of distribution and over long horizons, and agree better with real-world policy evaluation (up to 0.45 higher Pearson correlation). The key findings are qualitatively robust to changes in the initial encoder, world model size, dataset, and other factors.

Together, these results support the view that two camps in the world-model literature, generative video world models and latent world models for planning, can be fruitfully bridged to get the best of both worlds: stable training without collapse, a latent structure that makes prediction, training and sampling cheaper and more effective, and accurate reconstructions of observations as an interface with vision-language model (VLM) judges and policies to enable policy evaluation, as we show, and potentially policy improvement and planning. This view is clearest in our synthetic environment, where combining the two objectives gives the best of both worlds.

AuthorsOpenReview ↗

General response to all reviewers [2/2]

[2] Key new evidence and changes to the PDF.

Two questions were brought up by more than one reviewer, and we introduced new evidence to answer both. The first is whether our findings hold beyond Bridge V2 (QHHP, RNG6): we answer this positively with new experiments on nine Open X-Embodiment datasets (Appendix G.10). The second is whether the efficiency gains hold end to end (opdB, nV2P): we answer this positively, showing that a short fine-tuning run with our recipe maintains most of the improvements, and measuring large gains in the total training time and in the latency of complete rollouts (Appendix H).

Additionally, to answer a question of reviewer nV2P, we introduced new experiments showing that the temporal predictive formulation of the objectives, and not only the geometric change they induce, is responsible for the performance gains (Appendix G.9), and sharpened our presentation to make this aspect clearer (including the title, where 'predictable' will become 'predictive' in the camera-ready version).

We also complete the matched-budget comparison (opdB; Tables 6 and 8) and we extend the synthetic environment analysis with measurements of long-horizon prediction errors to better support the best-of-both-worlds interpretation (Figures 2 and 5).

Finally, we spared no effort in revising the presentation throughout the paper, including its structure, figures and tables, to convey the key ideas and the supporting evidence more crisply (nV2P). Some figure and table numbers have changed because figures and tables were added, moved or removed, and some reported values differ slightly from the submission, mainly because the tables now give absolute means with standard errors instead of per-seed relative changes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.