- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper presents Latent-Foresight, a framework that jointly trains a feature autoencoder and a flow-based predictor to address the limitations of separately trained tokenizers and predictors in VFM-based world models. Building upon existing approaches such as VFMF, the method introduces several training strategies to stabilize joint optimization and prevent latent collapse. Experimental comparisons with DINO-Foresight, VFMF, and internal two-stage baselines demonstrate certain benefits of joint training. The ablation studies are comprehensive, and the paper is generally well organized and clearly presented. However, the motivation relies on an overly broad characterization of existing methods, and the methodological novelty and evidence supporting the central claims remain insufficient.
Critical strengths
-
The paper identifies a reasonable limitation of approaches such as VFMF, where the tokenizer and predictor are trained separately. A representation optimized primarily for feature reconstruction may not necessarily be optimal for modeling temporal dynamics. Exploring joint optimization is therefore a meaningful research direction.
-
The comparison between the internal two-stage AE baseline and the proposed end-to-end framework provides evidence that joint optimization can improve forecasting performance while maintaining reconstruction quality.
-
The ablation studies in Tables 5–7 systematically investigate latent normalization, stop-gradient strategies, auxiliary reconstruction, latent dimensionality, and noise schedules. These experiments provide useful insights into the training framework. The overall presentation is also clear and well organized.
Critical weaknesses
-
Ambiguous and insufficiently justified motivation. In Lines 53–58, the paper characterizes existing VFM-based forecasting approaches as relying on a two-stage training paradigm. However, methods such as DINO-Foresight, FlowWM, and VGGT-World do not involve separately training a tokenizer and then freezing it to train a predictor. These approaches adopt different feature representations and forecasting strategies, which should not be broadly categorized under the same two-stage training limitation. While this concern is relevant to VFMF, its generalization to existing VFM-based world models is insufficiently justified. As a result, the stated limitations of prior methods are not sufficiently precise, weakening the central motivation.
-
Limited novelty over VFMF and insufficient evidence for the benefits of joint training. The proposed framework closely follows VFMF, which already combines learned VFM feature compression, autoregressive flow matching, and feature reconstruction. The main distinction appears to be the joint optimization of the tokenizer and predictor, supported by additional training stabilization techniques. While this is a reasonable extension, its methodological novelty seems limited. Moreover, the direct comparison with VFMF involves substantially different latent dimensionalities and tokenizer architectures, making it difficult to attribute the reported improvements to joint training. Although the internal two-stage AE baseline partially addresses this concern, the improvements are relatively modest in some settings. Consequently, the experimental evidence does not fully justify the claimed significance of the proposed end-to-end paradigm. Furthermore, recent approaches such as VGGT-World demonstrate effective forecasting directly in frozen foundation-model feature spaces, challenging the claimed necessity of a jointly learned latent representation.
Questions for authors
- Limited evidence for improved latent predictability. The claim that the learned representation has more predictable temporal dynamics is primarily supported by downstream perception performance. Direct analyses of latent-space predictability, temporal consistency, or optimization characteristics are missing. Consequently, the experimental results do not fully establish the central representation-learning hypothesis.
4 comments
Direct evidence of latent predictability
We thank the reviewer for this suggestion. To directly characterize the latent spaces, we perform a training-free analysis of ground-truth video latent dynamics, independently of any learned predictor. A closed-form ridge regression, fitted on a disjoint set of sequences, maps the latents of the last 1 or 3 frames to the latent 3 or 9 video frames ahead (0.18 s or 0.53 s). We report Ridge R² (variance-weighted) and Identity R², the score obtained by simply copying the current latent (). Both tokenizers have the same bottleneck dimension (), so the linear map has identical capacity in both latent spaces.
To separate the effect of the encoder from that of the decoder, we include the control from Table 2 (Two-Stage w/ Aux Recon): the Two-Stage encoder is kept frozen, and the decoder is trained on predicted latents with the auxiliary reconstruction loss.
Table 13: Training-free analysis of latent dynamics. Cityscapes val; ridge fitted on 300 train sequences. Recon. Cos is measured in the normalized feature space.
| Res. | Method | Recon. Cos↑ | Identity R²↑ 3 fr. | Identity R²↑ 9 fr. | Ridge R²↑, 1 ctx 3 fr. | Ridge R²↑, 1 ctx 9 fr. | Ridge R²↑, 3 ctx 3 fr. | Ridge R²↑, 3 ctx 9 fr. |
|---|---|---|---|---|---|---|---|---|
| 224×448 | Two-Stage with AE | 0.927 | 0.294 | −0.004 | 0.471 | 0.320 | 0.493 | 0.345 |
| 224×448 | Two-Stage w/ Aux Recon (frozen encoder) | 0.923 | 0.294 | −0.004 | 0.471 | 0.320 | 0.493 | 0.345 |
| 224×448 | End-to-End (Ours) | 0.928 | 0.384 | 0.123 | 0.539 | 0.400 | 0.558 | 0.423 |
| 448×896 | Two-Stage with AE | 0.937 | 0.140 | −0.111 | 0.393 | 0.280 | 0.419 | 0.307 |
| 448×896 | End-to-End (Ours) | 0.938 | 0.316 | 0.110 | 0.517 | 0.413 | 0.537 | 0.437 |
The results reveal three points:
- Comparable information retention: all latent spaces reconstruct DINOv2 features equally well, so the differences below are not explained by discarded information.
- End-to-end training makes the latent more persistent and more linearly predictable in every setting (both resolutions, both horizons, one or three context frames), and the advantage grows with the horizon (Ridge R² gap with one context frame: +0.068 → +0.080 at 224×448 and +0.124 → +0.133 at 448×896). The w/ Aux Recon control shares the Two-Stage encoder, so its latent-space metrics are identical to Two-Stage by construction: this difference comes from training the encoder with the forecaster, not from the decoder or the auxiliary loss.
- The more predictable latent translates into better forecasts. Compared with the control (same predictor, same decoder training, equal reconstruction: 68.08/66.94 vs. 68.29/67.26 ALL/MO), End-to-End improves short-term and mid-term movable-object segmentation from 62.63 to 64.10 and from 53.12 to 53.86 (Table 2).
Visualization. Figure 3 and Appendix Figures 6–8 show PCA projections of the latents of both tokenizers over time. The end-to-end latents are smoother over time and group objects such as vehicles and pedestrians into compact regions that remain stable across frames, whereas the two-stage latents are noisier with weaker object separation. This is a qualitative illustration, consistent with the quantitative results above.
Together, these analyses show that, at equal reconstruction quality, joint training shapes a latent space that is more temporally coherent and more predictable, and that this improves downstream forecasting beyond what decoder adaptation alone provides.
Thank you for your reply. However, my main concern regarding the methodological novelty remains unresolved. While I acknowledge that naive joint training can lead to latent collapse and that the proposed strategy addresses this issue, the method largely follows the overall framework of VFMF. The main modifications, including normalization, stop-gradient, and an auxiliary reconstruction loss, appear to be primarily training optimizations rather than substantial methodological innovations. Therefore, my primary concern about novelty remains.
Dear Reviewer Ndpg,
Thank you for your follow-up and for acknowledging that naive joint training can lead to latent collapse and that our approach addresses it. We understand that the remaining concern is methodological novelty, and hope our previous responses have addressed the earlier points (prior work characterization, comparisons with direct forecasting, and latent predictability). We would be glad to discuss any that remain.
We agree with your initial assessment that "a representation optimized primarily for feature reconstruction may not necessarily be optimal for modeling temporal dynamics" and that joint optimization is "a meaningful research direction". Our contribution is precisely to make this joint learning possible; the novelty lies not in individual training components, but in what they enable:
- The forecasting objective shapes the representation. In VFMF, the latent is determined by reconstruction alone and frozen before forecasting. Adding auxiliary reconstruction to a frozen encoder brings only marginal changes (Table 2), whereas joint training yields substantially more temporally coherent and predictable latents at comparable reconstruction quality (Table 13).
- Joint learning is a higher-performing and more practical alternative to fixed representations. It consistently improves forecasting over two-stage and raw-feature baselines (Tables 2–4), eliminates separate tokenizer training, and reduces training compute by 14–22% (Table 8).
- We make joint optimization work within an established generative framework. Our Two-Stage baseline matches our end-to-end architecture and follows the same two-stage autoencoder and flow-matching formulation as VFMF, isolating the effect of joint training; without our components, joint training collapses (Table 5, rows g–h). Making this formulation trainable end-to-end is exactly the direction you recognized as meaningful.
Precedent. Enabling a previously unsuccessful training formulation is widely recognized as a methodological contribution, even with simple components: SimSiam [A] demonstrated collapse-free Siamese representation learning using stop-gradient, while REPA-E [B] enabled end-to-end VAE–diffusion training where naive joint optimization fails. Our work makes an analogous contribution for generative VFM feature forecasting, where, to our knowledge, such joint training had not previously been demonstrated.
We will revise the contributions paragraph to make this positioning more explicit, and hope this clarifies our contribution and resolves the remaining concern.
Best regards, The Authors
[A] Exploring Simple Siamese Representation Learning. In CVPR 2021.
[B] REPA-E: Unlocking VAE for End-to-End Tuning of Latent Diffusion Transformers. In ICCV 2025.
We thank the reviewer for the thorough review, and for recognizing that joint optimization is a meaningful research direction, that our controlled two-stage comparison provides evidence for its benefits, and that our ablations are systematic and the paper is clearly presented. Given this recognition of our motivation and empirical methodology, we would be grateful if the reviewer could point out which specific soundness concerns remain after the clarifications and new analyses below, so that we can address them during the discussion period.
Characterization of existing methods
We thank the reviewer for this point and agree that our original characterization was too broad. We have clarified the introduction to distinguish between different approaches. DINO-Foresight uses PCA computed on VFM features and kept fixed during predictor training, which motivated our original two-stage characterization. Although PCA is not a separately trained tokenizer, it serves the same functional role of compressing features before forecasting. FlowWM and VGGT-World, in contrast, forecast directly on frozen features without a compression stage; we agree that direct forecasting is a valid alternative.
To explicitly evaluate this alternative, we have extended our existing Raw VFM Features baseline to high-resolution Cityscapes (Table 2), nuScenes (Table 3), and Kubric (Table 4). This baseline forecasts directly on frozen DINOv2 features (3072 channels), using x-prediction, as in VGGT-World, to stabilize flow matching in high-dimensional feature spaces. Our end-to-end approach consistently outperforms this baseline, e.g., mid-term movable-object segmentation on Cityscapes (55.48→59.92, Table 2), longer-term depth accuracy on nuScenes (67.8→72.0 δ1, Table 3), and 8-step movable-object IoU on Kubric (30.33→35.29, Table 4). Summarized results at the longest horizon of each benchmark are shown in the table below.
The common limitation across these approaches is not two-stage training, but that the representation used for forecasting remains fixed rather than being jointly optimized with the predictor. This is precisely the limitation our work addresses.
Novelty and evidence for joint training.
Our contribution is not merely to simplify the training pipeline of VFMF, but to jointly learn the VFM representation and its temporal dynamics, allowing the forecasting objective to directly shape the representation. This is nontrivial: naive joint training leads to latent collapse (Table 5, rows g–h), and we propose an effective training strategy that prevents it. Our new latent predictability analysis (Table 13; detailed in the next response) further supports that joint training produces representations with more predictable temporal dynamics.
We emphasize that our Two-Stage with AE baseline was explicitly designed to isolate the contribution of joint training. It matches our architecture, latent dimensionality, and training data, with the key difference being whether the tokenizer is optimized jointly with the predictor. Notably, this baseline already outperforms VFMF on low-resolution Cityscapes (Table 2), confirming that it provides a competitive reference point. The gains are consistent and particularly substantial in challenging settings: +2.2 short-term MO and +1.9 mid-term MO on high-resolution Cityscapes (Table 2), +4.9 mid-term δ1 on nuScenes (Table 3), and a 20.5% relative reduction in longer-term AbsRel (0.390→0.310) on nuScenes (Table 3). These controlled comparisons provide direct evidence of the benefits of joint optimization, rather than differences in architecture or latent dimensionality. We would appreciate clarification on which aspects of this comparison the reviewer considers insufficient to isolate the contribution of joint training.
Beyond forecasting accuracy, end-to-end training simplifies the training pipeline by eliminating separate tokenizer training and checkpoint selection, particularly benefiting high-resolution fine-tuning and adaptation to new datasets. It also reduces total training compute by 14–22% (see our response to Reviewer 1 and new Table 8).
Finally, we do not claim that jointly learned latents are necessary for forecasting. Direct forecasting on frozen features, as in VGGT-World, is a valid alternative. Nevertheless, our Raw VFM Features baseline (Tables 2–4; discussed in the previous response) shows that jointly learned latents consistently outperform raw-feature forecasting.