acceptodds
Under review as a conference paper at ICLR 2027

Latent-Foresight: End-to-End Learning Predictable Representations for Latent World Models

Abstract

Predicting the future evolution of a scene is a fundamental capability for world modeling. Recent work has shown that operating in the feature space of Vision Foundation Models (VFMs) yields semantically rich representations that support diverse future scene understanding tasks. However, existing approaches rely on two-stage pipelines, where VFM features are first compressed using fixed dimensionality reduction (e.g., PCA) or independently trained autoencoders, and a separate predictor is trained on top of the resulting frozen latent space. This decoupling between representation learning and temporal prediction, as well as approaches that apply predictors directly on raw VFM features, provides no guarantee that the latent space is structured for predictable dynamics. In this work, we propose Latent-Foresight, an end-to-end framework that jointly learns a latent tokenizer and a flow-based generative dynamics model, explicitly shaping the representation to support temporal predictability. To enable stable joint optimization, we introduce several key design choices that prevent latent collapse and align reconstruction with generative objectives. Extensive experiments show that our approach learns more temporally coherent latent representations and consistently outperforms two-stage baselines across multiple future scene understanding tasks and prediction horizons, while eliminating separate training stages, including during high-resolution adaptation.

Primary area:
generative models
Keywords:
Future Feature Prediction, World Models, Self-supervised learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NdpgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents Latent-Foresight, a framework that jointly trains a feature autoencoder and a flow-based predictor to address the limitations of separately trained tokenizers and predictors in VFM-based world models. Building upon existing approaches such as VFMF, the method introduces several training strategies to stabilize joint optimization and prevent latent collapse. Experimental comparisons with DINO-Foresight, VFMF, and internal two-stage baselines demonstrate certain benefits of joint training. The ablation studies are comprehensive, and the paper is generally well organized and clearly presented. However, the motivation relies on an overly broad characterization of existing methods, and the methodological novelty and evidence supporting the central claims remain insufficient.

Critical strengths
  1. The paper identifies a reasonable limitation of approaches such as VFMF, where the tokenizer and predictor are trained separately. A representation optimized primarily for feature reconstruction may not necessarily be optimal for modeling temporal dynamics. Exploring joint optimization is therefore a meaningful research direction.

  2. The comparison between the internal two-stage AE baseline and the proposed end-to-end framework provides evidence that joint optimization can improve forecasting performance while maintaining reconstruction quality.

  3. The ablation studies in Tables 5–7 systematically investigate latent normalization, stop-gradient strategies, auxiliary reconstruction, latent dimensionality, and noise schedules. These experiments provide useful insights into the training framework. The overall presentation is also clear and well organized.

Critical weaknesses
  1. Ambiguous and insufficiently justified motivation. In Lines 53–58, the paper characterizes existing VFM-based forecasting approaches as relying on a two-stage training paradigm. However, methods such as DINO-Foresight, FlowWM, and VGGT-World do not involve separately training a tokenizer and then freezing it to train a predictor. These approaches adopt different feature representations and forecasting strategies, which should not be broadly categorized under the same two-stage training limitation. While this concern is relevant to VFMF, its generalization to existing VFM-based world models is insufficiently justified. As a result, the stated limitations of prior methods are not sufficiently precise, weakening the central motivation.

  2. Limited novelty over VFMF and insufficient evidence for the benefits of joint training. The proposed framework closely follows VFMF, which already combines learned VFM feature compression, autoregressive flow matching, and feature reconstruction. The main distinction appears to be the joint optimization of the tokenizer and predictor, supported by additional training stabilization techniques. While this is a reasonable extension, its methodological novelty seems limited. Moreover, the direct comparison with VFMF involves substantially different latent dimensionalities and tokenizer architectures, making it difficult to attribute the reported improvements to joint training. Although the internal two-stage AE baseline partially addresses this concern, the improvements are relatively modest in some settings. Consequently, the experimental evidence does not fully justify the claimed significance of the proposed end-to-end paradigm. Furthermore, recent approaches such as VGGT-World demonstrate effective forecasting directly in frozen foundation-model feature spaces, challenging the claimed necessity of a jointly learned latent representation.

Questions for authors
  1. Limited evidence for improved latent predictability. The claim that the learned representation has more predictable temporal dynamics is primarily supported by downstream perception performance. Direct analyses of latent-space predictability, temporal consistency, or optimization characteristics are missing. Consequently, the experimental results do not fully establish the central representation-learning hypothesis.
4 comments

We thank the reviewer for the thorough review, and for recognizing that joint optimization is a meaningful research direction, that our controlled two-stage comparison provides evidence for its benefits, and that our ablations are systematic and the paper is clearly presented. Given this recognition of our motivation and empirical methodology, we would be grateful if the reviewer could point out which specific soundness concerns remain after the clarifications and new analyses below, so that we can address them during the discussion period.

Characterization of existing methods

We thank the reviewer for this point and agree that our original characterization was too broad. We have clarified the introduction to distinguish between different approaches. DINO-Foresight uses PCA computed on VFM features and kept fixed during predictor training, which motivated our original two-stage characterization. Although PCA is not a separately trained tokenizer, it serves the same functional role of compressing features before forecasting. FlowWM and VGGT-World, in contrast, forecast directly on frozen features without a compression stage; we agree that direct forecasting is a valid alternative.

To explicitly evaluate this alternative, we have extended our existing Raw VFM Features baseline to high-resolution Cityscapes (Table 2), nuScenes (Table 3), and Kubric (Table 4). This baseline forecasts directly on frozen DINOv2 features (3072 channels), using x-prediction, as in VGGT-World, to stabilize flow matching in high-dimensional feature spaces. Our end-to-end approach consistently outperforms this baseline, e.g., mid-term movable-object segmentation on Cityscapes (55.48→59.92, Table 2), longer-term depth accuracy on nuScenes (67.8→72.0 δ1, Table 3), and 8-step movable-object IoU on Kubric (30.33→35.29, Table 4). Summarized results at the longest horizon of each benchmark are shown in the table below.

MethodCityscapes, mid-term (Cos↑ / ALL↑ / MO↑)nuScenes, 27 frames (Cos↑ / δ₁↑ / AbsRel↓)Kubric, 8-step (BG↑ / ALL↑ / MO↑)
Raw VFM Features (3072 ch.)0.929 / 58.45 / 55.480.857 / 67.8 / 0.34890.90 / 60.62 / 30.33
End-to-End (Ours, 256 ch.)0.937 / 61.74 / 59.920.874 / 72.0 / 0.31091.64 / 63.47 / 35.29
improvement+0.008 / +3.3 / +4.4+0.017 / +4.2 / -0.038+0.7 / +2.9 / +5.0

The common limitation across these approaches is not two-stage training, but that the representation used for forecasting remains fixed rather than being jointly optimized with the predictor. This is precisely the limitation our work addresses.

Novelty and evidence for joint training.

Our contribution is not merely to simplify the training pipeline of VFMF, but to jointly learn the VFM representation and its temporal dynamics, allowing the forecasting objective to directly shape the representation. This is nontrivial: naive joint training leads to latent collapse (Table 5, rows g–h), and we propose an effective training strategy that prevents it. Our new latent predictability analysis (Table 13; detailed in the next response) further supports that joint training produces representations with more predictable temporal dynamics.

We emphasize that our Two-Stage with AE baseline was explicitly designed to isolate the contribution of joint training. It matches our architecture, latent dimensionality, and training data, with the key difference being whether the tokenizer is optimized jointly with the predictor. Notably, this baseline already outperforms VFMF on low-resolution Cityscapes (Table 2), confirming that it provides a competitive reference point. The gains are consistent and particularly substantial in challenging settings: +2.2 short-term MO and +1.9 mid-term MO on high-resolution Cityscapes (Table 2), +4.9 mid-term δ1 on nuScenes (Table 3), and a 20.5% relative reduction in longer-term AbsRel (0.390→0.310) on nuScenes (Table 3). These controlled comparisons provide direct evidence of the benefits of joint optimization, rather than differences in architecture or latent dimensionality. We would appreciate clarification on which aspects of this comparison the reviewer considers insufficient to isolate the contribution of joint training.

Beyond forecasting accuracy, end-to-end training simplifies the training pipeline by eliminating separate tokenizer training and checkpoint selection, particularly benefiting high-resolution fine-tuning and adaptation to new datasets. It also reduces total training compute by 14–22% (see our response to Reviewer 1 and new Table 8).

Finally, we do not claim that jointly learned latents are necessary for forecasting. Direct forecasting on frozen features, as in VGGT-World, is a valid alternative. Nevertheless, our Raw VFM Features baseline (Tables 2–4; discussed in the previous response) shows that jointly learned latents consistently outperform raw-feature forecasting.

Direct evidence of latent predictability

We thank the reviewer for this suggestion. To directly characterize the latent spaces, we perform a training-free analysis of ground-truth video latent dynamics, independently of any learned predictor. A closed-form ridge regression, fitted on a disjoint set of sequences, maps the latents of the last 1 or 3 frames to the latent 3 or 9 video frames ahead (0.18 s or 0.53 s). We report Ridge R² (variance-weighted) and Identity R², the score obtained by simply copying the current latent (zt+k=ztz_{t+k}=z_t). Both tokenizers have the same bottleneck dimension (C=256C=256), so the linear map has identical capacity in both latent spaces.

To separate the effect of the encoder from that of the decoder, we include the control from Table 2 (Two-Stage w/ Aux Recon): the Two-Stage encoder is kept frozen, and the decoder is trained on predicted latents with the auxiliary reconstruction loss.

Table 13: Training-free analysis of latent dynamics. Cityscapes val; ridge fitted on 300 train sequences. Recon. Cos is measured in the normalized feature space.

Res.MethodRecon. Cos↑Identity R²↑ 3 fr.Identity R²↑ 9 fr.Ridge R²↑, 1 ctx 3 fr.Ridge R²↑, 1 ctx 9 fr.Ridge R²↑, 3 ctx 3 fr.Ridge R²↑, 3 ctx 9 fr.
224×448Two-Stage with AE0.9270.294−0.0040.4710.3200.4930.345
224×448Two-Stage w/ Aux Recon (frozen encoder)0.9230.294−0.0040.4710.3200.4930.345
224×448End-to-End (Ours)0.9280.3840.1230.5390.4000.5580.423
448×896Two-Stage with AE0.9370.140−0.1110.3930.2800.4190.307
448×896End-to-End (Ours)0.9380.3160.1100.5170.4130.5370.437

The results reveal three points:

  • Comparable information retention: all latent spaces reconstruct DINOv2 features equally well, so the differences below are not explained by discarded information.
  • End-to-end training makes the latent more persistent and more linearly predictable in every setting (both resolutions, both horizons, one or three context frames), and the advantage grows with the horizon (Ridge R² gap with one context frame: +0.068 → +0.080 at 224×448 and +0.124 → +0.133 at 448×896). The w/ Aux Recon control shares the Two-Stage encoder, so its latent-space metrics are identical to Two-Stage by construction: this difference comes from training the encoder with the forecaster, not from the decoder or the auxiliary loss.
  • The more predictable latent translates into better forecasts. Compared with the control (same predictor, same decoder training, equal reconstruction: 68.08/66.94 vs. 68.29/67.26 ALL/MO), End-to-End improves short-term and mid-term movable-object segmentation from 62.63 to 64.10 and from 53.12 to 53.86 (Table 2).

Visualization. Figure 3 and Appendix Figures 6–8 show PCA projections of the latents of both tokenizers over time. The end-to-end latents are smoother over time and group objects such as vehicles and pedestrians into compact regions that remain stable across frames, whereas the two-stage latents are noisier with weaker object separation. This is a qualitative illustration, consistent with the quantitative results above.

Together, these analyses show that, at equal reconstruction quality, joint training shapes a latent space that is more temporally coherent and more predictable, and that this improves downstream forecasting beyond what decoder adaptation alone provides.

Thank you for your reply. However, my main concern regarding the methodological novelty remains unresolved. While I acknowledge that naive joint training can lead to latent collapse and that the proposed strategy addresses this issue, the method largely follows the overall framework of VFMF. The main modifications, including normalization, stop-gradient, and an auxiliary reconstruction loss, appear to be primarily training optimizations rather than substantial methodological innovations. Therefore, my primary concern about novelty remains.

Dear Reviewer Ndpg,

Thank you for your follow-up and for acknowledging that naive joint training can lead to latent collapse and that our approach addresses it. We understand that the remaining concern is methodological novelty, and hope our previous responses have addressed the earlier points (prior work characterization, comparisons with direct forecasting, and latent predictability). We would be glad to discuss any that remain.

We agree with your initial assessment that "a representation optimized primarily for feature reconstruction may not necessarily be optimal for modeling temporal dynamics" and that joint optimization is "a meaningful research direction". Our contribution is precisely to make this joint learning possible; the novelty lies not in individual training components, but in what they enable:

  1. The forecasting objective shapes the representation. In VFMF, the latent is determined by reconstruction alone and frozen before forecasting. Adding auxiliary reconstruction to a frozen encoder brings only marginal changes (Table 2), whereas joint training yields substantially more temporally coherent and predictable latents at comparable reconstruction quality (Table 13).
  2. Joint learning is a higher-performing and more practical alternative to fixed representations. It consistently improves forecasting over two-stage and raw-feature baselines (Tables 2–4), eliminates separate tokenizer training, and reduces training compute by 14–22% (Table 8).
  3. We make joint optimization work within an established generative framework. Our Two-Stage baseline matches our end-to-end architecture and follows the same two-stage autoencoder and flow-matching formulation as VFMF, isolating the effect of joint training; without our components, joint training collapses (Table 5, rows g–h). Making this formulation trainable end-to-end is exactly the direction you recognized as meaningful.

Precedent. Enabling a previously unsuccessful training formulation is widely recognized as a methodological contribution, even with simple components: SimSiam [A] demonstrated collapse-free Siamese representation learning using stop-gradient, while REPA-E [B] enabled end-to-end VAE–diffusion training where naive joint optimization fails. Our work makes an analogous contribution for generative VFM feature forecasting, where, to our knowledge, such joint training had not previously been demonstrated.

We will revise the contributions paragraph to make this positioning more explicit, and hope this clarifies our contribution and resolves the remaining concern.

Best regards, The Authors

[A] Exploring Simple Siamese Representation Learning. In CVPR 2021.

[B] REPA-E: Unlocking VAE for End-to-End Tuning of Latent Diffusion Transformers. In ICCV 2025.

Reviewer_5uQpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors proposed to train the autoencoder and flow-matching predictor jointly. To prevent collapse and ensure well-structured latent spaces, latent normalization, stop-gradients and reconstruction regularization are proposed.

Critical strengths
  1. The problem is well motivated: Instead of separately optimizing compression and predictin, joint optimization is a reasonable direction.
  2. Promising results: Table 3 reports promising improvements over the two-stage baseline at all three horizons on nuScenes.
  3. Comprehensive ablations: The component ablations in Table 5 are informative, where the components needed to avoid collapse are well ablated.
Critical weaknesses
  1. The idea of using feature encoder instead of VAE is broadly studied in RAE [1], VideoRAE [2], these works should be cited and compared. Explain the reason if the comparison is not feasible.
  2. For the baselines in Table 1, is DeltaTok finetuned on Cityscapes dataset like VISTA to make sure the comparison is fair given that the improvement over DeltaTok is not substantial?

[1] Diffusion Transformers with Representation Autoencoders, https://arxiv.org/abs/2510.11690 [2] VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoders, https://arxiv.org/abs/2607.14088

Questions for authors

Can the authors report and compare with RAE, VideoRAE and related works because the ideas are similar in terms of the representation?

2 comments

We thank the reviewer for the review and for recognizing the novelty of jointly optimizing the tokenizer and the flow-matching predictor, the strength of our results on Cityscapes and nuScenes, and the clarity of the paper. We address the two points raised below, with changes included in the revised manuscript.

Relation to RAE and VideoRAE.

We thank the reviewer for these references, which we now cite and discuss in Related Work. RAE and VideoRAE primarily target realistic image/video generation, whereas our objective is to predict future VFM features from past observations for downstream scene understanding. Moreover, forecasting in VFM feature space instead of VAE was already explored by earlier works such as DINO-Foresight and DINO-WM, preceding RAE and VideoRAE. VideoRAE in particular appeared on arXiv only in July 2026, about two months before the submission deadline. Given their different objectives, direct comparisons are not applicable, but their underlying representation-learning strategies are already covered by our baselines.

RAE: Its central idea of directly modeling frozen pretrained features is closely related to our Raw VFM Features baseline, which forecasts frozen DINOv2 features (3072 channels) without a learned tokenizer. Originally evaluated on low-resolution Cityscapes (Table 2), we now extend this baseline to high-resolution Cityscapes (Table 2), nuScenes (Table 3), and Kubric (Table 4). Latent-Foresight consistently outperforms it, e.g., mid-term movable-object segmentation on Cityscapes (55.48→59.92, Table 2), longer-term depth accuracy on nuScenes (67.8→72.0 δ1, Table 3), and 8-step movable-object IoU on Kubric (30.33→35.29, Table 4).

VideoRAE: It learns compact latents from frozen video VFM features for video generation. Its objective differs from our future-feature forecasting task, making direct comparisons unsuitable. Our Two-Stage with AE baseline provides a relevant comparison to separately training and freezing a tokenizer; end-to-end training improves over it, e.g., by +1.9 mid-term MO on high-resolution Cityscapes, +4.9 δ1 on mid-term nuScenes, and +3.8 δ1 on longer-term nuScenes.

Thus, while these works share our motivation of leveraging pretrained visual representations, they keep the representation space fixed during generator training, whereas our contribution is to jointly optimize the tokenizer and temporal forecasting model.

Fairness of the DeltaTok comparison.

The DeltaTok numbers come from Table D of its supplementary material, where the authors integrate DeltaTok into the official DINO-Foresight implementation and evaluate on Cityscapes at 448×896. This is the setting of our Table 1: the DINO-Foresight row there is taken from the DINO-Foresight paper and matches ours. This DeltaTok variant is already adapted to Cityscapes: its tokenizer was trained on BDD100K and fine-tuned on Cityscapes, and the world model was retrained on Cityscapes, so it uses more data than our model. Like VISTA_ft in Table 1 (as reported in DINO-Foresight), it is compared at its best.

Dear Reviewer 5uQp,

Thank you again for your feedback, which led to several improvements to our paper. Among other changes, your pointers to RAE and VideoRAE helped us sharpen our positioning, we extended the raw-feature baseline to all benchmarks (Tables 2–4), and we clarified the DeltaTok evaluation (Table 1). As the discussion period ends on November 18, we would be grateful to know whether our response addresses your concerns, and we are happy to clarify anything further.

Best regards, The Authors

Reviewer_MDE2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Latent-Foresight, an end-to-end framework for learning predictable latent representations for world modeling. While existing approaches employ a two-stage pipeline, this paper jointly optimizes a VFM feature tokenizer and a flow-matching-based future predictor, allowing the latent representations to adapt to the requirements of future prediction. To stabilize end-to-end training and prevent latent collapse, the authors introduce several techniques, including latent normalization, stop-gradient operations, and an auxiliary reconstruction loss. Experiments on multiple datasets demonstrate that the proposed method consistently improves future scene understanding performance across different prediction horizons while maintaining reconstruction quality.

Critical strengths
  • This paper makes a valuable technical and empirical contribution to the joint optimization of autoencoders and future predictors. The proposed training strategies provide useful insights into achieving stable end-to-end learning while preventing latent collapse.
  • The central hypothesis of this paper is effectively validated through extensive experiments. The consistent improvements over two-stage baselines support the effectiveness of jointly learning latent representations and future predictors.
  • The paper is well-organized and easy to follow.
Critical weaknesses
  • The conflict between reconstruction and prediction objectives has already been explored in conventional video world models. This paper essentially revisits the same problem in the VFM feature space, limiting its conceptual novelty.
  • Although the paper claims to simplify the training pipeline, it lacks comparisons of computational costs and training time with two-stage methods.
  • The evaluation mainly focuses on autonomous driving scenarios, with limited experiments on other real-world environments. More diverse benchmarks are needed to demonstrate the generalizability of the method.
  • The contribution of end-to-end learning is not fully isolated from the auxiliary reconstruction loss. A comparison with a two-stage variant that freezes the encoder while jointly optimizing the predictor and decoder using the auxiliary reconstruction loss would help clarify the source of the improvements.
Questions for authors

Please refer to the weaknesses part.

5 comments

We thank the reviewer for the careful review and for recognizing our technical contribution to joint optimization, the effectiveness of our training strategies against latent collapse, and the clarity of the paper. We address each point below, with changes included in the revised manuscript.

Relation to the reconstruction–generation trade-off

We agree that the tension between reconstruction and generation quality has been studied in prior work, although the reviewer does not provide references, making it difficult to identify which works they refer to. However, our setting and contribution are fundamentally different. Prior works (e.g., VA-VAE [1]) predominantly study this trade-off in two-stage image/video generation, where the tokenizer and generator are trained separately. Their primary objective is generating realistic visual content, whereas we study temporal forecasting of VFM features, where representations must simultaneously preserve semantic information and capture predictable scene dynamics for downstream scene understanding. To this end, we jointly optimize the tokenizer and the flow-matching dynamics model. While REPA-E and other works cited in our Related Work also perform end-to-end training for image generation, to our knowledge, joint tokenizer–predictor optimization for temporal feature forecasting has not been previously investigated.

Moreover, joint optimization in our setting is nontrivial: naive joint training leads to latent collapse (Table 5, rows g–h), which our proposed approach successfully prevents. Thus, our contribution goes beyond improving the forecasting model: joint training learns representations intrinsically better suited to temporal prediction without sacrificing reconstruction quality.

[1] J. Yao et. al, "Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models", In CVPR 2025

Computational cost and training time

We now report training cost of the pipelines producing the 448×896 Cityscapes results (new Table 8). All runs use 16 GH200 GPUs; times are exact job elapsed times.

Table 8: Training cost on Cityscapes.

SetupTraining jobsWall-clockGPU-hours
Two-Stage, low-resAE (1:52) → predictor (3:49)5:4190.9
End-to-End (Ours), low-resjoint (4:25)4:2570.8
Two-Stage, high-reslow-res stages → high-res adaptation (0:28)6:0998.5
End-to-End (Ours), high-reslow-res joint → high-res adaptation (0:53)5:1884.9

Our end-to-end approach reduces total training compute by 22% at low resolution and 14% including high-resolution adaptation, while requiring fewer sequential training stages and eliminating separate tokenizer training and checkpoint selection. Importantly, inference cost remains identical, as both approaches share the same architecture at test time.

Evaluation beyond autonomous driving

We evaluate our method on three datasets: Cityscapes, nuScenes, and Kubric. Autonomous driving is a challenging and practically important application of world modeling, requiring the prediction of complex real-world scene dynamics, with direct implications for safety-critical autonomous systems. To evaluate beyond driving, we additionally consider Kubric (Table 4), a challenging benchmark featuring general multi-object dynamics, physical interactions, and occlusions, following the evaluation protocol of VFMF. Our results demonstrate that Latent-Foresight generalizes effectively to this substantially different setting, outperforming both VFMF (a two-stage approach) and direct forecasting on raw VFM features (which we have extended to Kubric in response to Reviewer 3). Together, these evaluations provide evidence that the benefits of our approach extend beyond autonomous driving to more general visual dynamics.

Isolating end-to-end learning from the auxiliary reconstruction loss

Thank you for this suggestion. We ran the requested control: a frozen two-stage encoder, with the predictor and decoder trained jointly using the auxiliary loss (224×448). We added this comparison to Table 2 (under the Two Stage AE w/ Aux Recon).

EncoderAux-reconRecon ALL↑Recon MO↑Short ALL↑Short MO↑Mid ALL↑Mid MO↑
Frozen✗67.9666.8864.6962.5756.3153.03
Frozen✓68.0866.9464.5762.6356.2553.12
Trained (Ours)✗68.2767.1965.0963.4356.5553.36
Trained (Ours)✓68.2967.2665.4564.1056.8453.86

As shown in summary table above, end-to-end training improves downstream forecasting even without the auxiliary loss (row 3 vs. row 1), improving short-term and mid-term movable-object segmentation from 62.57 to 63.43 and from 53.03 to 53.36, respectively. Adding the auxiliary loss to the frozen-encoder baseline provides only marginal changes (row 2 vs. row 1). Combining end-to-end training with the auxiliary loss yields the strongest results (row 4), with short-term and mid-term movable-object segmentation reaching 64.10 and 53.86, respectively.

Dear Reviewer MDE2,

Thank you again for your feedback, which led to several improvements to our paper. Among other changes, following your suggestion, we added a controlled comparison isolating joint training from the auxiliary loss (Table 2), as well as a training cost comparison showing 14–22% lower compute (Table 8). As the discussion period ends on November 18, we would be grateful to know whether our response addresses your concerns, and we are happy to clarify anything further.

Best regards, The Authors

Comments on the paper

2 comments

We thank the reviewers for their constructive feedback. We have uploaded a revised manuscript in which all new additions are marked in blue, and we have addressed the concerns through the following revisions and experiments:

  • Related work and positioning: Clarified distinctions from DINO-Foresight, FlowWM, VGGT-World, RAE, and VideoRAE (Introduction and Related Work), and highlighted how our existing baselines evaluate relevant alternative representation-learning strategies, demonstrating the benefits of joint training (Tables 2–4). We also extended the raw-feature comparison to high-resolution Cityscapes, nuScenes, and Kubric (Tables 2–4) and clarified the DeltaTok evaluation (Table 1).

  • Contribution and attribution: Added controlled comparisons, including a frozen encoder with auxiliary reconstruction ("Two-Stage w/ Aux Recon", Table 2), to disentangle the benefits of joint training from those of the auxiliary loss.

  • Latent predictability: Added training-free probes (Table 13) and PCA visualizations of the latents (Fig. 3 and the appendix), showing that our latents are more temporally coherent and predictable while maintaining comparable reconstruction quality.

  • Efficiency and generalization: Reported 14–22% lower training compute (Table 8) and clarified that our evaluation covers both challenging, real-world driving scenarios (Cityscapes in Tables 1–2, nuScenes in Table 3) and general multi-object dynamics beyond driving (Kubric, Table 4).

We believe that, together, these additions address the concerns regarding positioning, baseline fairness, the benefits of joint training, and the strength and scope of the empirical evidence.

Dear AC and Reviewers,

As methodological novelty remains a central concern, we would like to clarify the specific novelty and contribution of Latent-Foresight.

  1. What is new? Existing generative VFM forecasting methods operate in fixed representation spaces: raw VFM features, fixed PCA projections, or separately trained tokenizers (as in VFMF). In contrast, we jointly optimize the VFM-feature tokenizer and the temporal flow-matching predictor, allowing the forecasting objective to shape the representation itself.
  2. What is our methodological contribution? Naive joint training leads to latent collapse (Table 5). Our main contribution is making this joint learning possible. The novelty lies not in the individual training components, but in overcoming the fundamental optimization challenge of jointly learning representations and generative temporal dynamics.
  3. Why does it matter? Our experiments demonstrate that joint optimization:
    • Learns more temporally coherent and predictable representations, at equal reconstruction quality, the gap over two-stage latents grows with the prediction horizon (Table 13).
    • Consistently improves forecasting over matched two-stage and raw-feature baselines (Tables 2–4).
    • Simplifies training, eliminating a separate tokenizer-training stage and reducing training compute by 14–22% (Table 8).

Specifically regarding Reviewer Ndpg's comparison with VFMF: The key difference is not the individual training components, but what is learned. In VFMF, the tokenizer is optimized solely for reconstruction and remains fixed during forecasting, whereas our forecasting objective directly shapes the learned representation. Our matched two-stage baseline isolates the benefits of this distinction (Tables 2 and 13).

Regarding RAE and VideoRAE (Reviewer 5uQp): These methods target image/video generation rather than VFM feature forecasting (now discussed in Section 2). Moreover, their underlying strategies, modeling raw frozen features (RAE) and using a separately trained tokenizer (VideoRAE), are covered by analogous baselines in our paper, which our end-to-end approach consistently outperforms(Tables 2–4).

In summary, to our knowledge, none of the methods cited by the reviewers jointly learns a VFM-feature tokenizer and a generative temporal predictor for future feature forecasting. Our contribution is to make this joint learning possible and demonstrate its benefits.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.