acceptodds
Under review as a conference paper at ICLR 2027

Seeing Ahead: Plug-in World Models for Predictive Reasoning in Frozen VLMs

Abstract

Understanding a video involves recognizing the visible scene and anticipating how it may change. Video world models learn predictive representations; vision–language models (VLMs) connect visual evidence to questions and answers. We study how to combine these capabilities while retaining the VLM's own perception. We propose FoReST (Forecast Residual State Transfer), which adds predictive updates to a frozen VLM's visual tokens. One variant translates changes predicted by a pretrained world model; the other forecasts the update from that model's observed-video features. Only a compact head is trained from video continuations, without question–answer supervision. We introduce CLEVRER-Next, a next-event QA dataset built from existing CLEVRER videos and annotations, with explicit observation cutoffs and retained continuations for diagnosis. On CLEVRER-Next and ComPhy, latent transfer improves accuracy by 11.5 and 11.7 percentage points over the same frozen VLM given observed video. Against a trained forecaster of the VLM's own features, latent transfer provides additional gains on ComPhy, while direct forecasting improves CLEVRER-Next collision prediction. Controlled comparisons show that retaining native vision reduces degradation from imperfect forecasts and that prediction-input training can improve their translation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
World Models, Physical Reasoning, Vision–Language Models, Latent-Space Prediction, Representation Alignment, Video Question Answering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dRT9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FoReST (Forecast Residual State Transfer), whose goal is to let a frozen VLM "see ahead" by utilizing world models. Four copies of the last observed frame are appended to the video and encoded with the VLM's own vision encoder. The head predicts a residual, i.e., how those four frames' tokens should change to look like the real future. The head is trained on unlabeled video without QA labels, and uses a frozen world model (V-JEPA 2) to predict the residual. Two variants are offered: FoReST-Latent translates the pretrained predictor's latent forecast (P − R) into the VLM's feature space via a pointwise zero-preserving MLP, and FoReST-Direct learns the residual from V-JEPA 2's observed encoder features via cross-attention, bypassing the released predictor entirely.

They also propose a new dataset, CLEVRER-Next, a next-event question-answering dataset built directly on an existing dataset (CLEVRER videos plus simulator annotations), with five-choice questions, explicit observation cutoffs, and retained continuations for diagnostic evaluation.

Critical strengths
  • The idea is clean and cheap. Everything pretrained stays frozen, the head is small (~12.6M parameters), and one update is reused across all questions about a clip. The question-independence of the update is a genuinely attractive property that is easy to overlook.

  • The controls are rigorous: static continuation, shuffled updates from another video, temporal reversal, a training-mean update, three seeds and paired bootstrap intervals. The shuffle control is also executed properly in details that are easy to get wrong; donors are matched within event family and future-duration bin, and a donor's prediction is always paired with that donor's own persistence reference.

  • Results are reported honestly, including unfavourable ones (e.g., Latent losing to the native baseline on CLEVRER). This extends further than the main text suggests: the CTX-P / CTX-O contextual controls are reported as failing to establish a predictive-query advantage (Appendix D.2), and Appendix D.4 reports that per-scene representation error barely correlates with QA accuracy (|ρ| ≤ 0.106), with ABS-add relative native MSE improving (1.1719 -> 1.1159 -> 1.1128) while accuracy falls (57.2 -> 53.6 -> 50.4%). These are negative results most authors would bury.

  • The add-vs-replace interface claim is the paper's most transferable finding and is cleanly isolated. Section 5.2 and Appendix D.1 hold the head fixed and interpolate the evaluation input between actual and predicted states; replacement starts higher at λ = 0 (60.9% vs. 57.2%) but degrades more, giving addition a 6.1 pp [2.0, 10.2] smaller drop. The authors correctly note this is a statement about degradation, not final accuracy, and that the predicted-endpoint advantage is only 2.4 pp [−1.1, 6.0].

Critical weaknesses
  • The world model adds less than the title suggests. The fairest baseline is VLM-Forecast, the same head and target, but predicting from the VLM's own features, with no world model at all; which still generates most of the gains on CLEVRER-Next, 46.2 -> 61.2, whereas FoReST-Latent (57.6) actually performs worse than this baseline. I can understand that the gain mostly comes from training a future-prediction head, not from plugging in a world model.
  • FoReST-Direct doesn't really use the world model's forecasting. It only uses V-JEPA 2's encoder features, never its predictor (confirmed in §2.3 and Appendix A.3). Its advantage over VLM-Forecast could simply come from a different visual encoder. So the one configuration that beats VLM-Forecast on CLEVRER-Next is not testing world-model forecasting at all; it is testing whether a self-supervised video encoder's features are better input for a learned forecaster than the VLM's own visual tokens.
Questions for authors
  • How does FoReST compare to directly fine-tuning the VLM on CLEVRER-Next or ComPhy using question-answer supervision? This would help contextualize the value of the proposed unsupervised, plug-in approach relative to standard end-to-end adaptation.
  • Why was a shuffled-update control not included for VLM-Forecast, given that both FoReST variants are evaluated against their own shuffled counterparts? This asymmetry limits the ability to fairly assess content specificity across methods.
  • Could you report the inference-time computational cost—specifically FLOPs, latency, and peak memory—for FoReST-Latent, FoReST-Direct, and VLM-Forecast? This would clarify the practical trade-offs of using an external world model.
Reviewer_rxgpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores how to endow a VLM with the predictive capabilities of a pretrained world model, without sacrificing its perception ability. The proposed method, FoReST, keeps every pretrained component frozen (V-JEPA 2 as the world model; Qwen3.5-27B as the primary VLM) and trains a ~12.6M-parameter head that predicts an additive residual on the VLM’s visual tokens. They introduce two variants: FoReST-Latent translates the output of a pretrained world-model predictor into the VLM's feature space, whereas FoReST-Direct learns the future residual directly from the world-model encoder's observed-video features. This paper also introduces its own five-choice future QA benchmark called CLEVRER-Next. Experiments on CLEVRER-Next and ComPhy show improvements over baselines. Additional experiments and analyses cover the effect of retaining native visual tokens, using predicted rather than actual latent states during mapper training, and the dependence of the learned updates on video content.

Critical strengths
  1. The authors explore an important and timely research direction of combining the predictive capabilities of world models with the visual grounding and latent understanding of VLMs. The proposed solution is conceptually straightforward and potentially useful in cases in which VLMs struggle with implicitly doing future prediction. A particularly appealing aspect is that only a small head is trained from paired video continuations, rather than VQA or some other question-dependent supervision.

  2. The empirical gains over the frozen VLM are evident. This shows that on this task, the primary frozen VLM struggles out of the box on future prediction. The ablations are also strong and straightforward: a static-continuation control that matches the input layout, a VLM-Forecast baseline that removes the world model at matched head size, shuffled-update controls, three training seeds, and paired confidence intervals.

Critical weaknesses
  1. The VLM-Forecast baseline, which uses no world model, captures most of the gain. On CLEVRER-Next, FoReST-Direct exceeds VLM-Forecast by approximately 2.0 points and FoReST-Latent underperforms VLM-Forecast by approximately 3.6 points. Although on ComPhy, FoReST-Latent outperforms VLM-Forecast, and the paper does not sufficiently explain this discrepancy. The stated contribution seems to hinge on plug-in world models rather than a learned future residual more generally, but Figure 4 further complicates this. The 29.1-point gap between mapped real-future and native real-future features suggests a substantial representation-mapping bottleneck, even without forecasting error. A clearer analysis of when and why external world-model representations provide benefits would strengthen the central contribution.

  2. The construction and presentation of CLEVRER-Next makes its difficulty hard to assess. The cutoff for continuation is fixed relative to the target event and event types are imbalanced. The motivation for introducing a new benchmark rather than using existing future-event QA evaluations could be better established. There are no concrete, visualized examples pairing observed scenes with complete questions and answer choices. This makes it difficult to understand what is actually being asked. Simple heuristic baselines, related work, and a figure with a couple example questions could address most of this.

Questions for authors
  1. How much of the observed performance gap reflects limitations in the VLM's predictive capabilities versus the inference setup? Particularly, what does the frozen VLM score with no visual input (to measure answer bias)? What does it score with thinking enabled or more sampled frames? This would provide insight into whether the gap closes with a stronger VLM baseline.

  2. Why not also evaluate on CLEVRER’s predictive category of questions ? This would allow comparison with other methods who’ve been evaluated on this benchmark.

Reviewer_oAwoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FoReST, which feeds predictions from a video world model into a frozen VLM by adding a residual to the VLM’s visual tokens. FoReST-Latent maps the world model’s predicted residual (P − R) into a residual in VLM space. FoReST-Direct skips the world-model predictor and predicts the VLM residual directly from observed world-model features. Only a ~12.6M-parameter head is trained, on paired video continuations, with no QA labels. The authors also introduce CLEVRER-Next, a next-event QA benchmark. On CLEVRER-Next and ComPhy, both variants beat the observed-video baseline. Against VLM-Forecast, a head trained on the VLM’s own features, the gains depend on the task.

Critical strengths
  1. The method is simple and easy to use. The VLM and its visual pathway stay frozen, and only a small head is trained without QA supervision. Adding predictive information to an existing VLM this way is cheap.

  2. The diagnostics are informative. VLM-Forecast is a strong baseline, and the shuffled-update, temporal-reversal, and interface (addition vs. replacement) experiments help show when the predicted updates help and how the two variants differ.

Critical weaknesses
  1. Limited added value of the external world model. VLM-Forecast is already strong. Over it, FoReST-Direct gains only 1.98 points on CLEVRER-Next and 1.35 on ComPhy, and the per-seed intervals mostly cross zero. FoReST-Latent is better on ComPhy (+4.71) but worse on CLEVRER-Next (−3.55). As the authors note in Appendix C.3, this does not show a general advantage for external representations.

  2. Weak support for residualization and narrow evaluation. In Appendix D.1, residual-input and absolute-input addition reach the same predicted-endpoint accuracy (0.0 [−2.4, 2.5]). The 6.1-point gap in Figure 3 is addition vs. replacement, not residual vs. absolute input. Evaluation is also limited to synthetic benchmarks and a single world model. Some figures are hard to read on their own; Figures 3 and 5, for example, depend on other figures or tables for their actual values.

  3. Figures need to be refined

Questions for authors
  1. Residualization: Is there multi-seed or main-evaluation evidence that mapping P − R beats mapping P directly?

  2. External vs. native forecasting: Could you add a shuffled-update control for VLM-Forecast, to separate the value of the world model from that of forecasting on the VLM’s own features?

  3. Data splits: Are the training and validation scenes disjoint from the main evaluation set? Is the 500-scene analysis set a subset of the main evaluation? Both contain exactly 162 entrance scenes. If so, please report results with the analysis scenes excluded.

Comments on the paper

1 comment

Dear reviewers, the authors have posted their responses to your reviews. Pls have a look and discuss.

Thanks, AC for #26470

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.