- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes FoReST (Forecast Residual State Transfer), whose goal is to let a frozen VLM "see ahead" by utilizing world models. Four copies of the last observed frame are appended to the video and encoded with the VLM's own vision encoder. The head predicts a residual, i.e., how those four frames' tokens should change to look like the real future. The head is trained on unlabeled video without QA labels, and uses a frozen world model (V-JEPA 2) to predict the residual. Two variants are offered: FoReST-Latent translates the pretrained predictor's latent forecast (P − R) into the VLM's feature space via a pointwise zero-preserving MLP, and FoReST-Direct learns the residual from V-JEPA 2's observed encoder features via cross-attention, bypassing the released predictor entirely.
They also propose a new dataset, CLEVRER-Next, a next-event question-answering dataset built directly on an existing dataset (CLEVRER videos plus simulator annotations), with five-choice questions, explicit observation cutoffs, and retained continuations for diagnostic evaluation.
Critical strengths
-
The idea is clean and cheap. Everything pretrained stays frozen, the head is small (~12.6M parameters), and one update is reused across all questions about a clip. The question-independence of the update is a genuinely attractive property that is easy to overlook.
-
The controls are rigorous: static continuation, shuffled updates from another video, temporal reversal, a training-mean update, three seeds and paired bootstrap intervals. The shuffle control is also executed properly in details that are easy to get wrong; donors are matched within event family and future-duration bin, and a donor's prediction is always paired with that donor's own persistence reference.
-
Results are reported honestly, including unfavourable ones (e.g., Latent losing to the native baseline on CLEVRER). This extends further than the main text suggests: the CTX-P / CTX-O contextual controls are reported as failing to establish a predictive-query advantage (Appendix D.2), and Appendix D.4 reports that per-scene representation error barely correlates with QA accuracy (|ρ| ≤ 0.106), with ABS-add relative native MSE improving (1.1719 -> 1.1159 -> 1.1128) while accuracy falls (57.2 -> 53.6 -> 50.4%). These are negative results most authors would bury.
-
The add-vs-replace interface claim is the paper's most transferable finding and is cleanly isolated. Section 5.2 and Appendix D.1 hold the head fixed and interpolate the evaluation input between actual and predicted states; replacement starts higher at λ = 0 (60.9% vs. 57.2%) but degrades more, giving addition a 6.1 pp [2.0, 10.2] smaller drop. The authors correctly note this is a statement about degradation, not final accuracy, and that the predicted-endpoint advantage is only 2.4 pp [−1.1, 6.0].
Critical weaknesses
- The world model adds less than the title suggests. The fairest baseline is VLM-Forecast, the same head and target, but predicting from the VLM's own features, with no world model at all; which still generates most of the gains on CLEVRER-Next, 46.2 -> 61.2, whereas FoReST-Latent (57.6) actually performs worse than this baseline. I can understand that the gain mostly comes from training a future-prediction head, not from plugging in a world model.
- FoReST-Direct doesn't really use the world model's forecasting. It only uses V-JEPA 2's encoder features, never its predictor (confirmed in §2.3 and Appendix A.3). Its advantage over VLM-Forecast could simply come from a different visual encoder. So the one configuration that beats VLM-Forecast on CLEVRER-Next is not testing world-model forecasting at all; it is testing whether a self-supervised video encoder's features are better input for a learned forecaster than the VLM's own visual tokens.
Questions for authors
- How does FoReST compare to directly fine-tuning the VLM on CLEVRER-Next or ComPhy using question-answer supervision? This would help contextualize the value of the proposed unsupervised, plug-in approach relative to standard end-to-end adaptation.
- Why was a shuffled-update control not included for VLM-Forecast, given that both FoReST variants are evaluated against their own shuffled counterparts? This asymmetry limits the ability to fairly assess content specificity across methods.
- Could you report the inference-time computational cost—specifically FLOPs, latency, and peak memory—for FoReST-Latent, FoReST-Direct, and VLM-Forecast? This would clarify the practical trade-offs of using an external world model.
Dear reviewers, the authors have posted their responses to your reviews. Pls have a look and discuss.
Thanks, AC for #26470