- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper tries to incorporate a predicted short-horizon future state into a language decoder to improve single-image action anticipation for language decoding. Instead of using future prediction purely as an auxiliary training objective, as is typically done in feature learning, the authors propose to use the future prediction head as context for improved decoding. Specifically, given a single current image, the proposed WorldPatch architecture extracts a set of visual state tokens representing the present scene and uses a lightweight residual network to predict a corresponding set of future state tokens. During training, these predicted tokens are aligned with visual tokens extracted from a future frame using Hungarian matching. At test time, because only the current image is seen, the decoder conditions on both the present and predicted future tokens (since the true future tokens are unknown).
The authors present controlled comparisons on EPIC-KITCHENS-100 and zero-shot transfer to Ego4D using two backbones (Slot-MLLM and Qwen2.5-VL), showing that the predicted future tokens improves next-action prediction over an auxiliary-only baseline that discards them before decoding. The difference is particularly noticeable when the current and future actions differ. The paper also reports broader evaluations on image understanding, image generation/editing, and multi-frame video QA.
Critical strengths
I think the idea is simple and interesting.
Also, the controlled anticipation study in Sections 3–5.4 is well-designed. The key objective is to analyze whether future-frame supervision helps multimodal decoding merely by improving the representation of the present, or whether the decoder actively benefits from having access to the predicted future state.
Within its experimental framework, the paper includes several baselines (primarily as ablations), such as matching the number of decoder tokens without future supervision, blocking task gradients to the predicted state, swapping in mismatched predictions from the same video at test time, and breaking down gains by whether the action changes between the current and future frames. There is also statistical analysis by repeating experiments across 8 paired seeds on two backbones to improve the empirical support for the main claims.
Critical weaknesses
While the internal ablations in Sections 5.1–5.4 are rigorous, my main concern is that the quality of the actual baselines that the paper compares against and the lack of SOTA methods as of 2026 makes it difficult to understand its true utility.
More specifically, I think the key weaknesses are as follows:
- Over-reliance on internal ablations and outdated or incomplete external baselines:
- In the main anticipation experiments (Sections 5.1–5.4), the paper compares WorldPatch almost entirely against ablated versions of itself rather than external anticipation baselines or unmodified base models (for instance, Qwen2.5-VL is modified to bottleneck its visual features into 32 slot tokens rather than also showing the unmodified Qwen2.5-VL model's performance on the same task).
- Where external baselines are shown, e.g., in Table 5 (image benchmarks) and Table 11 (Video-MME), the comparisons are either dated or missing key controls. Table 5 builds on a 2023 model stack (Vicuna-7B and DINOv2) and compares primarily against 2023–2024 models alongside Slot-MLLM, omitting stronger recent vision-language models.
- In Appendix E (Table 11, Video-MME), the paper omits stronger models from the leaderboard and does not report the baseline Slot-MLLM or present-only control under the same 128-frame pooling setup. Without that matched control, it is difficult to evaluate how a frozen 2023 image-level backbone with simple average pooling and no video training or subtitles achieves 77.1% accuracy, beating dedicated 72B video models that use subtitles.
- Another issue is that the paper seems to use unreported training data and an extra 3D objective. In the main paper, WorldPatch is presented as a temporal prediction method trained with a task loss and a future-prediction loss. Section 5.5 (Table 5) then compares WorldPatch against published numbers for Slot-MLLM on standard image understanding and generation benchmarks, concluding that WorldPatch's retained future state improves general image tasks. However, Appendices B and G.1 show that the model evaluated in Table 5 was actually trained on a different data mix (adding Ego4D, NExT-QA, nuScenes, and ScanNet) and with an additional multi-camera 3D cross-view consistency loss that is never mentioned in the main text. Even more concerning, the ablation table in Appendix B (Table 7) shows that when this undisclosed cross-view consistency loss is removed, leaving the actual WorldPatch objective described in the main paper, the model's scores on GQA, MME-P, and GenEval drop below the baseline Slot-MLLM.
Questions for authors
My main questions are related to the weaknesses above, especially on the lack of recent baselines and state-of-the-art VLMs as of September 2026.
In addition, I also have a question about the utility of Hungarian matching. The paper motivates this by stating that the visual state slots are an unordered collection with no fixed index semantics. Yet the predictor itself is a residual layer that updates each slot index-by-index from a fixed set of learned queries, and the resulting tokens are fed in a fixed order to the language decoder. In fact, Appendix A.2 shows that shuffling the slot order at test time noticeably hurts performance. Why is unordered set matching necessary during training if the architecture and decoder treat the slots as ordered throughout?
keeping my score
The authors did not submit a rebuttal, and all other reviewers are also negative. As a result, I'm keeping my weak reject rating. In addition, I used this opportunity to clarify some of the main points in my review and my overall impression.