acceptodds
Under review as a conference paper at ICLR 2027

WORLDPATCH: RETAINED PREDICTIVE STATES FOR ANTICIPATORY MULTIMODAL DECODING

Abstract

Predictive representation learning can use future observations to shape visual representations, yet the predicted state is often discarded before downstream decoding. Consequently, gains from future supervision do not reveal whether predictive training merely improves the representation of the present or whether the predicted state itself is useful when retained. We introduce WorldPatch, which predicts a short-horizon latent successor from the current image and retains it alongside the present representation for decoding. Paired future images supervise the successor during training but are unavailable at inference. We vary decoder access under matched predictive supervision to distinguish the effects of retaining the predicted successor from those of predictive training. On EPIC-KITCHENS-100, WorldPatch improves anticipatory decoding over an auxiliary-only model, with consistent gains on a second backbone and on Ego4D. Further analysis on the primary backbone shows that retaining the successor yields larger gains when current and future action labels differ. These findings show that future prediction can be useful beyond its role as a training objective: retaining its output can improve short-horizon multimodal decoding.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal language models, Predictive representation learning, Future state prediction, Anticipatory decoding, Retained predictive states

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Ey8gOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper tries to incorporate a predicted short-horizon future state into a language decoder to improve single-image action anticipation for language decoding. Instead of using future prediction purely as an auxiliary training objective, as is typically done in feature learning, the authors propose to use the future prediction head as context for improved decoding. Specifically, given a single current image, the proposed WorldPatch architecture extracts a set of visual state tokens representing the present scene and uses a lightweight residual network to predict a corresponding set of future state tokens. During training, these predicted tokens are aligned with visual tokens extracted from a future frame using Hungarian matching. At test time, because only the current image is seen, the decoder conditions on both the present and predicted future tokens (since the true future tokens are unknown).

The authors present controlled comparisons on EPIC-KITCHENS-100 and zero-shot transfer to Ego4D using two backbones (Slot-MLLM and Qwen2.5-VL), showing that the predicted future tokens improves next-action prediction over an auxiliary-only baseline that discards them before decoding. The difference is particularly noticeable when the current and future actions differ. The paper also reports broader evaluations on image understanding, image generation/editing, and multi-frame video QA.

Critical strengths

I think the idea is simple and interesting.

Also, the controlled anticipation study in Sections 3–5.4 is well-designed. The key objective is to analyze whether future-frame supervision helps multimodal decoding merely by improving the representation of the present, or whether the decoder actively benefits from having access to the predicted future state.

Within its experimental framework, the paper includes several baselines (primarily as ablations), such as matching the number of decoder tokens without future supervision, blocking task gradients to the predicted state, swapping in mismatched predictions from the same video at test time, and breaking down gains by whether the action changes between the current and future frames. There is also statistical analysis by repeating experiments across 8 paired seeds on two backbones to improve the empirical support for the main claims.

Critical weaknesses

While the internal ablations in Sections 5.1–5.4 are rigorous, my main concern is that the quality of the actual baselines that the paper compares against and the lack of SOTA methods as of 2026 makes it difficult to understand its true utility.

More specifically, I think the key weaknesses are as follows:

  1. Over-reliance on internal ablations and outdated or incomplete external baselines:
  • In the main anticipation experiments (Sections 5.1–5.4), the paper compares WorldPatch almost entirely against ablated versions of itself rather than external anticipation baselines or unmodified base models (for instance, Qwen2.5-VL is modified to bottleneck its visual features into 32 slot tokens rather than also showing the unmodified Qwen2.5-VL model's performance on the same task).
  • Where external baselines are shown, e.g., in Table 5 (image benchmarks) and Table 11 (Video-MME), the comparisons are either dated or missing key controls. Table 5 builds on a 2023 model stack (Vicuna-7B and DINOv2) and compares primarily against 2023–2024 models alongside Slot-MLLM, omitting stronger recent vision-language models.
  • In Appendix E (Table 11, Video-MME), the paper omits stronger models from the leaderboard and does not report the baseline Slot-MLLM or present-only control under the same 128-frame pooling setup. Without that matched control, it is difficult to evaluate how a frozen 2023 image-level backbone with simple average pooling and no video training or subtitles achieves 77.1% accuracy, beating dedicated 72B video models that use subtitles.
  1. Another issue is that the paper seems to use unreported training data and an extra 3D objective. In the main paper, WorldPatch is presented as a temporal prediction method trained with a task loss and a future-prediction loss. Section 5.5 (Table 5) then compares WorldPatch against published numbers for Slot-MLLM on standard image understanding and generation benchmarks, concluding that WorldPatch's retained future state improves general image tasks. However, Appendices B and G.1 show that the model evaluated in Table 5 was actually trained on a different data mix (adding Ego4D, NExT-QA, nuScenes, and ScanNet) and with an additional multi-camera 3D cross-view consistency loss that is never mentioned in the main text. Even more concerning, the ablation table in Appendix B (Table 7) shows that when this undisclosed cross-view consistency loss is removed, leaving the actual WorldPatch objective described in the main paper, the model's scores on GQA, MME-P, and GenEval drop below the baseline Slot-MLLM.
Questions for authors

My main questions are related to the weaknesses above, especially on the lack of recent baselines and state-of-the-art VLMs as of September 2026.

In addition, I also have a question about the utility of Hungarian matching. The paper motivates this by stating that the visual state slots are an unordered collection with no fixed index semantics. Yet the predictor itself is a residual layer that updates each slot index-by-index from a fixed set of learned queries, and the resulting tokens are fed in a fixed order to the language decoder. In fact, Appendix A.2 shows that shuffling the slot order at test time noticeably hurts performance. Why is unordered set matching necessary during training if the architecture and decoder treat the slots as ordered throughout?

1 comment
Reviewer_Ey8gOpenReview ↗

keeping my score

The authors did not submit a rebuttal, and all other reviewers are also negative. As a result, I'm keeping my weak reject rating. In addition, I used this opportunity to clarify some of the main points in my review and my overall impression.

Reviewer_MVA9OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies training anticipative features within multimodal LLMs. The study focuses mostly on finding the best training mechanism that yields anticipative present features within encoded present features. The method is applied to action anticipation, image understanding and image generation.

Critical strengths

The amount of presented experiments is commendable. These results can serve as a basis to present interesting findings.

Critical weaknesses

I think that a substantial rework might yield an interesting paper, considering the amount of presented experiments. Primarily, I expect that readers might benefit from reading accuracy measurements alongside log-likelihood measurements. In other words, I that find the likelihood measurements insufficient in this analysis. For example, the implication of results from Table 1 are not sufficiently discussed, nor is it possible to estimate the result differences within table entries. Secondly, the paper follows a common structure, but the content is confusing at times. For example, the second introduction paragraph reads more like related work, and the placement of the paper is not discussed in sufficient detail at all (e.g. which tasks/datasets/problems are addressed), also I expected the second paragraph in 5.1 would discuss Table 1 results but instead mixes the training details across multiple datasets, which is confusing. Thirdly, I would encourage the authors to include a paragraph that connect this method with existing work, and highlight their methodological contributions as delta wrt. published work.

Questions for authors

Can you include accuracy/recall@5 (or a relevant success measurement) alongside all likelihood measurements? Can you establish a baseline from the literature, measure its likelihood, and report accuracy of your re-implementation?

Reviewer_6zndOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The author introduces a common but difficult question about whether gains from future supervision as part of the decoder input can improve the action anticipation. They introduce WorldPatch, as a pipeline to predict a short horizon latent successor from the current image and retain it alongside the present representation for decoding. In addition to that, they claim that paired future images provide supervision during training but not for inference.

In order to identify the effects of keeping this predicted successor compared with normal training, they compare models trained with same kind of predictive supervision but using different access mechanisms for the predicted states. Also, they conduct experiments on EPIC-KITCHENS-100, and it improves future Gold-LL over the auxiliary-only baseline by 0.032 on Slot-MLLM and 0.026 on Qwen2.5-VL. They also claim the improvement reflected in future-action exact-match accuracy by 6.0 and 4.0 percentage points and shows consistent transfer performance on Ego4D without extra training. They also conduct separate experiments on broader multimodal tasks including image understanding, generation, and editing.

Critical strengths

• The author claims that retraining the predicted successor improves anticipatory decoding beyond predictive training alone and their comprehensive experiments partly show the performance by showing on Table 2, On EPIC-KITCHENS, their different Slot-MLLM A/R future-action EM is 42.0/48.0%, a +6.0-point; and Qwen2.5VL is 50.0/54.0. On Ego4D, both backbones improve by 2.5 points. In addition, on Table 3 they show future target alignment and decoder use that their native successor did well on compared with other controlled states, which supported their claim that their design is effective.

• WorldPatch shows not only their result compared with different state but also shows across boarder usage for its architecture on other downstream tasks, which seems the gains extend beyond anticipatory decoding.

• The paper is well-written and its central attribution’s comparison is well-controlled.

Critical weaknesses

• The main result didn't fully support its generalizability claim. The Table 5 used as the claim for broader evaluation are not aligned with the previous paper. The authors claimed that “WorldPatch outperforms the full Slot-MLLM model on all eight image-understanding benchmarks”. However, in the Slot-MLLM paper, which author mainly compare with, their 7B model results are Slot-MLLM(7B) 57.3 1225.2 83.0 66.8 33.2 58.7 66.1 on GQA MME-P POPE MMB MMMU SEED-IMG NaturalBench, which nearly all baseline number are higher than WorldPatch’s number. In addition to that, the same pattern happens for text-to-image and editing benchmark, which T2I-CompBench, GenEval GenAI-Bench MagicBrush EVR’s number are not matched and Slot-MLLM did significantly better on T2I-CompBench, GenAI-Bench, GenEval but only MagicBrush took the lead. Based on the current result, it’s seems the result is overstated compared with baselines the same Vicuna-7B backbone.

• On Table 2, the author uses A and R setting to make the claim that “Retaining the predicted successor improves anticipatory decoding beyond predictive training alone” However, there are multiple setting changed from A to R and adding G control on table 2 might makes the pictures more clear on whether the second backbone independently helps over nonpredictive extra context.

• The analysis using Table 1 and 3 didn’t fully support that the general future semantics is independent to the learned representation. The author makes their claim that “A predictive-state interpretation requires evidence that the successor reflects the paired future rather than merely copying the present, and that its effect on decoding depends on its content.”

Questions for authors
  1. How Table 5 helps us successfully shows the insights that we can leverage WorldPatch on those tasks given the facts that those numbers are not matched? Can you re-evaluate the baseline under your matched protocols and compare with the paper you compared with and make analysis for why there is a difference?

  2. Can you add a G condition states with the same context and training budge as your R condition using both future Gold-LL and future EM to make the claim more clear?

  3. Can you make a clarification on current Table 3 state description for what is the difference among states “native successor”, “post-abstraction current state”, “training-target medoid”?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.