acceptodds
Under review as a conference paper at ICLR 2027

Prediction Can Benefit Understanding: Visual Prediction as Complementary Supervision for Video LLMs

Abstract

Learning to predict how the visual world evolves is a powerful source of self-supervision, yielding strong video representations and world models that support planning and control. Video large language models (video LLMs), however, are trained almost entirely with language supervision; in the standard recipe, the video serves only as input and is never itself predicted. Prior work has unified visual generation and understanding in single models, yet whether learning to predict the visual world benefits understanding has not been systematically reported. In this work, we systematically investigate how visual prediction and understanding interact within a video LLM, asking whether learning to predict the visual world improves understanding. During model training, we remove a segment from a video and train the model to predict the visual features of that segment from the rest of the video and a description of the segment. A counterpart trained on the same data with language supervision alone serves as the control. At all three scales, from 0.8B to 4B, learning to predict improves every video understanding and temporal grounding benchmark we evaluate, with gains at 0.8B and 2B of up to 3.3 points on MVBench, 3.3 points on TempCompass and 3.4 points in mean grounding mIoU. Controlled ablations at multiple levels confirm that the gain is robust across design choices and data sources. These results show that visual prediction can serve as a general supervision paradigm for video LLMs, complementing language with a signal drawn from the visual world itself.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Large Language Models, Complementary Visual Supervision, Temporal Video Understanding, Temporal Grounding, Language-Conditioned Visual Prediction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_V4AhOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

Although the paper claims to investigate whether 'prediction can benefit understanding,' it does not actually conduct a substantive study on this underlying mechanism. Instead, it merely demonstrates that adding a prediction loss during the fine-tuning stage yields better performance than standard fine-tuning.

Critical strengths

The paper is quite comprehensive; setting aside the research question and its value.

Critical weaknesses
  1. Although the paper claims to investigate whether "prediction can benefit understanding," the experiments fail to explain why this phenomenon occurs. The authors merely list empirical improvements without providing a substantive analysis or deeper insights into the underlying mechanisms driving this benefit.
  2. The baseline comparisons are notably weak and fail to adequately engage with or compare against existing state-of-the-art related work in the field.
Questions for authors

Refer to weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer V4Ah

We thank the reviewer for the time spent on our paper and for noting that it is comprehensive. We read the review as raising two concerns: that the paper establishes an effect without explaining it, and that the baselines are not strong enough. Both are also raised by Reviewers 5Emt and sx36, and we answer them directly below.


1. Why does prediction benefit understanding?

Concern: the experiments list empirical improvements without a substantive analysis of the mechanism.

Establishing the effect under controlled conditions is itself a contribution, because prior evidence is mixed, and the submission also analyses where the effect comes from. Prior evidence on whether visual prediction helps a model understand is mixed (Section 5 of the paper):

  • In Cambrian-S, next-frame latent prediction leaves general video benchmarks essentially unchanged in its own controlled sweep.
  • Studies of unified models find small or architecture-dependent effects of generation on understanding.
  • Auxiliary prediction losses can improve while downstream performance does not.

Against this background, our compute-matched comparison shows a consistent effect. The objective improves 27 of 27 paired comparisons across nine benchmarks and three independently trained scales, while the baseline receives about one third more text-supervised passes.

The submission also contains four controlled ablations, each varying one aspect of the objective with the backbone, the language data and the compute fixed. They cover where the supervision is applied (read-out depth), how strongly (loss weight), what conditions the prediction (the description), and which data provide the prediction examples (grounded descriptions or captions). Together with the measurement of prediction error, they show three things:

  • The gain is robust to depth and weight, since every setting improves on SFT.
  • About half of the gain depends on the language that selects the removed span.
  • The gain is not explained by how accurately the target is predicted.

We now add three analyses of the trained models that show how the objective acts (reply to Reviewer 5Emt, Tables R1–R3):

  • It improves both anticipation and temporal-order questions at all three scales, so the gain is a broad improvement in temporal understanding.
  • It keeps the LLM's visual representation closer to the structure of its visual input in the upper layers, where language-only training erodes that structure.
  • Relative to other windows of the same video, it ties each description to its own span more strongly than SFT does.

With Table 5, these results show that the benefit comes from the visual target acting on the shared representation, not merely from a generic regularisation effect.


2. Baselines and related work

Concern: the baseline comparisons are weak and do not engage with state-of-the-art related work.

The primary baseline is the controlled one, and it is strong. Base+SFT is trained on the same data with the same number of forward passes, so it is the only comparison that isolates the objective. It is also a strong model. At 0.8B and 2B it exceeds the released Qwen3.5 Instruct models on all nine benchmarks, for example 76.9 vs 63.3 on NExT-QA and 43.4 vs 32.2 in mean grounding mIoU at 0.8B.

Prior visual objectives are addressed in our pipeline and through their published controlled results. With identical backbone, data and compute, removing the span without a description, which gives the model only the target's position as masked video modelling does, reaches 59.5 on average (ours 60.8, SFT 58.0). Our reply to Reviewer 5Emt (W1) measures in our models the property that ROSS and PRe preserve, and discusses the controlled results of ST-LLM, Cambrian-S and a ROSS-style objective; Part 1 of the General Response compares them side by side.

Published state-of-the-art models are included in Table 2. Table 2 lists twelve published video LLMs, including 7B models trained with reinforcement learning (Video-R1, VideoP2R, APPO) and the grounding specialist TimeLens-7B. Our 4B model reaches 70.4 on MVBench and 77.3 on TempCompass, the highest in the table. Our 2B model exceeds TimeLens-7B on all three TimeLens benchmarks (52.5 / 50.4 / 59.3 vs 48.8 / 46.2 / 56.0). TimeLens-7B is trained on TimeLens-100K, which is also in our mixture; our 2B Base+SFT is comparable to it (47.6 / 48.5 / 56.4), and the objective adds 1.9 to 4.9 points on each benchmark.

If the reviewer has specific methods in mind for comparison, we would be glad to discuss them. We hope that the new evidence addresses the reviewer's concerns, and would be grateful if the reviewer would reconsider the assessment in light of it.

Reviewer_V4AhOpenReview ↗

Response to authors

I believe the authors have clearly recognized the limitations of this work and have made an effort to address them in the rebuttal. I appreciate the authors' efforts, so I may raise my score to 2. However, the paper still lacks sufficient interpretability. Furthermore, in Section 4, the authors attempt to analyze the claim that "prediction benefits understanding" using rather disorganized and elementary concepts, such as loss weights and the functions of different layers in neural networks. It is clear that they have missed the point here, which prevents me from raising the score to 3.

AuthorsOpenReview ↗

Response to Reviewer V4Ah's follow-up

We sincerely thank the reviewer for the prompt reply during the discussion and for being willing to reconsider the score. We also thank the reviewer for the precise criticism of Section 4, and we agree with it: the loss weight and the read-out depth are design choices. They show that the effect does not depend on one setting, but they do not explain why it occurs. Below we state the mechanism we propose, the evidence for each of its steps, and its limits.

The proposed mechanism. In SFT, the visual tokens are trained only through the text the model writes, so the representation shared with text generation keeps from the video what text generation needs. Our objective adds a signal defined on the video itself: an event named by a sentence is removed, and the model must produce the visual features of that event from the rest of the video. If this is how prediction helps understanding, three things should hold:

StepWhat should holdWhat we measuredResult
1. A signal on the videoThe visual structure that text-only training erodes is keptSimilarity structure of the hidden states against the visual input (Table R2)SFT moves the upper layers away from it; with our objective they stay close to the released model
2. The named content mattersRemoving the description, with everything else fixed, reduces the gainDescription ablation (Table 5)The gain over SFT falls from +2.8 to +1.5 points
3. The kind of target mattersObjectives with other targets do not help, even when they keep the visual structureThree published objectives in our pipeline (Table R4, in our new comment to the Area Chair and all reviewers) and two controls (in our follow-up to Reviewer sx36)Targets on the visible tokens (PRe), the next frame (Cambrian-S), randomly dropped tokens (ST-LLM), the span of another video, or the description as text all fall below SFT; PRe keeps the most visual structure yet falls furthest

Step 3 also answers the reviewer's original concern about baselines: under the same backbone, data and compute, three published visual objectives fall below SFT, while ours is the best of these models on every benchmark.

The downstream pattern agrees with this account. Questions that ask what happens next or in which order events occur improve at all three scales (STAR, Table R1: +2.40 / +1.12 / +4.81 on Prediction and +2.65 / +1.90 / +1.12 on Sequence at 0.8B / 2B / 4B), and these are the questions that relating one event to the rest of the video should help most.

Limits of the account. We do not claim that the model reconstructs the removed content: RQ4 shows that a lower prediction error does not come with a larger gain, so the objective acts through the representation it shapes, not through the accuracy of its predictions. About half of the gain does not depend on the description (Table 5) and comes from the visual signal itself. Step 1 describes what changes in the representation, while Steps 2 and 3 test what the gain depends on: the content named by the description, and a target on a removed event rather than on the visible tokens, the next frame, dropped tokens, another video or text. We therefore offer these steps as consistent evidence for the mechanism, not as a proof of it.

We are grateful to the reviewer for engaging with our work during the discussion. The reviewer's comments have shown us how the analysis should be organised. In the final version of the paper, Section 4 will be organised around this account: it will open with the mechanism and the evidence for each step, bring the comparisons of Table R4 and the controls into the main text, and condense the loss-weight and read-out-depth ablations into a short paragraph on design choices. We would welcome any further suggestion of an analysis that the reviewer would find convincing.

Reviewer_sx36OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper asks whether learning to predict visual content helps video LLMs understand video. During SFT, some training examples remove a temporal span from the video and ask the model to predict the frozen-encoder features of that span from the remaining video and a description of the span; predictions are read from placeholder tokens at the middle layer of the LLM with an L1 loss. Inference is unchanged. On Qwen3.5 at 0.8B, 2B and 4B, compared with a compute-matched text-only SFT baseline, the method improves video QA and temporal grounding benchmarks.

Critical strengths
  1. The motivation is good. Video LLMs are trained almost only with text targets, and the paper brings in a self-supervised visual prediction signal that needs no extra annotation. To some extent this borrows the scalability of self-supervised video pre-training.

  2. The method is simple, adds no inference cost, and the gains are consistent in direction across three scales and nine benchmarks. The authors also honestly report that lower prediction loss does not mean better downstream results.

Critical weaknesses
  1. The paper does not show that the gain comes from learning to predict the visual world. Removing the description still gives about half of the gain; the model with description has a higher held-out prediction error yet better downstream results; deeper read-outs predict better but do not help more. These are also consistent with a generic regularization effect, or with the baseline over-training on text (2.0 vs. 1.5 epochs). Key controls are missing: random or shuffled targets, the same span-removed input with a text target, and a baseline with matched text passes at 0.8B/2B.

  2. Several claims go beyond the evidence. At 4B, Base+SFT is below the released Instruct model on grounding (40.9 vs. 46.4) and close to Base on ANet, so the large 4B gains may mostly fix a weak SFT baseline. Table 2 compares with models on older backbones while the training data includes the train splits of NExT-QA, STAR, Charades, YouCook2 and TimeLens-100K, so it mainly reflects backbone and data, as Appendix C admits.

  3. The results are not statistically solid. Every number comes from a single run without variance, while most gains are 1–3 points and differences between ablation settings are often below 1 point (e.g. 60.1 vs. 60.8 for L/4 vs. L/2 depth). The read-out layer and λ are also chosen by looking at the same test benchmarks.

Questions for authors

N/A

7 comments
AuthorsOpenReview ↗

Response to Reviewer sx36 (Part 1/3)

We thank the reviewer for the positive and precise review, and for recognising the motivation, the simplicity of the method, the consistency of the gains, and our report that a lower prediction loss does not mean better downstream results. The controls the reviewer lists all address one question: is the gain specific to the objective, or a generic effect of training? We answer it below with the evidence that bears on each control. The new analyses we cite are in Tables R1–R3 of our reply to Reviewer 5Emt.


W1. Is the gain due to learning to predict the visual world?

Concern: removing the description still gives about half of the gain, the model with the description has a higher held-out prediction error but better downstream results, and deeper read-outs predict better without helping more. This is also consistent with generic regularisation or with the baseline over-training on text. Random or shuffled targets, the same span-removed input with a text target, and a baseline with matched text passes are missing.

The evidence shows that the gain is neither a generic regularisation effect nor an artefact of over-training the baseline on text. We take the three requested controls in turn, and then explain why the three observations are expected for an auxiliary prediction objective.

(1) Random or shuffled targets. Such a control tests whether the gain depends on the content of the target. Two results in our setting answer this directly.

  • Part of the gain depends on what the description says about the target (Table 5). The description-removed arm and ours share the loss, its weight, the read-out, the data and the number of passes, and differ only in the conditioning text. The average gain changes from +1.5 to +2.8, on NExT-QA from 0.0 to +1.2 and on STAR from +1.1 to +3.0. The description can matter only through the information it carries about the target; with a random or shuffled target, it carries none.
  • The objective binds each description to its own span. In the trained model, the hidden state of a description at the last layer is closer to the visual tokens of its own span than to other windows of the same video by 0.0387, against 0.0338 after SFT, and its attention to the span rises from 1.54 to 1.59 times the uniform level (Table R3 in our reply to Reviewer 5Emt). A random or shuffled target carries no information about which part of the video a description refers to, so it gives the model no reason to develop this span-specific alignment.

(2) The same span-removed input with a text target. In both arms, every sentence–span pair used for prediction is also in the language-modelling mixture as a text target (Appendix H), and the baseline sees these text targets more often (2.0 against 1.5 epochs). What our arm adds is visual supervision of the removed span, not more language supervision about it.

(3) Matched text passes. Our comparison holds the number of forward passes fixed, which is the standard way to compare an auxiliary objective without granting it extra compute. Under this protocol the baseline already receives one third more language-modelling passes than our arm (about 1.15M against 864K, or 2.0 against 1.5 epochs), so the comparison favours the baseline in text supervision. Over-training on text would make the baseline sensitive to the number of text passes, and it is not: between its 4k- and 6k-step checkpoints the 4B SFT baseline changes no benchmark by more than 0.4 points (Appendix D). Nor is the baseline degraded: at 0.8B and 2B it exceeds the released Instruct model on all nine benchmarks.

AuthorsOpenReview ↗

Response to Reviewer sx36 (Part 2/3)

W1 (continued). The three observations

Regarding the three observations, the reviewer is right that how accurately the target is fitted does not explain the gain, and we reported them in this sense (RQ2, RQ4). This is what one expects of an auxiliary objective, which acts on the model through the gradients it sends into the shared representation, not through the accuracy of its own output.

  • Half of the gain remains without the description. The description-free arm still predicts the removed span from its visual context, as in masked video modelling, so its gain is the expected effect of a target given only by its position. The description then adds a further 1.3 points, which a content-agnostic regulariser would not produce.
  • Higher prediction error for the model trained with descriptions. We report this ourselves (RQ4), both in each model's own training format and on identical held-out inputs. It is the clearest instance of the dissociation below: the description raises the downstream gain without lowering the prediction error, so the gain does not come through the accuracy of the prediction.
  • Deeper read-outs fit better but help less. A deeper read-out leaves more layers to fit the target, so a lower loss there constrains the shared middle representation less. This matches the middle-layer preference reported by PRe and VIRAL.

The same dissociation between pretext loss and representation is well established in self-supervised learning: in MAE (He et al., 2022), a high masking ratio makes the prediction problem harder and yields better representations. For auxiliary losses in language and multimodal models, it is reported by Cosine Misleads (Zhang et al., 2026) and Representation Without Reward (Sengupta, 2026), both cited in Section 5.

In short, about half of the gain depends on the language that selects the target, the gain comes with a span-specific alignment between descriptions and video, and it is measured against a baseline that receives more text supervision and is stable across checkpoints. Neither generic regularisation nor over-training of the baseline accounts for it. Our claim is that training with this visual prediction objective improves understanding. The paper does not claim that the gain follows the accuracy of the prediction itself, and RQ4 reports that it does not.


W2. Claims that go beyond the evidence

Concern: at 4B, Base+SFT is below the released Instruct model on grounding and close to Base on ANet, so the large 4B gains may mostly fix a weak SFT baseline; Table 2 compares with models on older backbones while our training data includes the training splits of several benchmarks, so it mainly reflects backbone and data.

4B: the 4B comparison remains paired and controlled, and our headline claims already rest on 0.8B and 2B. The submission reports the weaker 4B grounding baseline itself (RQ1): with one learning rate and data budget shared by all scales, the 4B model moves least from its base during SFT. This follows from the same protocol that answers W3, under which no setting was tuned for any scale. Three facts show that the 4B gains are not explained by fixing a weak baseline:

  • A gain that only fixes a weak baseline would vanish where the baseline is strong. At 0.8B and 2B the SFT baseline exceeds the released Instruct model on all nine benchmarks (mean grounding mIoU 43.4 vs 32.2 and 46.6 vs 38.3), and the objective improves all nine at both scales, by +1.2 to +5.2. The headline numbers in the abstract are taken from these two scales.
  • At 4B the SFT baseline is at or above the Instruct model on six of the nine benchmarks (MVBench, TempCompass, NExT-QA, STAR, Charades-STA, YouCook2), and the objective improves all six (+2.5, +1.5, +0.5, +2.2, +2.1, +0.5).
  • Where the 4B baseline is low, both arms still see the same data and compute. On ActivityNet, SFT moves little from Base (39.7 vs 39.3), while adding the objective reaches 49.0, above the Instruct model (48.8). The larger magnitudes there reflect the headroom, which is why they are not used in our headline numbers; the direction matches every other benchmark and scale.
AuthorsOpenReview ↗

Response to Reviewer sx36 (Part 3/3)

W2 (continued). Table 2

Table 2: it supports a claim of competitiveness, while the evidence for the objective is the paired comparison, which the training mixture does not affect. Both arms of Table 1 are trained on exactly the same data, including the training splits the reviewer lists, and every evaluation uses held-out test splits or the TimeLens re-annotated test sets, which do not overlap with the training data (Appendix H). Appendix C states that much of the absolute level comes from the backbone, and the paper does not attribute that level to the objective. The claim in RQ1 that visual prediction contributes substantially is narrower, and the TimeLens comparison supports it with shared grounding data. TimeLens-7B is trained on TimeLens-100K, the grounding data the reviewer lists, and evaluated on the same TimeLens benchmarks. Our 2B Base+SFT is comparable to it (47.6 / 48.5 / 56.4 vs 48.8 / 46.2 / 56.0 on Charades-STA / ActivityNet / QVHighlights), and adding the objective raises each of the three by 1.9 to 4.9 points (to 52.5 / 50.4 / 59.3), above TimeLens-7B on all three.


W3. Statistical reliability

Concern: every number comes from a single run; most gains are 1–3 points; differences between ablation settings are often below one point; the read-out layer and λ are chosen on the test benchmarks.

Single runs. The result is replicated across ten separately trained models with the objective (eight configurations at 0.8B, including the default, and one run each at 2B and 4B), which is the evidence that matters for a comparison of training recipes. Each configuration is trained once, the standard protocol for comparing training recipes of multimodal LLMs, and within this protocol the comparison is repeated many times:

  • The three scales are trained independently, each with its own SFT baseline, so Table 1 contains three independent paired comparisons. The objective improves all nine benchmarks in each of them, 27 of 27, at no inference cost.
  • At 0.8B the objective is trained eight times, including the default, with different read-out depths, loss weights, conditioning and data (Tables 3–6). Each is a separate training run, and every one improves the seven-benchmark average over SFT, by +1.1 to +2.8.
  • The baseline is neither weak nor an unlucky run. At 0.8B and 2B it exceeds the released Instruct model on all nine benchmarks, and between its 4k- and 6k-step checkpoints the 4B baseline changes no benchmark by more than 0.4 points (Appendix D).

Run-to-run variation does not explain why ten separately trained models all land above their matched baselines, at every scale. We report three training runs per scale in our follow-up of 17 November.

Differences below one point. Our conclusions from the ablations rest on differences above one point, on average or on the benchmarks they concern.

  • All eight 0.8B configurations in Tables 3–6 improve the seven-benchmark average over SFT, by +1.1 to +2.8. They cover read-out depth, loss weight, the description and the data source.
  • The description adds 1.3 points, and removing it lowers six of the seven benchmarks (Table 5).
  • λ = 2 is 1.7 points below λ = 1 and lower on all seven benchmarks (Table 4).
  • Read-outs at 3L/4 and L are 1.2 and 1.0 points below L/2, although they reach a lower prediction loss (Table 3, Figure 5a). L/2, the default, has the highest average and the highest score on all three grounding benchmarks; its margin over L/4 is 0.7 points on average but 2.2 on Charades-STA and 2.1 on QVHighlights.

Hyper-parameters chosen on the test benchmarks. Selection does not account for the gains. The defaults, half depth and λ = 1, were fixed at 0.8B and transferred unchanged to 2B and 4B, where no setting was tuned, so the gains at those two scales involve no per-scale selection. At 0.8B, the least favourable configuration we tested (λ = 2) still improves on SFT by 1.1 points, so the gain does not depend on choosing the best setting; selection affects only how large the reported 0.8B gain is.

We thank the reviewer for the careful reading and the precise list of controls. We hope that our responses address the reviewer's concerns, and we are glad to answer further questions.

Reviewer_5EmtOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether visual prediction, used as an auxiliary loss during fine-tuning, improves understanding in VideoLLMs, and proposes a latent prediction loss for this purpose. Given a temporal grounding sample, the video segment described by the grounding sentence is removed from the input. The LLM then receives the remaining visual tokens, the grounding sentence, and one learnable placeholder token for each removed visual token. The placeholders' hidden states at the middle layer are trained to reconstruct the removed visual tokens, i.e., the frozen visual encoder's outputs that would otherwise be fed to the LLM, using an L1 loss on layer-normalized features. Added to the standard cross-entropy loss under matched compute, this objective improves four video QA and five temporal grounding benchmarks, mostly by 1–3 points, on Qwen3.5 at 0.8B, 2B, and 4B.

Critical strengths
  1. An interesting and clearly presented idea. Turning existing video–language data, such as temporally grounded descriptions, into a visual prediction target is an interesting and intuitive way to add visual supervision to video LLM training. The paper is well organized and easy to follow, with the method, training setup, and ablations clearly described.

  2. A controlled comparison with consistent gains at no inference cost. The comparison with the cross-entropy-only baseline is carefully matched in backbone, data, steps, and forward passes, and the baseline even receives about one third more text-supervised passes (Table 11). Under this setting, the objective improves all nine benchmarks at all three scales without adding any module or inference cost, which makes it a simple and practical addition to video LLM fine-tuning.

Critical weaknesses

While the paper presents an interesting idea, two issues regarding its contribution and analysis remain to be resolved:

  1. Unclear differentiation from prior visual prediction objectives.

    • Claimed novelty. The paper presents itself as "the first systematic study of whether visual prediction improves understanding in video LLMs" (Sec. 1). However, similar objectives already exist for MLLMs, such as reconstructing visual tokens from the LLM's hidden states (Ross [1]) and predicting the initial visual features from intermediate layers (PRe [2]). Within video LLMs, ST-LLM [3] reports gains from masked video modeling on MVBench, and Cambrian-S [4] ablates its next-frame latent prediction loss on general video benchmarks.
    • No controlled comparison. The paper distinguishes its objective from these works only by listing their properties (Appendix A, Table 7), and compares only with the cross-entropy-only baseline. Without reproducing prior objectives under the same setting, it remains unclear whether the proposed design, i.e., removing a text-described span and predicting it through placeholder tokens, is more effective.
  2. Lack of in-depth analysis of the source of the gains. The paper attributes the gains to learning to predict the visual world, claiming that the results "point toward unifying video understanding and world modelling within a single model" (Sec. 1), but provides no direct evidence for this mechanism. The task infills the features of a span from the surrounding frames and its description, which is closer to text-conditioned feature reconstruction than to world modeling. The gains may instead come from (i) preserving visual information across the LLM's layers, as reported in prior work [1, 2], or (ii) cross-modal alignment between the description and the corresponding visual tokens. It therefore remains unclear whether the gains come from visual prediction, as claimed, or from these effects, which limits the insight the study provides.

[1] H. Wang et al. Reconstructive Visual Instruction Tuning. ICLR, 2025.

[2] E. Wang et al. Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models. CVPR, 2026.

[3] R. Liu et al. ST-LLM: Large Language Models Are Effective Temporal Learners. ECCV, 2024.

[4] S. Yang et al. Cambrian-S: Towards Spatial Supersensing in Video. arXiv:2511.04670, 2025.

Questions for authors
  1. What are the key differences from the objectives in [1–4], and how does the proposed objective compare with them under the same backbone, data, and compute? This would show whether removing a text-described span and predicting it through placeholder tokens is more effective than these existing designs.
  2. Do the gains come from visual prediction, from preserving visual information across the LLM's layers, or from cross-modal alignment between the description and the corresponding visual tokens? For example, are the gains larger on questions that require anticipating future events (e.g., the Prediction questions in STAR), is visual information better preserved across layers, as measured in PRe [2], and do the description tokens attend more to, or become more similar to, the visual tokens of the corresponding span after training?
8 comments
AuthorsOpenReview ↗

Response to Reviewer 5Emt (Part 1/4)

We thank the reviewer for the detailed and constructive review, and for finding the idea interesting and clearly presented and the comparison carefully matched. We also thank the reviewer for pointing us to ST-LLM, which we had not cited. We answer the two weaknesses and the two questions in turn. The analyses the reviewer suggested are reported in Tables R1–R3 below, and a side-by-side comparison of the objectives is in Part 1 of the General Response.


W1(a). Differentiation from prior visual prediction objectives

Concern: ROSS, PRe, ST-LLM and Cambrian-S already use similar objectives, so the claim of "the first systematic study" is not justified.

The claim concerns the study rather than the objective, and we believe it holds in the scope in which the paper states it: whether visual prediction improves understanding in video LLMs. The paper does not claim the first visual objective for MLLMs. Section 1 cites feature-reconstruction and prediction objectives as the motivation for the question, and Appendix A positions our objective among related visual prediction objectives. What the paper claims is the first systematic test of the question: a language-only counterpart trained on the same data with the same compute, at three scales, on nine benchmarks in two task families, with controlled ablations of read-out depth, loss weight, conditioning and data source. The four works differ from ours both in what they test and in the objective itself:

  • ROSS and PRe study image MLLMs, and their targets are visual tokens present in the input. ROSS reconstructs the input-image latents by denoising, and PRe predicts the pre-LLM features of the same visual tokens from a middle layer; both counteract the loss of visual information inside the LLM. Our target is not in the input, so it cannot be met by carrying visible tokens through the network.
  • ST-LLM's masked video modelling is a consistency objective. Under masking, the LLM's outputs at the unmasked positions are matched to its own outputs on the full input; the content of the masked tokens is never a target. Its reported gain (+1.1 on MVBench, Table 4 of ST-LLM) adds masking and the loss together, on one benchmark.
  • Cambrian-S predicts the next frame's encoder feature as one component of a system for spatial supersensing. Its own controlled sweep (Appendix F.1, Table 16), with a frozen encoder as in our setting, finds that at the default weight next-frame prediction leaves general video benchmarks essentially unchanged (VideoMME +0.5, EgoSchema +0.1, PerceptionTest −0.2) and lowers VSI-Bench by 1.4; at weight 1.0 all four drop. Its reported benefit is on VSI-SUPER, through surprise-based memory.

For video LLMs, the prior evidence on the question is therefore limited and mixed. Under matched compute, our objective improves all nine benchmarks at all three scales (Table 1). One reason for the difference lies in the target: a next frame is largely revealed by its neighbours, whereas a described span of several seconds is not. Consistent with this, the language that selects the span accounts for about half of our gain (Table 5: +2.8 with the description, +1.5 without).

AuthorsOpenReview ↗

Response to Reviewer 5Emt (Part 2/4)

W1(b) / Q1. Comparison under the same backbone, data and compute

Concern: without reproducing prior objectives under the same setting, it is unclear whether removing a text-described span and predicting it through placeholder tokens is more effective.

The paper's question, whether a visual prediction target improves understanding, is controlled by the language-only counterpart, trained on the same data with the same compute. Which visual objective helps most is a second question. Within our pipeline, removal without a description is the closest analogue of a target given only by its position, and the new analyses measure in our models the property that visible-token objectives are built to preserve.

  • A target given only by its position. Removing the same span without a description (Table 5) keeps our backbone, data, schedule, read-out and loss, and tells the model only where the target is, as masked video modelling does; unlike masked modelling, the span still follows an annotated event, and the target is frozen features rather than the model's own outputs. It reaches 59.5 on average, against 58.0 for SFT and 60.8 for ours. The position alone therefore retains 1.5 of the 2.8 points, and the language that selects the span accounts for the rest.
  • What visible-token objectives preserve. ROSS and PRe supervise the visual tokens present in the input in order to counteract the loss of visual information inside the LLM. Table R2 below measures this property directly in our models: language-only training erodes the visual structure in the upper half of the network, and the model trained with our objective, whose target is not in the input, stays closer to it. Our objective also ties descriptions to their own spans (Table R3), which an objective on visible tokens does not target.
  • Published controlled evidence. For ST-LLM and Cambrian-S, the controlled results discussed under W1(a) already measure each objective in its own pipeline: masking together with its loss adds 1.1 on MVBench, and next-frame prediction leaves general video benchmarks essentially unchanged. A contemporaneous study of an image MLLM (GAS, arXiv 2608.12209, Table 10) reports that a controlled adaptation of ROSS-style reconstruction to Qwen3-VL-2B falls below its own baseline (45.70 vs 47.25). Objectives that supervise visible tokens therefore do not transfer automatically to a new backbone and data.

W2 / Q2. Source of the gains

Concern: the gains may come from preserving visual information across layers or from aligning the description with the visual tokens, rather than from visual prediction; the task is closer to text-conditioned feature reconstruction than to world modelling.

On terminology, "prediction" is the accurate term for the objective, and the paper presents world modelling as an outlook rather than as a conclusion. We welcome the chance to state the distinction explicitly. Reconstruction, as in ROSS and PRe, recovers a target that is present in the model's input. In our objective the span is removed before the LLM sees the video, so the target must be produced from the remaining frames and the description. This is feature prediction of unobserved content in the sense of V-JEPA (Bardes et al., Revisiting Feature Prediction for Learning Visual Representations from Video, cited in Section 5), whose objective likewise predicts the features of masked regions from their context. The description states what happens in the span but not how it looks or moves, so it does not contain the target. The prediction is indeed text-conditioned, as the reviewer notes; this is a design choice of the objective, and Table 5 measures its contribution.

The sentence on world modelling closes Section 1 as an outlook ("these results point toward…"). It rests on the fact that latent prediction of unobserved video is the objective of JEPA-style world models such as V-JEPA 2, which Section 1 cites. None of our conclusions depends on it, and the paper does not claim that the trained model is a world model.

AuthorsOpenReview ↗

Response to Reviewer 5Emt (Part 3/4)

W2 / Q2 (continued). Mechanism

On mechanism, the two effects the reviewer names are candidate routes through which a prediction objective can act, and we test both directly, following the reviewer's three suggestions. Table 5 already splits the gain into a part that needs no description (+1.5 on average) and a part that depends on it (a further +1.3). Cross-modal alignment alone cannot explain the first part, and preservation of visible tokens alone cannot explain the second.

(a) Questions that require anticipation. STAR labels each question by type. Table R1 compares Prediction questions, which ask what happens next, with Sequence questions, which ask about temporal order, at all three scales.

Table R1. STAR accuracy (%) by question type (Prediction: 624 questions; Sequence: 3,586 questions).

ScalePrediction, Base+SFTPrediction, oursGainSequence, Base+SFTSequence, oursGain
0.8B67.1569.55+2.4073.7076.35+2.65
2B71.7972.92+1.1277.9779.87+1.90
4B66.8371.63+4.8176.2777.38+1.12

The objective improves both question types at every scale, so its gain is not confined to anticipation. Anticipation gains most at 4B (+4.81), while at 0.8B and 2B Sequence questions gain as much or more (+2.65 and +1.90, against +2.40 and +1.12). Visual prediction therefore strengthens temporal understanding broadly, consistent with its gains on TempCompass and on all five grounding benchmarks, rather than only on the questions that most resemble the training objective.

(b) Preservation of visual information across layers. We test the reviewer's first route directly: does the LLM keep the structure of what it sees? On 300 Charades-STA test samples (at most two per video), with the video and no query, we average the hidden states of the visual tokens in each one-second group and compare the temporal similarity structure across groups with that of the frozen input features (representational similarity analysis: Spearman correlation between the two cosine-distance matrices, averaged over clips; 1 means the structure is fully preserved).

Table R2. Preservation of the temporal similarity structure of the visual input (RSA; Qwen3.5-0.8B, 300 samples, no query).

Layer612 (read-out)1824
Base (released)0.7610.7170.7630.829
Base+SFT0.7690.7150.7520.812
Ours0.7680.7180.7690.820
Ours − Base+SFT−0.001+0.004+0.017+0.008

Our objective preserves the visual structure that language-only training erodes. At layer 6, our model and Base+SFT are indistinguishable. In the upper half of the network, Base+SFT moves away from the structure of its visual input, by 0.011 at layer 18 and 0.017 at layer 24 relative to the released model. The model trained with our objective stays closer to it (0.820 against 0.812 at layer 24), and at layer 18 it even exceeds the released model (0.769 against 0.763).

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response to All Reviewers (Part 1/2)

We thank all reviewers for their careful reading and constructive comments. We appreciate that the reviewers found the motivation good and the method simple (sx36), the idea interesting, intuitive and clearly presented (5Emt), the comparison with the cross-entropy baseline carefully matched in backbone, data, steps and forward passes (5Emt), the gains consistent at no inference cost (5Emt, sx36), and the paper comprehensive (V4Ah).

The reviews converge on one central question, which Reviewers 5Emt, sx36 and V4Ah each raise in their own terms: where does the gain come from? Part 2 answers it with new analyses of the trained models. The table below summarises our answer to each concern and where it is detailed.

ConcernRaised byShort answerWhere
Relation to ROSS, PRe, ST-LLM and Cambrian-S, and the claim of the first systematic study5EmtThe claim concerns the study, not the existence of visual objectives. None of the cited works tests, under matched compute, whether visual prediction improves video LLM understanding across scales and task families.Table below; reply to 5Emt, W1(a)
Comparison with prior objectives in the same setting5Emt, V4AhIn our pipeline, removing the span without a description gives the model only the position of the target, as masked modelling does. A new analysis measures in our models the property that visible-token objectives are designed to preserve.Reply to 5Emt, W1(b)
Source of the gain: prediction, preservation of visual information, cross-modal alignment or generic regularisation5Emt, sx36, V4AhAbout half of the gain depends on the description that selects the target, and the gain comes with better-preserved visual structure and a span-specific alignment between descriptions and video.Part 2; reply to 5Emt, Tables R1–R3
"Prediction" versus "reconstruction", and world modelling5EmtThe target is removed from the input, so it must be predicted rather than copied, as in V-JEPA. World modelling is the outlook that motivates the study, not a conclusion drawn from it.Reply to 5Emt, W2
The 4B baseline, and Table 2sx36, V4AhThe headline numbers come from 0.8B and 2B, where the baseline exceeds the released Instruct model on all nine benchmarks. TimeLens-7B in Table 2 is trained on the same TimeLens-100K data.Reply to sx36, W2
Single runs, sub-point ablation differences, and hyper-parameters chosen on test benchmarkssx36Ten separately trained models with the objective (eight configurations at 0.8B, one run each at 2B and 4B) all improve on their matched baselines. Ablation conclusions rest on differences above one point on average or on the benchmarks concerned, and the defaults were transferred unchanged to 2B and 4B.Part 2; reply to sx36, W3

How the objectives differ. The objectives the reviewers cite differ from ours in what the target is and what selects it:

TargetVisible in the LLM inputSelected bySetting
ROSSlatents of the input image (denoising)yesevery visual tokenimage MLLM
PRepre-LLM features of the same visual tokens, from a middle layeryesevery visual tokenimage MLLM
ST-LLMthe LLM's own outputs at unmasked positions on the full inputyesrandom maskingvideo LLM
Cambrian-Sencoder feature of the next frameonly after the predicting positionthe next time stepvideo LLM
Oursfrozen features of a removed span of several secondsnoa description of the spanvideo LLM

For video LLMs, the prior evidence is limited and mixed: ST-LLM reports the joint effect of masking and its loss on one benchmark, Cambrian-S finds that next-frame prediction leaves general video benchmarks essentially unchanged, and the contemporaneous IVT evaluates future-frame prediction only on anticipation tasks. Our reply to Reviewer 5Emt (W1(a)) gives the details.

AuthorsOpenReview ↗

General Response to All Reviewers (Part 2/2)

What the gain depends on. The reviewers ask whether the gain reflects the visual target, or a generic effect of adding an auxiliary loss. Three results bear on this question.

  • About half of the gain depends on the language that selects the target (Table 5). Removing the description leaves the loss, its weight, the read-out, the data and the number of passes unchanged, yet the average gain falls from +2.8 to +1.5 (NExT-QA from +1.2 to 0.0, STAR from +3.0 to +1.1). The description can affect the gain only through what it says about the target, so a regulariser that ignores content would not produce this part of the gain.
  • The objective ties each description to its own span. At the last layer, the hidden state of a description is closer to the visual tokens of its span than to other windows of the same video by 0.0387 with our objective, against 0.0338 after SFT (+14%), and its attention to the span rises from 1.54 to 1.59 times the uniform level. The reference is other windows of the same video, so this effect is specific to what the span contains.
  • The objective preserves the visual structure that language-only training erodes. Relative to the released Base model, SFT moves the temporal similarity structure of the visual tokens away from that of the visual input in the upper half of the network (−0.011 at layer 18, −0.017 at layer 24). The model trained with our objective stays closer to it (0.769 vs 0.752 at layer 18, 0.820 vs 0.812 at layer 24).

The objective therefore does not act merely as a generic regulariser. It acts through the two routes that Reviewer 5Emt names, preservation of visual information and cross-modal alignment, and the alignment is specific to the span that the description selects. We regard these routes as the means by which visual prediction improves understanding, not as alternatives to it. The gain is also not confined to anticipation: on STAR, both Prediction and Sequence questions improve at all three scales. Tables R1–R3 and the protocol are in our reply to Reviewer 5Emt; all analyses use the checkpoints evaluated in Table 1, with identical inputs for the models compared.

Reliability and fairness of the comparison.

  • Replication across separately trained models. Each configuration is trained once, which is the standard protocol for comparing training recipes of multimodal LLMs. The objective has been trained in ten separate runs, the eight 0.8B configurations of Tables 3–6 and one run each at 2B and 4B, and every one improves on its matched baseline on average. The three independently trained scales give 27 of 27 paired comparisons in favour of the objective.
  • The baseline is neither weak nor an unlucky run. At 0.8B and 2B it exceeds the released Instruct model on all nine benchmarks, and between its 4k- and 6k-step checkpoints the 4B baseline changes no benchmark by more than 0.4 points (Appendix D).
  • The comparison favours the baseline in text supervision. Both arms share the backbone, data, schedule, batch size, number of forward passes, evaluation step and evaluation protocol, and the baseline receives about one third more language-modelling passes (about 1.15M against 864K).

We thank the reviewers again for their time and are glad to answer further questions during the discussion period.

AuthorsOpenReview ↗

Update to the General Response: Results of the Requested Experiments

We thank the Area Chair and all reviewers for their time and for the care with which they have read our work. The experiments that were still running when we posted our responses have now finished. They address the comparison with published visual objectives (Reviewers 5Emt and V4Ah), the baseline with matched text supervision, and the controls requested by Reviewer sx36, which we report in our follow-up to that review. All models are Qwen3.5-0.8B in the grounded-description setting of Table 1, trained on the same data and scored exactly as in Table 1.

Table R4. The seven benchmarks of Table 1 (Qwen3.5-0.8B).

ModelMVBenchTempC.NExT-QASTARCharadesANetQVHAvg
Base+SFT (Table 1)56.961.976.969.045.141.654.458.0
SFT, text passes matched to ours55.762.075.868.945.540.655.057.6
PRe, as published51.259.861.752.140.936.346.549.8
Next-frame prediction (Cambrian-S)54.162.174.366.044.742.155.156.9
Masked video modelling (ST-LLM)52.460.462.952.940.838.551.651.3
Ours59.665.278.172.048.244.757.560.8

Published objectives in our pipeline. PRe, Cambrian-S next-frame prediction and ST-LLM masked video modelling were trained with the same backbone, data, prediction examples, schedule and number of steps as ours. Each uses the layer, prediction head and loss weight of its paper (PRe: middle layer, λ = 0.5; next-frame prediction: last layer, 0.1; masked modelling: 1.0), without further tuning. All three fall below the language-only baseline (49.8, 56.9 and 51.3, against 58.0), while ours improves on it by 2.8 points and is the best model in the table on every benchmark.

Keeping visual information is not sufficient. As in Table R2, we measured how closely the similarity structure of the visual tokens inside the LLM follows that of the visual input (RSA), on the same 300 Charades-STA samples and with the checkpoints scored above. PRe keeps this structure best, as its objective is designed to do: its RSA at the last layer is 0.869, above the released model (0.829). Next-frame prediction and masked modelling lose it in the upper layers (0.739 and 0.584 at the last layer). Yet PRe, which keeps the most visual structure, scores lowest downstream. What our objective adds is therefore its target: an event that has been removed from the video and is named by a sentence. Removing the event already helps without the sentence, and the sentence adds about half of the gain (Table 5).

Matched text passes (Reviewer sx36). The language-only baseline of Table 1 sees about one third more language-modelling passes than our arm (about 1.15M against 864K), because a quarter of our batch carries the prediction examples. An SFT baseline that sees as many language-modelling passes as our arm averages 57.6, against 58.0 for the baseline of Table 1. Matching the text passes therefore does not close the gap: ours (60.8) is higher than the matched baseline on all seven benchmarks, so the gap is not an artefact of over-training the baseline on text.

We are grateful to the Area Chair and the reviewers for the time they have given to our submission. Their comments have clarified for us what the analysis of this work needs to show.

AuthorsOpenReview ↗

Revised PDF Uploaded: Summary of Changes

We thank the reviewers and the Area Chair for their comments, which shaped this revision. The revised PDF has new or changed text in blue; the main text stays within 10 pages.

  • Where the gain comes from (new Section 4.3). The published objectives, the matched-text baseline and the controls from the discussion are compared in Table 3, with measurements inside the trained models in Figure 4. Read-out depth and loss weight are kept as design choices in Section 4.4.
  • Framing. The objective is described as text-conditioned prediction of unobserved visual content, no longer as learning how the world evolves.
  • Three runs per scale (17 November). Both arms were trained three times per scale with different seeds; Figure 3 shows the mean gains with 95% confidence intervals, and Appendix J gives the full table. Table 3 also reports the matched-text baseline at 2B.

Tables 3–6 of the submission are now Tables 4–7, and its Figures 4 and 5 are now Figures 3 and 5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.