- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
Although the paper claims to investigate whether 'prediction can benefit understanding,' it does not actually conduct a substantive study on this underlying mechanism. Instead, it merely demonstrates that adding a prediction loss during the fine-tuning stage yields better performance than standard fine-tuning.
Critical strengths
The paper is quite comprehensive; setting aside the research question and its value.
Critical weaknesses
- Although the paper claims to investigate whether "prediction can benefit understanding," the experiments fail to explain why this phenomenon occurs. The authors merely list empirical improvements without providing a substantive analysis or deeper insights into the underlying mechanisms driving this benefit.
- The baseline comparisons are notably weak and fail to adequately engage with or compare against existing state-of-the-art related work in the field.
Questions for authors
Refer to weaknesses.
6 comments
Response to authors
I believe the authors have clearly recognized the limitations of this work and have made an effort to address them in the rebuttal. I appreciate the authors' efforts, so I may raise my score to 2. However, the paper still lacks sufficient interpretability. Furthermore, in Section 4, the authors attempt to analyze the claim that "prediction benefits understanding" using rather disorganized and elementary concepts, such as loss weights and the functions of different layers in neural networks. It is clear that they have missed the point here, which prevents me from raising the score to 3.
Response to Reviewer V4Ah's follow-up
We sincerely thank the reviewer for the prompt reply during the discussion and for being willing to reconsider the score. We also thank the reviewer for the precise criticism of Section 4, and we agree with it: the loss weight and the read-out depth are design choices. They show that the effect does not depend on one setting, but they do not explain why it occurs. Below we state the mechanism we propose, the evidence for each of its steps, and its limits.
The proposed mechanism. In SFT, the visual tokens are trained only through the text the model writes, so the representation shared with text generation keeps from the video what text generation needs. Our objective adds a signal defined on the video itself: an event named by a sentence is removed, and the model must produce the visual features of that event from the rest of the video. If this is how prediction helps understanding, three things should hold:
| Step | What should hold | What we measured | Result |
|---|---|---|---|
| 1. A signal on the video | The visual structure that text-only training erodes is kept | Similarity structure of the hidden states against the visual input (Table R2) | SFT moves the upper layers away from it; with our objective they stay close to the released model |
| 2. The named content matters | Removing the description, with everything else fixed, reduces the gain | Description ablation (Table 5) | The gain over SFT falls from +2.8 to +1.5 points |
| 3. The kind of target matters | Objectives with other targets do not help, even when they keep the visual structure | Three published objectives in our pipeline (Table R4, in our new comment to the Area Chair and all reviewers) and two controls (in our follow-up to Reviewer sx36) | Targets on the visible tokens (PRe), the next frame (Cambrian-S), randomly dropped tokens (ST-LLM), the span of another video, or the description as text all fall below SFT; PRe keeps the most visual structure yet falls furthest |
Step 3 also answers the reviewer's original concern about baselines: under the same backbone, data and compute, three published visual objectives fall below SFT, while ours is the best of these models on every benchmark.
The downstream pattern agrees with this account. Questions that ask what happens next or in which order events occur improve at all three scales (STAR, Table R1: +2.40 / +1.12 / +4.81 on Prediction and +2.65 / +1.90 / +1.12 on Sequence at 0.8B / 2B / 4B), and these are the questions that relating one event to the rest of the video should help most.
Limits of the account. We do not claim that the model reconstructs the removed content: RQ4 shows that a lower prediction error does not come with a larger gain, so the objective acts through the representation it shapes, not through the accuracy of its predictions. About half of the gain does not depend on the description (Table 5) and comes from the visual signal itself. Step 1 describes what changes in the representation, while Steps 2 and 3 test what the gain depends on: the content named by the description, and a target on a removed event rather than on the visible tokens, the next frame, dropped tokens, another video or text. We therefore offer these steps as consistent evidence for the mechanism, not as a proof of it.
We are grateful to the reviewer for engaging with our work during the discussion. The reviewer's comments have shown us how the analysis should be organised. In the final version of the paper, Section 4 will be organised around this account: it will open with the mechanism and the evidence for each step, bring the comparisons of Table R4 and the controls into the main text, and condense the loss-weight and read-out-depth ablations into a short paragraph on design choices. We would welcome any further suggestion of an analysis that the reviewer would find convincing.
Response to Reviewer V4Ah
We thank the reviewer for the time spent on our paper and for noting that it is comprehensive. We read the review as raising two concerns: that the paper establishes an effect without explaining it, and that the baselines are not strong enough. Both are also raised by Reviewers 5Emt and sx36, and we answer them directly below.
1. Why does prediction benefit understanding?
Establishing the effect under controlled conditions is itself a contribution, because prior evidence is mixed, and the submission also analyses where the effect comes from. Prior evidence on whether visual prediction helps a model understand is mixed (Section 5 of the paper):
Against this background, our compute-matched comparison shows a consistent effect. The objective improves 27 of 27 paired comparisons across nine benchmarks and three independently trained scales, while the baseline receives about one third more text-supervised passes.
The submission also contains four controlled ablations, each varying one aspect of the objective with the backbone, the language data and the compute fixed. They cover where the supervision is applied (read-out depth), how strongly (loss weight), what conditions the prediction (the description), and which data provide the prediction examples (grounded descriptions or captions). Together with the measurement of prediction error, they show three things:
We now add three analyses of the trained models that show how the objective acts (reply to Reviewer 5Emt, Tables R1–R3):
With Table 5, these results show that the benefit comes from the visual target acting on the shared representation, not merely from a generic regularisation effect.
2. Baselines and related work
The primary baseline is the controlled one, and it is strong. Base+SFT is trained on the same data with the same number of forward passes, so it is the only comparison that isolates the objective. It is also a strong model. At 0.8B and 2B it exceeds the released Qwen3.5 Instruct models on all nine benchmarks, for example 76.9 vs 63.3 on NExT-QA and 43.4 vs 32.2 in mean grounding mIoU at 0.8B.
Prior visual objectives are addressed in our pipeline and through their published controlled results. With identical backbone, data and compute, removing the span without a description, which gives the model only the target's position as masked video modelling does, reaches 59.5 on average (ours 60.8, SFT 58.0). Our reply to Reviewer 5Emt (W1) measures in our models the property that ROSS and PRe preserve, and discusses the controlled results of ST-LLM, Cambrian-S and a ROSS-style objective; Part 1 of the General Response compares them side by side.
Published state-of-the-art models are included in Table 2. Table 2 lists twelve published video LLMs, including 7B models trained with reinforcement learning (Video-R1, VideoP2R, APPO) and the grounding specialist TimeLens-7B. Our 4B model reaches 70.4 on MVBench and 77.3 on TempCompass, the highest in the table. Our 2B model exceeds TimeLens-7B on all three TimeLens benchmarks (52.5 / 50.4 / 59.3 vs 48.8 / 46.2 / 56.0). TimeLens-7B is trained on TimeLens-100K, which is also in our mixture; our 2B Base+SFT is comparable to it (47.6 / 48.5 / 56.4), and the objective adds 1.9 to 4.9 points on each benchmark.
If the reviewer has specific methods in mind for comparison, we would be glad to discuss them. We hope that the new evidence addresses the reviewer's concerns, and would be grateful if the reviewer would reconsider the assessment in light of it.