- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies video event prediction from observed video segments. It proposes IDS, which predicts discrete tokens representing future-video features before producing a textual event prediction. Its three-stage training pipeline combines future-token prediction, event-description grounding, and GRPO. The authors introduce VEPBench, containing approximately 3,000 manually verified samples with recorded futures, plausible alternatives, and typed distractors, alongside a separate 17K-video training set. Experiments using Qwen2.5-VL-7B on VEPBench and FutureBench show improvements over textual and continuous-latent baselines, with particularly strong results on recorded-future selection and one-hop prediction.
Critical strengths
-
Well-motivated visual intermediate prediction. IDS combines future-token prediction with event-description grounding, explicitly connecting future visual supervision to semantic event prediction. Discrete token targets allow future-feature prediction to use standard next-token training, providing a practical way to incorporate visual intermediate states into an MLLM.
-
Meaningful empirical gains. Under the reported common SFT budget, IDS achieves 67.00% SR@A, compared with 54.41% for textual reasoning using the same backbone and training data. The full pipeline reaches 72.78%, with additional gains on one-hop prediction in FutureBench. These results support the practical value of the pipeline.
-
Useful diagnostic benchmark. VEPBench distinguishes the recorded future from an annotated plausible alternative, with manually verified candidates, typed distractors, and candidate-free evaluation. This distinction helps account for future uncertainty and makes the evaluation more informative about whether models match the recorded outcome or select a reasonable continuation.
Critical weaknesses
-
Limited evaluation of plausible futures. SR@AB accepts only two designated continuations, while open-ended answers are scored against the recorded future. The evaluation therefore provides limited evidence about broader predictive plausibility and uncertainty handling.
-
Open-ended scoring lacks human calibration. Scores rely on GPT-5.4 and Gemini 3.1 Pro without reported agreement with human assessments. The influence of judge preferences on scores and differences between models remains unclear.
-
The role of visual intermediate states is insufficiently validated. Controlled interventions on visual tokens or comparisons with matched computational budgets are missing, making it difficult to distinguish meaningful future-visual information from additional supervision and intermediate computation.
-
Cross-backbone applicability remains unverified. IDS is trained and evaluated only with Qwen2.5-VL-7B. Results using Qwen3-VL or other backbones are not reported, leaving the broader applicability of the method unclear.
Questions for authors
see above
4 comments
W3
We added inference-time interventions with controlled conditions (Appendix J, Table 7, reproduced below). All start from the same checkpoint without further training, fixing questions, observed videos, and candidates. Token-replacement conditions use 84 positions; the codebook experiment reinitializes only its weights.
Matching GT improves final prediction over generated tokens. SR@A/SR@AB increase by 13.30/12.05 percentage points, quantifying the predicted-versus-GT conditioning gap and supporting the importance of future-token quality.
Cross-example replacement tests the correspondence between visual tokens and the current video. We retain the observed video but replace the 84 future tokens with another video's real tokens from the same visual vocabulary. SR@A/SR@AB fall to 58.40%/60.30%, with 30.87% Invalid, supporting the role of sample correspondence in prediction and valid continuation. Retained performance is consistent with continued reasoning from the observed video and some robustness to mismatched tokens.
Random-token and codebook interventions test dependence on learned visual representations. Replacing 21 GT positions with random visual tokens or all 84 positions with full-vocabulary random tokens lowers SR@A to 57.26%/26.16%, with 32.85%/69.13% Invalid; reinitializing codebook weights gives 100% Invalid. These perturbations affect both prediction and valid continuation, so Table 7 reports accuracy and Invalid together.
Supervision controls further support the visual prediction–alignment pathway. Stage 1 learns future-token prediction; Stage 2 learns to describe future events from those tokens, making the predicted visual information usable for event judgment (manuscript Table 3). Textual Caption already aligns historical video with future-event text, yet matched Stage 1+2 IDS performs substantially better (manuscript Table 4). The key difference is the future visual tokens and their prediction–alignment process. Continuous baselines also use the same text and future-video supervision. Except for method-specific and faithful-reproduction differences, comparable settings match samples, training parameters, and steps; IDS leads on most core same-backbone metrics (manuscript Tables 1, 2).
We additionally measured complete generation time to assess practical inference efficiency. Under identical hardware and input settings, mean model.generate time on 1,056 paired FutureBench questions is 6.632 s for IDS versus 7.972 s for textual CoT, 16.81% lower. This includes visual tokens and the subsequent answer; the 84-token stage was not timed separately. This is consistent with our motivation that visual states may reduce the burden of describing future changes entirely in text. Estimated IDS training cost is approximately 100 GPU·h.
We also added normal-inference diagnostics: 52.45% positionwise GT token agreement, 80.50% codebook utilization, and exactly 84 legal visual tokens in 95.17% of examples (Appendix J; sequence validity is distinct from final-answer Invalid). Together, these findings support the model's actual use of future visual information and the effectiveness and efficiency of the IDS pathway.
Visual-token conditioning and codebook interventions (revised manuscript, Appendix J, Table 7). All values are percentages; Invalid is the final-answer invalid rate.
| Condition | SR@A | SR@AB | SR@C | SR@D | SR@E | Invalid |
|---|---|---|---|---|---|---|
| Normal prediction | 13.41 | 9.41 | 0.57 | 0.00 | ||
| GT future tokens (84) | 86.08 | 88.67 | 2.13 | 8.59 | 0.61 | 0.00 |
| Cross-example real future tokens (84) | 58.40 | 60.30 | 7.15 | 0.61 | ||
| Full-vocabulary random tokens (84) | 26.16 | 27.15 | 0.91 | 2.74 | 0.08 | 69.13 |
| 21 GT positions replaced by random visual tokens | 57.26 | 59.16 | 1.52 | 32.85 | ||
| Randomly initialized codebook weights | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 100.00 |
W4
We are conducting additional experiments to evaluate IDS on other models. These experiments require some more time, and we will add the results to the main paper and this rebuttal as soon as they are available.
Q
VEPBench combines a defined plausible-future measure with candidate-free prediction. SR@A accepts recorded A; SR@AB also accepts human-verified plausible B. Open-set descriptions are judged against the actual future video.
We added Grok 4.7 and Kimi alongside Gemini/GPT to reduce single-judge dependence. IDS ranks first under three and second under one, exceeding same-backbone non-latent SFT/GRPO throughout. Added Kendall's W analysis gives mean 0.813 (bootstrap 95% CI [0.792, 0.833]; median 0.851), supporting relative-quality agreement and open-set evaluation's reasonableness (Appendix A.2). The complete four-judge comparison is in revised manuscript Table 1.
Added fixed-checkpoint interventions support token value and some robustness (Appendix J, Table 7, reproduced at the end of W3). Matching GT improves prediction; cross-sample replacement of 84 real codes lowers scores but retains contextual prediction ability. Perturbations affect valid continuation, so we report Invalid. Added full-IDS free-generation diagnostics: 52.45% positionwise GT agreement, 80.50% codebook utilization, and exactly 84 legal visual tokens in 95.17% of examples. Comparable training matches samples, parameters, steps, and text/future-video supervision except method/faithful-reproduction differences. The added cost analysis estimates IDS training at about 100 GPU·h; complete IDS/textual-CoT generate calls average 6.632/7.972 s on 1,056 paired FutureBench questions (16.81% less), including visual generation rather than timing 84 tokens alone.
We are conducting additional experiments to evaluate IDS on other models. These experiments require some more time, and we will add the results to the main paper and this rebuttal as soon as they are available.
Reviewer g249
Thank you again for your valuable comments. We have now completed the matched representation ablation alongside the reference-free evaluation and the additional-backbone comparison.
The new reference-free evaluation broadens the assessment of future plausibility. We reuse the same open-set prediction texts; judges receive only the observed video context and prediction, without the recorded future or restrictions to A/B candidates. IDS ranks first among all 17 configurations under each of the three reported judges, ahead of same-backbone SFT, GRPO, and Mirage (Table S1). This extends the paper's future-prediction findings (Abstract; Sec. 1) to contextual plausibility, addressing your concern that recorded-future and A/B metrics provide limited evidence about plausible continuations.
On Qwen3-VL-4B, both visual variants improve on text-only reasoning, extending the finding in Sec. 5.3 beyond the original backbone. The text-only SFT+GRPO baseline and both visual variants used the same Qwen3-VL-4B backbone and training examples. We matched all comparable training settings, including hyperparameters and step counts, and evaluated all three on the same observed videos under the same protocol. Compared with text-only SFT+GRPO reasoning, continuous visual states raised SR@A and SR@AB by 5.17 and 1.86 percentage points, respectively. Discrete visual tokens raised them by 16.07 and 9.93 points (Table S2).
Discrete tokens offer a further performance advantage over continuous states at the same sequence length and with the same future supervision. In the matched ablation, the only change was to replace IDS’s discrete visual tokens with continuous states. Both variants used the same 84 visual positions, future-video supervision, training data, stages, hyperparameters, training steps, and evaluation settings. The substitution lowered SR@A by 10.90 percentage points and SR@AB by 8.07 points (Table S2).
Thank you for your patience. We hope these results address your concerns.
Table S1. Newly added future-plausibility scores without a recorded-future reference (0–100; higher is better). All 17 configurations are included. Predictions are reused from the original open-set evaluation. NL denotes direct non-latent textual training.
| Judge / model | Qwen2.5-VL-7B | Qwen2.5-VL-7B | Qwen2.5-VL-7B | Qwen2.5-VL-7B | Video-Chat-A1 | Video-R1 | Keye-VL-8B | GLM-4.6V-Flash-9B | Kimi-VL-A3B | InternVL3-8B | LLaVA-NeXT-Video-7B | Qwen3-VL-8B | Qwen2.5-VL-32B | Qwen2.5-VL-7B | Qwen2.5-VL-3B | GPT-5.4 | Gemini 3.1 Pro |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | IDS (ours) | Mirage | GRPO (NL) | SFT (NL) | Vanilla | Vanilla | Vanilla | Vanilla | Vanilla | Vanilla | Vanilla | Vanilla | Vanilla | Vanilla | Vanilla | Vanilla | Vanilla |
| GPT-6.1-sol | 59.93 | 47.67 | 51.78 | 34.20 | 52.87 | 52.28 | 53.74 | 39.80 | 53.24 | 43.04 | 55.75 | 52.02 | 55.25 | 16.15 | 37.07 | 43.32 | |
| Grok 4.7 | 46.93 | 36.40 | 40.47 | 27.52 | 42.55 | 39.72 | 36.13 | 28.70 | 36.63 | 33.52 | 42.99 | 43.11 | 42.49 | 11.85 | 28.95 | 29.95 | |
| Kimi K3.1 Pro | 56.32 | 49.48 | 55.28 | 52.22 | 33.72 | 55.84 | 51.80 | 47.60 | 35.76 | 47.10 | 42.07 | 52.05 | 56.14 | 18.93 | 40.69 | 45.70 |
Table S2. VEPBench selection rates (%) on Qwen3-VL-4B. The continuous control follows all three IDS training stages.
| Model / backbone | Method | SR@A ↑ | SR@AB ↑ | SR@C ↓ | SR@D ↓ | SR@E ↓ |
|---|---|---|---|---|---|---|
| Qwen3-VL-4B | Vanilla | 42.23 | 61.34 | 28.50 | 7.20 | 2.96 |
| Qwen3-VL-4B | Text-only SFT+GRPO | 51.03 | 61.87 | 15.42 | 14.35 | 8.36 |
| Qwen3-VL-4B | Continuous (matched IDS pipeline) | 13.77 | 7.83 | |||
| Qwen3-VL-4B | IDS | 67.10 | 71.80 | 14.65 |
Reviewer g249
Thank you for your constructive review and positive assessment of the visual prediction framework and benchmark. We address your concerns below with additional visual-token interventions and multi-judge analyses, clarify the scope of plausible-future evaluation, and outline the ongoing cross-backbone experiments.
We label our responses to the weaknesses as W1, W2, … and to the questions as Q1, Q2, …, following their order in your review.
W1
SR@A and SR@AB distinguish prediction of the recorded future from acceptance of a plausible alternative. SR@A uses the actual continuation A as a verifiable target; SR@AB additionally accepts the manually verified alternative B, which is plausible but did not occur. This incorporates multiple possible futures into a common closed-set protocol, extending evaluation beyond a single recorded answer while preserving comparability across models.
Recorded-future references give candidate-free prediction a verifiable, reproducible evaluation basis. Models freely generate future-event descriptions without answer choices, and semantic agreement is assessed against the actual future video. This measures how well autonomous predictions match the actual continuation, complementing SR@AB's acceptance of a plausible alternative.
To directly assess broader future plausibility, we are adding an evaluation that withholds the recorded future from judges. Judges assess the plausibility of predicted events based only on the observed context and historical video, directly addressing broader predictive plausibility and multiple possible futures. Such evaluation relies more on subjective judgment and may have higher scoring variance and greater reproducibility challenges. We therefore favor treating it as supplementary evidence while retaining the benchmark's primary metrics; once the experiment is complete, we will add the results to the appendix and rebuttal.
W2
Model judges provide a standardized open-set evaluation protocol that supports reproducibility and benchmark reuse. Common inputs and scoring prompts allow subsequent methods to be evaluated under the same protocol. Judges receive the observed video, recorded future video, and prediction, without answer choices or model identity, grounding their scores in visual evidence.
To reduce the influence of judge preferences and improve open-set evaluation reliability, we added Grok 4.7 and Kimi alongside Gemini and GPT and report their scores separately in revised manuscript Table 1. The added multi-judge evaluation shows that IDS's advantage is consistent across evaluators. IDS ranks first under three judges and second under the fourth, outperforming same-backbone non-latent SFT/GRPO under all four. Its gains are therefore supported by multiple evaluators rather than appearing only under one judge, reducing dependence on a particular judge's preferences.
We further added Kendall's W to assess agreement on relative prediction quality. W ranges from 0 to 1; values closer to 1 indicate stronger agreement among judges on prediction rankings. Mean W is 0.813 (bootstrap 95% CI [0.792, 0.833]; median 0.851), indicating strong agreement and supporting the reasonableness of our open-set evaluation. Multi-judge cross-evaluation and agreement analysis provide a scalable reliability check while reducing reliance on a single judge. The computation and results are provided in Appendix A.2.