acceptodds
Under review as a conference paper at ICLR 2027

IMAGINE IN DISCRETE SPACE: DISCRETE VISUAL LATENT REASONING FOR VIDEO EVENT PREDICTION

Abstract

Video event prediction requires multimodal large language models (MLLMs) to infer events beyond an observed video segment. We investigate whether future-video supervision can support this task through discrete visual intermediate states. We propose **Imagine in Discrete Space (IDS)**, which quantizes future-video features into visual tokens and learns to predict these tokens autoregressively from observed videos. A three-stage pipeline learns future-token prediction, grounds quantized future tokens in event descriptions, and refines event prediction with reinforcement learning. At inference, only the observed segment is available as visual input, and the final prediction is conditioned on self-generated visual tokens. The formulation provides a next-token training interface for future visual features without a pixel-reconstruction objective. We also introduce **VEPBench**, a benchmark of approximately 3K samples with recorded and plausible alternative futures and typed distractors, alongside a source-disjoint 17K-video training set. On VEPBench, IDS before reinforcement learning achieves 67.00% recorded-future selection, compared with 54.41% for the text-reasoning variant using the same backbone and reported SFT budget. The full model reaches 72.78%. Results on FutureBench provide evidence of cross-benchmark transfer, with stronger performance on one-hop prediction than on multi-hop tasks. These findings support discrete future visual tokens as a useful intermediate representation in the evaluated setting.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Event Prediction, Latent Space Resaoning, VLM, Computer Vision

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_g249OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies video event prediction from observed video segments. It proposes IDS, which predicts discrete tokens representing future-video features before producing a textual event prediction. Its three-stage training pipeline combines future-token prediction, event-description grounding, and GRPO. The authors introduce VEPBench, containing approximately 3,000 manually verified samples with recorded futures, plausible alternatives, and typed distractors, alongside a separate 17K-video training set. Experiments using Qwen2.5-VL-7B on VEPBench and FutureBench show improvements over textual and continuous-latent baselines, with particularly strong results on recorded-future selection and one-hop prediction.

Critical strengths
  1. Well-motivated visual intermediate prediction. IDS combines future-token prediction with event-description grounding, explicitly connecting future visual supervision to semantic event prediction. Discrete token targets allow future-feature prediction to use standard next-token training, providing a practical way to incorporate visual intermediate states into an MLLM.

  2. Meaningful empirical gains. Under the reported common SFT budget, IDS achieves 67.00% SR@A, compared with 54.41% for textual reasoning using the same backbone and training data. The full pipeline reaches 72.78%, with additional gains on one-hop prediction in FutureBench. These results support the practical value of the pipeline.

  3. Useful diagnostic benchmark. VEPBench distinguishes the recorded future from an annotated plausible alternative, with manually verified candidates, typed distractors, and candidate-free evaluation. This distinction helps account for future uncertainty and makes the evaluation more informative about whether models match the recorded outcome or select a reasonable continuation.

Critical weaknesses
  1. Limited evaluation of plausible futures. SR@AB accepts only two designated continuations, while open-ended answers are scored against the recorded future. The evaluation therefore provides limited evidence about broader predictive plausibility and uncertainty handling.

  2. Open-ended scoring lacks human calibration. Scores rely on GPT-5.4 and Gemini 3.1 Pro without reported agreement with human assessments. The influence of judge preferences on scores and differences between models remains unclear.

  3. The role of visual intermediate states is insufficiently validated. Controlled interventions on visual tokens or comparisons with matched computational budgets are missing, making it difficult to distinguish meaningful future-visual information from additional supervision and intermediate computation.

  4. Cross-backbone applicability remains unverified. IDS is trained and evaluated only with Qwen2.5-VL-7B. Results using Qwen3-VL or other backbones are not reported, leaving the broader applicability of the method unclear.

Questions for authors

see above

4 comments

Reviewer g249

Thank you for your constructive review and positive assessment of the visual prediction framework and benchmark. We address your concerns below with additional visual-token interventions and multi-judge analyses, clarify the scope of plausible-future evaluation, and outline the ongoing cross-backbone experiments.

We label our responses to the weaknesses as W1, W2, … and to the questions as Q1, Q2, …, following their order in your review.

W1

SR@A and SR@AB distinguish prediction of the recorded future from acceptance of a plausible alternative. SR@A uses the actual continuation A as a verifiable target; SR@AB additionally accepts the manually verified alternative B, which is plausible but did not occur. This incorporates multiple possible futures into a common closed-set protocol, extending evaluation beyond a single recorded answer while preserving comparability across models.

Recorded-future references give candidate-free prediction a verifiable, reproducible evaluation basis. Models freely generate future-event descriptions without answer choices, and semantic agreement is assessed against the actual future video. This measures how well autonomous predictions match the actual continuation, complementing SR@AB's acceptance of a plausible alternative.

To directly assess broader future plausibility, we are adding an evaluation that withholds the recorded future from judges. Judges assess the plausibility of predicted events based only on the observed context and historical video, directly addressing broader predictive plausibility and multiple possible futures. Such evaluation relies more on subjective judgment and may have higher scoring variance and greater reproducibility challenges. We therefore favor treating it as supplementary evidence while retaining the benchmark's primary metrics; once the experiment is complete, we will add the results to the appendix and rebuttal.

W2

Model judges provide a standardized open-set evaluation protocol that supports reproducibility and benchmark reuse. Common inputs and scoring prompts allow subsequent methods to be evaluated under the same protocol. Judges receive the observed video, recorded future video, and prediction, without answer choices or model identity, grounding their scores in visual evidence.

To reduce the influence of judge preferences and improve open-set evaluation reliability, we added Grok 4.7 and Kimi alongside Gemini and GPT and report their scores separately in revised manuscript Table 1. The added multi-judge evaluation shows that IDS's advantage is consistent across evaluators. IDS ranks first under three judges and second under the fourth, outperforming same-backbone non-latent SFT/GRPO under all four. Its gains are therefore supported by multiple evaluators rather than appearing only under one judge, reducing dependence on a particular judge's preferences.

We further added Kendall's W to assess agreement on relative prediction quality. W ranges from 0 to 1; values closer to 1 indicate stronger agreement among judges on prediction rankings. Mean W is 0.813 (bootstrap 95% CI [0.792, 0.833]; median 0.851), indicating strong agreement and supporting the reasonableness of our open-set evaluation. Multi-judge cross-evaluation and agreement analysis provide a scalable reliability check while reducing reliance on a single judge. The computation and results are provided in Appendix A.2.

W3

We added inference-time interventions with controlled conditions (Appendix J, Table 7, reproduced below). All start from the same checkpoint without further training, fixing questions, observed videos, and candidates. Token-replacement conditions use 84 positions; the codebook experiment reinitializes only its weights.

Matching GT improves final prediction over generated tokens. SR@A/SR@AB increase by 13.30/12.05 percentage points, quantifying the predicted-versus-GT conditioning gap and supporting the importance of future-token quality.

Cross-example replacement tests the correspondence between visual tokens and the current video. We retain the observed video but replace the 84 future tokens with another video's real tokens from the same visual vocabulary. SR@A/SR@AB fall to 58.40%/60.30%, with 30.87% Invalid, supporting the role of sample correspondence in prediction and valid continuation. Retained performance is consistent with continued reasoning from the observed video and some robustness to mismatched tokens.

Random-token and codebook interventions test dependence on learned visual representations. Replacing 21 GT positions with random visual tokens or all 84 positions with full-vocabulary random tokens lowers SR@A to 57.26%/26.16%, with 32.85%/69.13% Invalid; reinitializing codebook weights gives 100% Invalid. These perturbations affect both prediction and valid continuation, so Table 7 reports accuracy and Invalid together.

Supervision controls further support the visual prediction–alignment pathway. Stage 1 learns future-token prediction; Stage 2 learns to describe future events from those tokens, making the predicted visual information usable for event judgment (manuscript Table 3). Textual Caption already aligns historical video with future-event text, yet matched Stage 1+2 IDS performs substantially better (manuscript Table 4). The key difference is the future visual tokens and their prediction–alignment process. Continuous baselines also use the same text and future-video supervision. Except for method-specific and faithful-reproduction differences, comparable settings match samples, training parameters, and steps; IDS leads on most core same-backbone metrics (manuscript Tables 1, 2).

We additionally measured complete generation time to assess practical inference efficiency. Under identical hardware and input settings, mean model.generate time on 1,056 paired FutureBench questions is 6.632 s for IDS versus 7.972 s for textual CoT, 16.81% lower. This includes visual tokens and the subsequent answer; the 84-token stage was not timed separately. This is consistent with our motivation that visual states may reduce the burden of describing future changes entirely in text. Estimated IDS training cost is approximately 100 GPU·h.

We also added normal-inference diagnostics: 52.45% positionwise GT token agreement, 80.50% codebook utilization, and exactly 84 legal visual tokens in 95.17% of examples (Appendix J; sequence validity is distinct from final-answer Invalid). Together, these findings support the model's actual use of future visual information and the effectiveness and efficiency of the IDS pathway.

Visual-token conditioning and codebook interventions (revised manuscript, Appendix J, Table 7). All values are percentages; Invalid is the final-answer invalid rate.

ConditionSR@ASR@ABSR@CSR@DSR@EInvalid
Normal prediction72.78‾\underline{72.78}76.62‾\underline{76.62}13.419.410.570.00
GT future tokens (84)86.0888.672.138.590.610.00
Cross-example real future tokens (84)58.4060.301.06‾\underline{1.06}7.150.6130.87‾\underline{30.87}
Full-vocabulary random tokens (84)26.1627.150.912.740.0869.13
21 GT positions replaced by random visual tokens57.2659.161.526.01‾\underline{6.01}0.46‾\underline{0.46}32.85
Randomly initialized codebook weights0.000.000.000.000.00100.00

W4

We are conducting additional experiments to evaluate IDS on other models. These experiments require some more time, and we will add the results to the main paper and this rebuttal as soon as they are available.

Q

VEPBench combines a defined plausible-future measure with candidate-free prediction. SR@A accepts recorded A; SR@AB also accepts human-verified plausible B. Open-set descriptions are judged against the actual future video.

We added Grok 4.7 and Kimi alongside Gemini/GPT to reduce single-judge dependence. IDS ranks first under three and second under one, exceeding same-backbone non-latent SFT/GRPO throughout. Added Kendall's W analysis gives mean 0.813 (bootstrap 95% CI [0.792, 0.833]; median 0.851), supporting relative-quality agreement and open-set evaluation's reasonableness (Appendix A.2). The complete four-judge comparison is in revised manuscript Table 1.

Added fixed-checkpoint interventions support token value and some robustness (Appendix J, Table 7, reproduced at the end of W3). Matching GT improves prediction; cross-sample replacement of 84 real codes lowers scores but retains contextual prediction ability. Perturbations affect valid continuation, so we report Invalid. Added full-IDS free-generation diagnostics: 52.45% positionwise GT agreement, 80.50% codebook utilization, and exactly 84 legal visual tokens in 95.17% of examples. Comparable training matches samples, parameters, steps, and text/future-video supervision except method/faithful-reproduction differences. The added cost analysis estimates IDS training at about 100 GPU·h; complete IDS/textual-CoT generate calls average 6.632/7.972 s on 1,056 paired FutureBench questions (16.81% less), including visual generation rather than timing 84 tokens alone.

We are conducting additional experiments to evaluate IDS on other models. These experiments require some more time, and we will add the results to the main paper and this rebuttal as soon as they are available.

Reviewer g249

Thank you again for your valuable comments. We have now completed the matched representation ablation alongside the reference-free evaluation and the additional-backbone comparison.

The new reference-free evaluation broadens the assessment of future plausibility. We reuse the same open-set prediction texts; judges receive only the observed video context and prediction, without the recorded future or restrictions to A/B candidates. IDS ranks first among all 17 configurations under each of the three reported judges, ahead of same-backbone SFT, GRPO, and Mirage (Table S1). This extends the paper's future-prediction findings (Abstract; Sec. 1) to contextual plausibility, addressing your concern that recorded-future and A/B metrics provide limited evidence about plausible continuations.

On Qwen3-VL-4B, both visual variants improve on text-only reasoning, extending the finding in Sec. 5.3 beyond the original backbone. The text-only SFT+GRPO baseline and both visual variants used the same Qwen3-VL-4B backbone and training examples. We matched all comparable training settings, including hyperparameters and step counts, and evaluated all three on the same observed videos under the same protocol. Compared with text-only SFT+GRPO reasoning, continuous visual states raised SR@A and SR@AB by 5.17 and 1.86 percentage points, respectively. Discrete visual tokens raised them by 16.07 and 9.93 points (Table S2).

Discrete tokens offer a further performance advantage over continuous states at the same sequence length and with the same future supervision. In the matched ablation, the only change was to replace IDS’s discrete visual tokens with continuous states. Both variants used the same 84 visual positions, future-video supervision, training data, stages, hyperparameters, training steps, and evaluation settings. The substitution lowered SR@A by 10.90 percentage points and SR@AB by 8.07 points (Table S2).

Thank you for your patience. We hope these results address your concerns.

Table S1. Newly added future-plausibility scores without a recorded-future reference (0–100; higher is better). All 17 configurations are included. Predictions are reused from the original open-set evaluation. NL denotes direct non-latent textual training.

Judge / modelQwen2.5-VL-7BQwen2.5-VL-7BQwen2.5-VL-7BQwen2.5-VL-7BVideo-Chat-A1Video-R1Keye-VL-8BGLM-4.6V-Flash-9BKimi-VL-A3BInternVL3-8BLLaVA-NeXT-Video-7BQwen3-VL-8BQwen2.5-VL-32BQwen2.5-VL-7BQwen2.5-VL-3BGPT-5.4Gemini 3.1 Pro
MethodIDS (ours)MirageGRPO (NL)SFT (NL)VanillaVanillaVanillaVanillaVanillaVanillaVanillaVanillaVanillaVanillaVanillaVanillaVanilla
GPT-6.1-sol59.9347.6759.66‾\underline{59.66}51.7834.2052.8752.2853.7439.8053.2443.0455.7552.0255.2516.1537.0743.32
Grok 4.746.9336.4045.40‾\underline{45.40}40.4727.5242.5539.7236.1328.7036.6333.5242.9943.1142.4911.8528.9529.95
Kimi K3.1 Pro56.3249.4855.2852.2233.7255.8451.8047.6035.7647.1042.0756.29‾\underline{56.29}52.0556.1418.9340.6945.70

Table S2. VEPBench selection rates (%) on Qwen3-VL-4B. The continuous control follows all three IDS training stages.

Model / backboneMethodSR@A ↑SR@AB ↑SR@C ↓SR@D ↓SR@E ↓
Qwen3-VL-4BVanilla42.2361.3428.507.202.96
Qwen3-VL-4BText-only SFT+GRPO51.0361.8715.4214.358.36
Qwen3-VL-4BContinuous (matched IDS pipeline)56.20‾\underline{56.20}63.73‾\underline{63.73}14.67‾\underline{14.67}13.777.83
Qwen3-VL-4BIDS67.1071.8014.6510.20‾\underline{10.20}3.35‾\underline{3.35}
Reviewer_1ZX4OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Imagine in Discrete Space (IDS), a method for video event prediction that uses discrete tokens derived from future-video features as intermediate representations. The model is trained to predict these tokens, associate them with event descriptions, and improve its final predictions through reinforcement learning. The paper also introduces VEPBench, which distinguishes the recorded continuation from plausible alternatives, and evaluates IDS against textual and latent-reasoning baselines on VEPBench and FutureBench. The reported improvements are greatest for recorded-future selection and one-hop prediction.

Critical strengths
  1. The improvement before reinforcement learning is substantial. IDS raises SR@A from 54.41% to 67.00% over textual reasoning with the same backbone, providing meaningful evidence that the future-supervised training pipeline is useful.
  2. The benchmark distinguishes agreement with the recorded future from selection of a plausible continuation. This is a sensible evaluation choice for a task in which several future events may be reasonable.
Critical weaknesses
  1. The results make a stronger case for future visual supervision than for discretization itself. Because the representation, objective, and training procedure change together, the paper leaves unclear what quantization contributes beyond a convenient next-token training interface. Besides, it remains unclear whether the generated tokens carry information used in the final prediction. Stage 2 conditions on ground-truth future tokens, while inference uses predicted ones, and no direct analysis establishes the effect of their sample-specific content.
  2. The FutureBench results are strongest for one-hop prediction and less competitive on longer event sequences. They provide limited insight into whether the proposed states support reasoning across events or primarily help predict the immediate continuation.
Questions for authors
  1. What is the main advantage of predicting quantized future-video features rather than future-event descriptions or continuous future-video features? The current results demonstrate a benefit from future supervision, but what evidence identifies the discrete representation as the important factor?
  2. How much does the final prediction depend on the content of the generated visual tokens? In particular, would replacing them with unrelated tokens from other examples change the answer, and how large is the gap between predicted and ground-truth future-token conditioning?
  3. What do the contrasting one-hop and multi-hop results suggest about the method’s capabilities? How far do the authors expect a single predicted future-token sequence to support reasoning over successive events? Table 6 shows that rewarding plausible alternatives reduces both primary metrics. How do the authors interpret this result in relation to the uncertainty of future events and the choice to optimize agreement with one recorded continuation?
  4. How should the FutureBench results be interpreted in light of the weaker performance on multi-hop prediction? Do the authors see this as a limitation of the fixed-length future-token representation, the training objective, or the available supervision?
5 comments

Reviewer 1ZX4

Thank you for your thoughtful review and for recognizing the gains before RL and the distinction between recorded and plausible futures. We clarify the role of discrete visual tokens through controlled interventions and training comparisons, and discuss the interpretation and scope of the multi-hop results.

We label our responses to the weaknesses as W1, W2, … and to the questions as Q1, Q2, …, following their order in your review.

W1

IDS's advantage comes from its complete future-prediction, semantic-alignment, and RL design and its fit to MLLM learning. Stages 1/2 use standard NTP; visual/text token probabilities support standard GRPO and the same inference interface, without extra feature-matching or visual-state RL objectives. The feature-similarity losses in the compared continuous methods may have optimization directions different from NTP; the smoother training curves in Figure 6 (Appendix F.1) provide an auxiliary observation. We hypothesize that avoiding extra objectives focuses learning on future visual prediction and semantic interpretation, helping explain gains under matched training. With comparable variables controlled, IDS leads on most core same-backbone metrics (manuscript Tables 1 and 2).

In the same-backbone performance comparisons in manuscript Tables 1 and 2, we control all other variables, including training parameters, except method-specific or faithful-reproduction differences: the sample set and SFT sample counts match, controlled RL uses 1,000 steps, and continuous baselines share the text and future-video supervision. These comparisons assess the integrated discrete prediction–alignment design, including its representation, objectives, and training procedure.

Stage 2 uses GT tokens for correct semantic alignment, and the added controlled interventions test the downstream role of this visual conditioning. Pairing tokens and event descriptions from the same actual future video avoids aligning erroneous predictions with correct descriptions, teaching the model to describe the future from visual tokens. Stage 3 further optimizes trajectories containing the model's own predicted tokens and subsequent answers; manuscript Table 3 evaluates this progression using predicted tokens throughout inference. We added conditioning interventions with the checkpoint, questions, observed videos, and candidates held fixed (Table 7, reproduced at the end of Q2; Appendix J). GT conditioning improves SR@A/SR@AB by 13.30/12.05 percentage points over predicted tokens, diagnosing their downstream gap in the same model. Replacing all 84 tokens with another sample's real tokens from the same visual vocabulary lowers performance, directly supporting the downstream role of sample-specific visual content. Cross-sample Invalid is 30.87%, so effects involve both prediction accuracy and valid continuation. Retained prediction ability is consistent with continued use of the observed video and some robustness to mismatched tokens.

W2

IDS is also competitive on multi-hop prediction; its gains are not confined to the immediate next event. In manuscript Table 2, IDS exceeds all evaluated same-backbone latent methods on 2-/3-Hop and improves on non-latent GRPO by 4.14/3.98 percentage points. SwimBird, which scores higher on multi-hop tasks, uses Qwen3-VL-8B rather than our Qwen2.5-VL-7B. The higher Qwen3-VL-30B-A3B scores in manuscript Table 2 also show substantial performance differences across backbones, providing important context for cross-model multi-hop comparisons. We will clarify these multi-hop results and comparisons in the main text.

FutureBench's hop count denotes the number of consecutive events to infer; its textual conditions also matter when interpreting the scores. The 1-/2-/3-Hop tasks require predicting one, two, or three consecutive future events, respectively. We added an original 3-Hop question from the official release in Appendix K. Such questions jointly assess visual prediction, event ordering, and consistency with a textual endpoint. Their scores therefore do not directly equate to the ability to predict longer future-event sequences without a textual endpoint cue. IDS's 3-Hop score is 0.99 percentage points below textual SFT, possibly because these textual conditions better suit language-relation judgments.

VEPBench also contains multi-hop samples; one 84-token sequence represents future features, not a fixed number of events. Neither the benchmark coverage nor the method design is restricted to single-event prediction. The event span represented by a visual sequence depends on the retained visual information and training supervision; multi-event videos and corresponding event-chain descriptions can provide learning signals for transitions across events. With more training data and compute, increasing the number of visual tokens may further expand future-event modeling capacity.

Q1

Compared with predicting only future-event text, IDS first learns future visual intermediate states and then learns to describe the future from them. Textual Caption directly trains alignment between observed videos and future-event descriptions, yet Stage 1+2 IDS performs substantially better under the same conditions (manuscript Table 4). This comparison supports the value of future visual tokens and their prediction–semantic-alignment pathway.

Compared with continuous future features, discrete tokens directly fit standard SFT/RL losses and autoregressive generation without additional optimization objectives. We hypothesize that this reduces the burden of coordinating objectives and focuses learning on future visual prediction and semantic interpretation. Continuous baselines also use the same text and future-video supervision; with comparable variables controlled as far as possible, IDS achieves the best results on the vast majority of core metrics in same-backbone comparisons (manuscript Tables 1 and 2). These comparisons support the performance value of the discrete prediction–alignment design, evaluating quantization together with its associated objectives and training procedure. See W1 for the optimization motivation, training controls, and descriptive observations from Figure 6 (Appendix F.1).

The added diagnostics and controlled interventions characterize discrete visual-token prediction quality and its influence on the final prediction. The results support the functional role of sample-corresponding visual content; see Q2, Table 7, and Appendix J for the detailed results and analysis.

Q2

We added GT-token substitution to quantify the performance gap between predicted-token and ground-truth future-token conditioning. Starting from the same checkpoint and holding the questions, observed videos, and candidates fixed, GT conditioning improves SR@A/SR@AB by 13.30/12.05 percentage points over normal prediction (Table 7, reproduced below). Normal inference generates visual tokens from the observed video; GT conditioning in this experiment is diagnostic.

Cross-sample substitution directly tests token–video correspondence and indicates some robustness to mismatched tokens. Both matching-GT and cross-sample conditions use 84 real future tokens and the same visual vocabulary, changing their correspondence to the current video. Cross-sample SR@A/SR@AB is 58.40%/60.30%, with Invalid at 30.87%. Relative to matching GT, both prediction performance and valid continuation are affected, supporting the downstream role of sample-specific visual content. Substantial prediction ability remains after substitution, consistent with continued reasoning from the observed video and some tolerance of mismatched tokens.

We added random perturbations and normal-generation diagnostics as complementary evidence. Replacing 21 GT positions with random visual tokens, replacing all 84 positions with random full-vocabulary tokens, and randomly initializing codebook weights affect prediction and valid continuation; Table 7 therefore reports accuracy alongside Invalid. We also added normal free-generation diagnostics: 52.45% positionwise GT agreement, 80.50% codebook utilization, and exactly 84 legal visual tokens in 95.17% of samples (Appendix J). Visual-sequence validity is measured separately from final-answer Invalid. Together, these results support the value of visual tokens as future intermediate information actually used by the model, alongside some tolerance of token errors.

Visual-token conditioning and codebook interventions (revised manuscript, Table 7). All entries are percentages; Invalid is the final-answer invalid rate.

ConditionSR@ASR@ABSR@CSR@DSR@EInvalid
Normal prediction72.78‾\underline{72.78}76.62‾\underline{76.62}13.419.410.570.00
GT future tokens (84)86.0888.672.138.590.610.00
Cross-sample real future tokens (84)58.4060.301.06‾\underline{1.06}7.150.6130.87‾\underline{30.87}
Random full-vocabulary tokens (84)26.1627.150.912.740.0869.13
GT: replace 21 with random visual tokens57.2659.161.526.01‾\underline{6.01}0.46‾\underline{0.46}32.85
Randomly initialized codebook weights0.000.000.000.000.00100.00

Q3

IDS's gains extend to multi-hop tasks and are not confined to immediate continuation prediction. In manuscript Table 2, IDS exceeds the evaluated same-backbone latent methods on 2-/3-Hop and improves on non-latent GRPO by 4.14/3.98 percentage points. Its 3-Hop score is 0.99 points below textual SFT, possibly because FutureBench's textual endpoints and candidate event chains better suit language-relation judgments. Cross-backbone comparisons must also account for SwimBird's Qwen3-VL versus IDS's Qwen2.5-VL (see W2 and Appendix K).

A single 84-token sequence represents future features, not a fixed event count or reasoning depth. VEPBench also contains multi-hop samples. The supported event span depends on retained information and supervision; successive-event videos and descriptions provide signals for learning event transitions. The method itself does not restrict prediction to a single event or prescribe a fixed hop count. With more data and compute, expanding supervision coverage and increasing the number of visual tokens may further extend its modeling capacity.

The recorded future provides verifiable training feedback, while SR@AB accepts a human-verified plausible alternative. B's plausibility annotation specifies neither its occurrence probability nor a requirement to receive the same accuracy reward as A. Rewarding A aligns with actual-future supervision in Stages 1/2; eta=0 gives the best primary metrics tested in Table 6 (Appendix H). Both SR@A and SR@AB also improve when accuracy rewards are given only to A (manuscript Table 3), supporting this reward under the current A/AB criterion. This gain measures overall accuracy within the designated alternatives and can also be driven by better A matching.

Adding B rewards changes selection preferences without improving the primary metrics in the tested configuration. At eta=0.3, B selection increases while A and AB decline. We hypothesize that, under limited RL training, B rewards may favor features shared by plausible continuations over fine-grained temporal/action cues distinguishing the recorded branch. The increase in E shows that the selection shift extends beyond A/B. These observations motivate further study of how reward design changes selection preferences.

To directly assess broader future plausibility, we are conducting an additional evaluation without providing the recorded future to the judges. Judges assess predictions using only the observed context and historical video; we will add the results to the appendix and rebuttal as complementary evidence once the evaluation is completed.

Q4

IDS leads the evaluated same-backbone latent methods on multi-hop tasks; its FutureBench gains are not limited to 1-Hop. In manuscript Table 2, IDS achieves 60.62%/56.22% on 2-/3-Hop, exceeding the evaluated same-backbone latent methods and improving on non-latent GRPO by 4.14/3.98 percentage points. Interpreting these results therefore requires distinguishing same-backbone method comparisons from rankings across backbones.

SwimBird's higher multi-hop scores should be interpreted in light of its different backbone. SwimBird uses the newer Qwen3-VL-8B, whereas IDS uses Qwen2.5-VL-7B. The higher Qwen3-VL-30B-A3B scores in manuscript Table 2 also highlight backbone differences relevant to these comparisons.

Textual SFT has only a small 3-Hop advantage, possibly related to FutureBench's textual conditions. IDS scores higher than textual SFT on 2-Hop and 0.99 percentage points lower on 3-Hop. The original question we added in Appendix K illustrates textual endpoints and candidate event chains: answering also involves judging event order, transitions, and endpoint consistency. Such conditions may more directly match the language relations learned by textual SFT, offering a possible explanation for this difference (see also W2).

Neither the method design nor its training objectives restrict prediction to a single event; we consider training-data coverage the more important factor for longer-sequence capability. One 84-token sequence represents future visual features, not a fixed number of events or hops. NTP, visual-text alignment, and GRPO can all be applied to successive-event sequences; learning the corresponding capability requires training data covering longer future segments, event transitions, and their descriptions. We thus focus on the temporal span and event-chain coverage of supervision.

More long-sequence supervision can be paired with a larger visual-token budget to extend long-sequence modeling capacity. With more long-sequence videos and descriptions, the 84-token budget can be expanded to retain richer event transitions. With appropriate data and compute, this may improve longer-sequence prediction. The future-token prediction, semantic alignment, and GRPO objectives can all be retained, so the underlying IDS method remains applicable.

Reviewer_ZLZcOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies video event prediction, where an MLLM predicts likely future events from an observed video segment. It proposes Imagine in Discrete Space (IDS), which represents future-video features as discrete visual tokens. A frozen vision encoder extracts future features, and an EMA-initialized codebook quantizes them into fixed-length token sequences that are added to the MLLM vocabulary. IDS uses a three-stage training pipeline: predicting future visual tokens autoregressively, grounding these tokens in textual event descriptions, and refining multiple-choice event prediction with GRPO. At inference, the model sees only the observed video, generates its own future visual tokens, and conditions the final prediction on them. The paper also introduces VEPBench, a roughly 3K-sample benchmark with recorded futures, plausible alternatives, and typed distractors, along with a source-disjoint 17K-video training set. On VEPBench, IDS raises SR@A from 54.41% for the textual-reasoning SFT baseline to 67.00% before RL and 72.78% after GRPO. It also transfers to FutureBench, showing its strongest gains on one-hop prediction while remaining less competitive on multi-hop tasks.

Critical strengths
  1. Simple and practical formulation. IDS converts future-video features into vocabulary tokens, enabling autoregressive cross-entropy training without pixel reconstruction, a video decoder, or backbone modifications. The paper also clearly frames these tokens as feature-level abstractions rather than simulated future videos.

  2. Strong controlled results. With the same Qwen2.5-VL-7B backbone and SFT budget, IDS improves SR@A from 54.41% to 67.00% before RL and 72.78% after GRPO. Stage-wise ablations further demonstrate the contributions of semantic grounding and RL.

  3. Informative benchmark and broad evaluation. VEPBench distinguishes recorded and plausible futures from three typed distractors, providing more diagnostic metrics than standard accuracy. The experiments cover textual and continuous-latent baselines, multiple MLLMs, ablations, open-set evaluation, and transfer to FutureBench, with appropriately cautious interpretation.

Critical weaknesses
  1. The source of the gains is not isolated. IDS changes the representation, supervision, objectives, training stages, and computation simultaneously, while the continuous and textual baselines are not fully matched. Since most improvement appears after semantic grounding rather than future-token prediction, the results support the full pipeline but not the superiority of discretization itself.

  2. The role of generated visual tokens remains unclear. Stage 2 trains on ground-truth tokens, whereas inference uses predicted ones, creating an unquantified exposure gap. No masking, shuffling, or corruption experiments establish that these tokens causally influence the answer, and token accuracy, codebook utilization, and sequence quality are not reported.

  3. The evaluation may favor recorded futures and dataset-specific artifacts. GRPO rewards only the recorded continuation, despite the motivation of modeling multiple plausible futures. Moreover, model-assisted data construction may introduce semantic or stylistic shortcuts that source separation and candidate randomization do not remove; open-set evaluation still relies on model judges and a single recorded future.

  4. Robustness, efficiency, and reproducibility evidence is incomplete. The paper lacks codebook diagnostics, and the inference cost of generating 84 tokens. Transfer gains are concentrated on one-hop prediction, and several dataset, initialization, decoding, and evaluation details remain insufficient for independent reproduction.

Questions for authors
  1. How strongly do the generated visual tokens affect the final prediction? Please report the gap between ground-truth and predicted-token conditioning, along with masking, shuffling, or corruption interventions and token/codebook diagnostics.

  2. How do the authors reconcile rewarding only the recorded future during GRPO with the motivation of modeling multiple plausible futures? Please clarify whether IDS primarily improves recorded-future matching rather than broader future plausibility, and explain what existing evidence rules out dataset-construction artifacts.

  3. What are the latency and computational costs of generating 84 visual tokens? Can the authors also provide codebook utilization statistics, multi-hop analyses, and the missing implementation details needed for independent reproduction?

6 comments

Reviewer ZLZc

Thank you for your careful review and constructive suggestions. We address your concerns below by clarifying the contribution and training comparisons, adding visual-token diagnostics and efficiency measurements, and explaining the benchmark and reward design.

We label our responses to the weaknesses as W1, W2, … and to the questions as Q1, Q2, …, following their order in your review.

W1

The advantage of discrete tokens lies in the full pipeline's compatibility with MLLM learning, rather than in greater expressive power relative to continuous representations. We use discrete tokens to unify NTP, reduce objective coordination, and focus on future prediction and semantic interpretation. The evaluated continuous methods add a feature-similarity loss whose direction may differ from NTP. IDS accommodates standard SFT/RL losses without extra objectives; Figure 6 (Appendix F.1) and related analyses provide auxiliary observations.

Experimental results show that the discrete-token IDS approach also delivers clear performance gains. For the performance comparisons in manuscript Tables 1 and 2, we compare multiple continuous latent methods, controlling all variables except method-specific and faithful-reproduction differences. Comparable settings match training samples, parameters, steps, and text/future-video supervision. IDS is best on the great majority of key metrics in same-backbone comparisons (manuscript Tables 1 and 2).

Stage 2 teaches the model to describe future events from visual tokens, making Stage 1's prediction useful for final event judgment. All manuscript Table 3 configurations use self-predicted tokens at inference; SR@A rises from 37.21% after Stage 1 to 67.00% after Stages 1+2. Gains concentrated in Stage 2 thus fit its intended role. Textual Caption already learns direct observed-video/future-event-text alignment, yet IDS after Stages 1+2 substantially outperforms this baseline under the same conditions (manuscript Table 4). The key difference is future visual tokens and the process of predicting and semantically aligning them. These comparisons support discrete future-token prediction and semantic alignment as an integrated design, evaluating the representation together with its associated learning procedure.

W2

GT tokens provide consistent semantic pairings in Stage 2; Stage 3 learns to use self-predicted tokens. Pairing tokens and descriptions from the same future video avoids aligning erroneous predictions with correct text. Stage 3 optimizes trajectories containing its own generated tokens and subsequent answers. Manuscript Table 3 evaluates this progression using predicted tokens at inference throughout. The added GT-token substitution experiment quantifies the downstream performance difference between predicted-token and GT-token conditioning in the same model (Table 7, reproduced below), providing a diagnostic of the potential benefit of better future-token prediction.

Our added interventions support visual tokens' downstream value and some robustness to intermediate-state errors (Table 7). With the checkpoint, questions, observed videos, and candidates fixed, matching GT outperforms normal prediction. Substituting another sample's real tokens while retaining 84 positions and the visual vocabulary lowers performance, supporting the value of visual information corresponding to the current video. Retained prediction ability under cross-sample substitution is consistent with continued reasoning from the observed video and some robustness to mismatched tokens. Random-token corruption and randomly initialized codebook weights also affect valid continuation, so we report Invalid alongside prediction metrics to capture effects on both prediction and output validity.

We additionally computed three normal-inference diagnostics. The full IDS model's freely generated tokens achieve 52.45% positionwise GT agreement; codebook utilization is 80.50% (entries emitted at least once); and 95.17% of examples generate exactly 84 legal visual tokens. Sequence validity is measured separately from final-answer Invalid. We have added the definitions and results to Appendix J.

Visual-token conditioning and codebook interventions (revised manuscript, Table 7). All entries are percentages; Invalid is the final-answer invalid rate.

ConditionSR@ASR@ABSR@CSR@DSR@EInvalid
Normal prediction72.78‾\underline{72.78}76.62‾\underline{76.62}13.419.410.570.00
GT future tokens (84)86.0888.672.138.590.610.00
Cross-sample real future tokens (84)58.4060.301.06‾\underline{1.06}7.150.6130.87‾\underline{30.87}
Random full-vocabulary tokens (84)26.1627.150.912.740.0869.13
GT: replace 21 with random visual tokens57.2659.161.526.01‾\underline{6.01}0.46‾\underline{0.46}32.85
Randomly initialized codebook weights0.000.000.000.000.00100.00

W3

Accuracy rewards target the recorded future, while SR@AB recognizes the designated plausible alternative. Table 6 (Appendix H) shows that eta=0 gives the best tested primary metrics and aligns with actual-future supervision in Stages 1/2, motivating zero B accuracy reward. Rewarding only A improves both SR@A and SR@AB (manuscript Table 3), supporting recorded-future prediction and overall A/B accuracy. SR@AB can also increase through better A matching; it measures acceptance within these designated alternatives. At eta=0.3, B selection increases while A and AB decline. We hypothesize that, under limited RL training, B rewards may favor features shared by plausible continuations over fine-grained temporal/action cues distinguishing the recorded branch. The increase in E shows that the selection shift extends beyond A/B. These observations motivate further study of how reward design changes selection preferences.

Human semantic checks and cross-setting results jointly support construction quality. All approximately 3K evaluation samples are checked for A correctness, B plausibility, and distractor validity; unsuitable samples are revised or removed. Human inspection found concise, accurate options with complete event elements and no identified semantic or stylistic shortcuts unrelated to video content. Appendix A gives this example in canonical A–E order:

A. The woman caresses the cheek of the deceased soldier in the coffin. B. The woman stands up and delivers a eulogy in honor of the deceased soldier. C. The woman leans over and touches the face of the deceased soldier in the coffin. D. A man caresses the cheek of the deceased soldier in the coffin. E. The woman closes the coffin lid, covering the soldier inside.

Stages 1/2 receive no training candidates, yet manuscript Table 4 shows clear gains; external FutureBench results further support transferable benefits beyond VEPBench's candidate wording.

We added Grok 4.7, Kimi, and judge-agreement analysis to reduce single-judge dependence. IDS remains among the leading methods under all four judges (revised manuscript Table 1); mean Kendall's W is 0.813 (Appendix A.2), indicating strong agreement on relative prediction quality and supporting the open-set protocol. Open-set scoring uses the actual future video as a visually verifiable reference to support objective, reproducible evaluation and complement closed-set multiple-choice tests, which do not directly assess open-set future prediction. SR@AB additionally accepts the human-verified plausible alternative B, providing a complementary measure of acceptance of multiple plausible futures.

To directly assess broader future plausibility, we are conducting an additional evaluation without providing the recorded future to the judges. Judges assess the plausibility of predicted events using only the observed context and historical video. Once the evaluation is completed, we will add the results to the appendix and rebuttal as complementary evidence.

W4

We added full-IDS normal-inference diagnostics (Appendix J): freely generated tokens have 52.45% positionwise GT agreement, 80.50% codebook utilization (entries emitted at least once), and 95.17% of samples generate exactly 84 legal visual tokens. Sequence validity differs from final-answer Invalid. Controlled perturbations (Table 7, reproduced in R1-W2) indicate some robustness to mismatched visual conditioning.

We added complete-generation timing including the 84 visual tokens; these were not timed separately. On FutureBench, with all inference settings held identical, mean model.generate wall time for complete IDS/textual-CoT responses is 6.632/7.972 s, a 16.81% reduction. This accords with our motivation that visual intermediate states may reduce the burden of expressing future changes entirely in text. Estimated IDS training cost is approximately 100 GPU·h.

IDS also performs strongly on multi-hop tasks in same-backbone comparisons. In manuscript Table 2, IDS exceeds all evaluated same-backbone latent methods on 2-/3-Hop and improves on non-latent GRPO by 4.14/3.98 points. SwimBird scores higher on multi-hop tasks but uses Qwen3-VL-8B versus our Qwen2.5-VL-7B; the higher Qwen3-VL-30B-A3B scores also illustrate substantial differences across backbones. IDS nevertheless scores much higher than SwimBird on 1-Hop. Its 3-Hop score is 0.99 points below textual SFT, possibly because FutureBench's textual endpoints/candidate event chains better suit language-relation judgments (Appendix K).

We will add key implementation details to the appendix and release complete reproduction materials. These details cover codebook initialization, code selection, stage-wise training, and generation/evaluation settings. We commit to open-sourcing all training and evaluation data, the complete training code, and official evaluation scripts for independent reproduction.

Q1

The added controlled interventions and token diagnostics are reported in full in our response to W2. At the same checkpoint, GT conditioning improves SR@A/SR@AB by 13.30/12.05 percentage points over predicted tokens. Cross-sample substitution and random corruption further support the downstream value of visual tokens; retained prediction ability after cross-sample substitution also indicates some robustness. See Table 7 in our response to W2 for the full comparisons; prediction accuracy, codebook utilization, and sequence validity are reported in W2 or Appendix J.

Q2

IDS uses the verifiable recorded future as its direct training target, while SR@AB also accepts a plausible alternative. A is grounded in the actual continuation video; B is annotated as a plausible possibility, without specifying its occurrence probability or corresponding training reward. Even with accuracy rewards given only to A, SR@A and SR@AB both improve (manuscript Table 3), and Table 6 in Appendix H supports our current reward configuration. The current evidence primarily supports improved recorded-future prediction and overall accuracy within the designated A/B set; the AB gain can be driven by better A matching. See our response to W3 for the detailed analysis.

To directly assess broader future plausibility, we are conducting an additional evaluation without providing the recorded future to the judges. Judges assess the plausibility of predicted events using only the observed context and historical video. Once the evaluation is completed, we will add the results to the appendix and rebuttal as complementary evidence.

Human verification and cross-setting results jointly support benefits beyond candidate wording. Verification of all evaluation samples and the example in Appendix A support candidate quality. Candidate-free training in Stages 1/2, open-set prediction, and external FutureBench results provide complementary evidence (manuscript Tables 1, 2, and 4). See our response to W3 for the verification procedure, example, and related analysis.

Q3

We added complete-generation timing that includes the 84 visual tokens, together with codebook diagnostics. Under identical inference settings, mean generation time for complete IDS/textual-CoT responses is 6.632/7.972 s, a 16.81% reduction. Estimated IDS training cost is approximately 100 GPU·h; measurement details are in our response to W4. Codebook utilization is 80.50%; the remaining token diagnostics are reported in W4 or Appendix J.

IDS also benefits multi-hop prediction; comparisons and interpretations are provided in W4 and manuscript Table 2. Appendix K adds an original multi-hop question to illustrate its textual conditions. We will add key implementation details to the appendix and open-source all training and evaluation data, the complete training code, and official evaluation scripts to support independent reproduction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.