acceptodds
Withdrawn from ICLR 2027

Learning Predictively Grounded Event Chains for Video Event Prediction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video event prediction requires inferring an unseen continuation from visual history. An explicit event chain may be visually supported but omit predictive evidence, or appear predictive by inventing events. We define predictive grounding as the ideal conjunction of interval-level visual consistency (VC) and candidate-conditioned predictive sufficiency (PS). We introduce \methodfull (), which optimizes verifier-relative proxies for both criteria without event-chain annotations or a task-specific supervised cold start. A shared policy first samples an option-blind timestamped history, then predicts from the video, candidates, and an immutable copy of that history. Pass 2 may revisit the video but cannot revise the chain. Frozen visual and blindfolded text verifiers supply training rewards; PS measures discriminative usefulness, not factual correctness. Five-state perception-aware credit routing assigns separate advantages to chain and prediction tokens. All verifiers are removed at inference. Reported point estimates on FutureBench, VEPBench, and AVEP favor the combined method under separate benchmark-specific training. Single-verifier ablations show an alignment–sufficiency trade-off, and chain interventions support behavioral dependence without establishing a causal mechanism. Aggregate metric inconsistencies and diagnostic aggregation remain to be reconciled with evaluation records, limiting the quantitative conclusions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Computer Vision, Vision-Language Mode, Video Event Prediction

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_ZTEpOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies video event prediction and introduces PGE-RL, a reinforcement learning method for generating timestamped event chains that are both consistent with the observed video and useful for predicting subsequent events. A shared policy operates in two passes: it first describes the observed history without access to the question or candidate answers, then predicts using the original video, candidates, and an immutable copy of the chain. During training, frozen visual and text verifiers provide visual-consistency and predictive-sufficiency rewards, while perception-aware credit routing (PACR) assigns separate learning signals to the chain and prediction stages based on answer correctness and verifier assessments.

Critical strengths
  1. The paper clearly distinguishes visual consistency from predictive usefulness, recognizing that an accurate but generic description and a discriminative but unsupported description are both insufficient for reliable prediction. Jointly optimizing these properties is well motivated by the task.

  2. Generating an option-blind history and preserving it as fixed context for prediction makes the information boundaries between the two stages explicit. Stage-specific credit assignment is also well motivated, seeking to avoid rewarding incorrect observations after a correct answer or suppressing correct observations after a prediction failure.

Critical weaknesses
  1. The main experimental results contain unresolved aggregation discrepancies. Appendix A.6 identifies FutureBench averages that cannot be reconstructed using the official category weights, VEPBench rates that do not correspond to the stated sample count, and AVEP metrics and split labels requiring reconciliation; without per-example predictions and the actual aggregation rules, the reported gains and comparisons cannot be reliably verified.

  2. The training rewards and primary chain diagnostics rely on the same frozen verifiers, with limited evidence of robustness under alternative verifiers or independent evaluation. The 500-example human audit reports only overall agreement above 90%, without class distributions, classwise errors, or inter-annotator agreement, leaving uncertainty about whether the policy learns reliable visual evidence extraction or adapts to the particular verifiers’ preferences.

Questions for authors
  1. Please explain in detail the metric-aggregation and evaluation-record inconsistencies discussed in Appendix A.6, including why the FutureBench averages cannot be reconstructed using the official category weights, why the VEPBench rates do not correspond to the stated sample count, and which AVEP metrics and split labels require reconciliation. Please clarify the specific discrepancies, their causes, and their implications for the experimental conclusions.

  2. How does an answer-only reward baseline perform under the same two-pass interface? Could the authors also compare event chains without timestamps and length-matched free-form reasoning, and report realized generated-token counts and training/inference costs to distinguish the effects of the representation, reward design, and additional computation?

  3. Do the full method’s visual-consistency and predictive-sufficiency advantages over the single-verifier variants persist under independent verifiers or human evaluation? Please provide the audit sampling procedure, positive/negative class distributions, classwise error rates, and inter-annotator agreement, and clarify the sensitivity of PACR to changes in verifier thresholds.

Reviewer_raA2OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes PGE-RL (Predictively Grounded Event-chain Reinforcement Learning), a reinforcement learning framework designed for Video Event Prediction (VEP)—the task of predicting future events following an observed video clip. The primary goal of PGE-RL is to train a policy model to generate timestamped event chains that simultaneously satisfy Visual Consistency (VC) and Predictive Sufficiency (PS), a conjunction the authors define as Predictive Grounding (PG). Notably, the framework optimizes the policy via Group Relative Policy Optimization (GRPO) without relying on event-chain annotations or a Supervised Fine-Tuning (SFT) cold start.

Critical strengths
  1. Well-motivated supervision. Interval-level VC and blindfolded PS target complementary failure modes (true-but-generic vs. useful-but-hallucinated chains), and Eq. (7) explicitly discounts candidate priors through an options-only control.
  2. Careful interface and credit design. Option-blind, immutable chains remove a direct route for copying candidates into the history, and PACR avoids penalizing useful perception for downstream errors or rewarding unsupported chains for lucky answers.
Critical weaknesses
  1. The reported numbers cannot be traced to a consistent evaluation protocol. On FutureBench, PGE-RL's AVG (86.10) is exactly the unweighted mean of its category scores, whereas the example-weighted mean using the official counts (173/193/201/489), the convention stated in Appendix A.6, is 85.56. Six baseline rows follow the weighted rule exactly, and the SFT/GRPO controls match neither rule. None of PGE-RL's four category scores is attainable from the official counts even at one decimal (e.g., 1-Hop 87.0 vs. 150/173 = 86.71 or 151/173 = 87.28), and 18 of 22 rows contain such a value. On VEPBench, SR@A of 68.95 is not attainable with n = 3,000 (68.93/68.97), and 15 of 21 rows contain at least one unattainable value. Appendix A.6 acknowledges part of this, but disclosure does not establish what was evaluated.

  2. Anomalies compared to prior work. In Table 2, the LVR, Monet, SwimBird, and the authors' own Qwen3-VL-8B-GRPO rows have nearly identical fractional parts on all five metrics (e.g., SR@A .41 and SR@C .17 in every row), differing mainly by integer offsets. On AVEP, every row attributed to prior work satisfies F1 = harmonic mean(P, R) within rounding, but both rows from this study fail it. Their noun F1 exceeds the harmonic mean (e.g., PGE-RL validation 69.00 vs. 66.61), which per-item averaging cannot produce because the harmonic mean is concave. The manuscript also describes its tables as preserving supplied point estimates that have not been independently reproduced here.

  3. Most of the gain persists without the chain. Table 5 shows that the policy uses the chain (removing it costs 12.60/5.15 points). Yet the condition without the chain mechanism still scores 73.50 on FutureBench and 63.80 on VEPBench, 19.83/9.39 points above the answer-only GRPO control evaluated with video. This means that roughly 61%/65% of the gain remains without any chain. On FutureBench, question and options alone also beat vanilla Qwen3-VL-8B and GRPO with video. Because the baseline template in Appendix A.3 has no question field and FutureBench questions contain outcome or intermediate-event anchors, unequal access to the question must be ruled out.

  4. The controls are not enough and not matched with interface. On FutureBench, the SFT and answer-only GRPO controls fall below the untrained backbone, suggesting under-tuning, and the validation split used for checkpoint selection is not stated. The GRPO control is single-pass, and there is no two-pass answer-only control or MoCA-style routing baseline. Since event chains with clip-similarity rewards (Video-CoE) and blindfolded perception credit (MoCA) both have precedents, these controls are needed to show what the combination adds.

Questions for authors
  1. Results. Please release per-example predictions and split manifests, regenerate Tables 1–5 under one stated aggregation rule, and identify any rows copied from prior work. Do the claimed gains and rankings survive?
  2. Anomalies. How were the four Table 2 rows with shared fractional parts produced, and which evaluator and aggregation produced the AVEP P/R/F1 values for your own rows?
  3. Protocol. Did PGE-RL, the controls, and the baselines receive the same question text (including FutureBench anchors), candidates, and frames? Please show serialized inputs and state the training and validation splits for each benchmark.
  4. Attribution. Why does the no-chain condition outperform answer-only GRPO by about 20 points? Do the gains hold over a tuned two-pass answer-only GRPO and a MoCA-style routing rule, with learning curves and at least three seeds?
Reviewer_TAumOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents a method for video event prediction.

Critical strengths

See weaknesses below.

Critical weaknesses

I find the paper of questionable novelty [1-3] and very difficult to understand. The paper uses overly complex math notation and unclear language that is difficult to follow. The paper is not well placed with respect to related tasks, such as early action prediction and action anticipation [4, 5].

Regarding results, there are some important issues to address: i) FutureBench results are an unweighted mean, and this is not consistent with the related work, ii) All of PGE-RL’s AVEP numbers are rounded integers (15.00, 52.00, 95.00, ...), as are the GRPO results, while CoE’s are not, and this raises suspicion, iii) there are unaddressed LLM outputs in the manuscript ("Tables preserve the supplied point estimates", "adapted from the author-provided slide", "No executable training/evaluation package ... accompanies these PDFs; the recorded point estimates have not been independently reproduced here")

I do not find myself an expert in the field, however I think the conference audience would not benefit from the current state of this manuscript.

[1] Qile Su (Video-CoE, CVPR 2026): https://cvpr.thecvf.com/virtual/2026/poster/36744 [2] Haozhe Wang (MoCA, “Bad Seeing or Bad Thinking?”, ICML 2026 oral): https://icml.cc/virtual/2026/oral/71095 [3] Yifan Jiang (VideoP2R, CVPR Findings 2026): https://openaccess.thecvf.com/content/CVPR2026F/html/Jiang_VIDEOP2R_Video_Understanding_from_Perception_to_Reasoning_CVPRF_2026_paper.html [4] Sović, Iva, Ivan Martinović, and Marin Oršić. "EAST: Early Action Prediction Sampling Strategy with Token Masking." arXiv preprint arXiv:2604.18367 (2026). [5] Girdhar, Rohit, and Kristen Grauman. "Anticipative video transformer." 2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2021.

Questions for authors

See weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.