acceptodds
Withdrawn from ICLR 2027

VepAgent: Bridging Causal-Transition via Tool-Augmented Reinforcement Learning for Video Event Prediction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated remarkable potential in video understanding, yet their reliance on retrospective summarization and text-centric priors often limits their ability to bridge unobserved causal transitions when applied to Video Event Prediction (VEP). To address this, we propose VepAgent, an agentic framework that integrates causal-transition reasoning with tool-augmented reinforcement learning (RL) for robust VEP. Unlike prior methods that passively project future trajectories from historical dependencies, our approach explicitly models the logical progression from terminal observed states to future events. Specifically, we first construct futurebench-4K, a high-quality chain-of-thought dataset for supervised fine-tuning (SFT) that effectively bridges the causal-logic gap by structuring the deduction of unobserved intermediate states. Subsequently, we develop a diagnostic tool library integrating state tracking, frame retrieval, and region magnification, enabling the agent to dynamically augment reasoning with external tools to recover missing spatio-temporal evidence and resolve visual ambiguities during inference. Moreover, we propose a composite reward mechanism that jointly optimizes prediction accuracy, causal coherence, and reliable prior, compelling the agent to rely on genuine visual grounding rather than superficial textual similarities. Extensive evaluations on FutureBench and NEPBench datasets demonstrate that our method achieves state-of-the-art performance, significantly outperforming larger MLLMs and validating the empirical effectiveness of our agentic, future-oriented reasoning paradigm.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision Language Model, Agentic Reinforcement Learning, Video Event Prediction, Chain-of-Thought

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_J8dTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This article focuses on the problem of video event prediction. It introduces an agentic framework VepAgent that is trained with tool-augmented RL over a self-built CoT dataset futurebench-4k. For RL training, the authors introduce two dedicated rewards, i.e., a causal gap reward and an anti-prior reward. VepAgent is compared against a series of MLLMs on FutureBench and NEPBench, and shows promising performance.

Critical strengths
  • The motivation to explore an agentic framework for the complex video event prediction task makes sense to me, and offer a promising direction for the domain.
  • The approach demonstrates substantial performance improvements over existing MLLMs on standard datasets. Particularly noteworthy is that VepAgent, with a relatively small 4B-parameter base model, outperforms some larger models, e.g., Qwen2.5-VL-72B.
Critical weaknesses
  • While the motivation is clear, the proposed framework primarily integrates existing techniques (e.g., SFT+GRPO), with limited methodological novelty. Although this integration is practically useful, the paper does not sufficiently distinguish its technical contributions from those of existing tool-augmented video MLLMs.
  • The experimental evaluation has several important limitations. First, the approach is compared only against standalone, general-purpose MLLMs. However, it is clear that the tool access and computational budgets between the proposed approach and the competitors are different. And task-specific VEP methods are not compared. This makes the comparison unfair and the results unconvinced. Second, experiments are conducted exclusively with a 4B-parameter backbone, and the generalizability and scalability of the method across different backbones/model sizes are unexplored. Third, the evaluation does not include proprietary MLLMs (e.g., GPT or Gemini).
Questions for authors
  • It is unclear how the agent learns to invoke the diagnostic tools. Does the futurebench-4K dataset include tool-call annotations in CoTs?
  • The paper only provides a conceptual description to the causal gap reward, a more rigorous formulation is better to help understanding.
  • What is the motivation for treating the format reward as a hard gate rather than an additive reward component? The design choice should be justified.
  • How are the coefficients of the rewards determined? Here lacks a sensitivity analysis to them.
Reviewer_UKYdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VepAgent, a framework designed for video event prediction, a task that requires inferring the most likely subsequent event based on a partially observed video and a set of candidate options. The approach employs supervised fine-tuning using "causal reasoning chains" and incorporates diagnostic tools, including state tracking, focused evidence retrieval, region zooming, and visual detail extraction. Furthermore, the model is optimized via reinforcement learning using a composite reward function that includes an "anti-language-prior" term, compelling the model to perform causal transition reasoning grounded in visual evidence.

Critical strengths

The problem formulation is well-defined, specifically targeting failure modes such as weak causal transition reasoning, reliance on linguistic shortcuts, and the omission of visual evidence.

The design of the causal reasoning chains is logical, requiring the model to distinguish between observed final states, incomplete actions, and future events.

Critical weaknesses

The benchmark coverage is limited; the framework has not been evaluated across a broader range of video understanding scenarios, and its cross-domain generalization remains unclear.

There is a heavy reliance on external models and APIs, data generation, verification, and several diagnostic tools depend on external vision-language models, which makes reproduction costly.

The interaction with diagnostic tools lacks an iterative process; once a tool is selected, no further selections are made, which may limit the flexibility of the evidence chain when dealing with complex videos.

The reward function relies on semantic similarity and embedding-based ranking, which may be sensitive to the quality of reference reasoning, the specific phrasing of options, and the underlying embedding model.

Questions for authors

Please refer to the weaknesses.

Reviewer_j1A2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes VepAgent, a tool-augmented reinforcement learning framework for Video Event Prediction (VEP). It aims to improve future-event reasoning by addressing three challenges: insufficient causal reasoning, reliance on language priors, and missing visual evidence. The method consists of three components: (1) a 4K reasoning dataset for SFT, (2) four visual diagnostic tools for retrieving additional evidence, and (3) GRPO training with rewards for prediction accuracy, causal consistency, and reducing language-prior shortcuts. Experimental results demonstrate the effectiveness of its major components.

Critical strengths
  • Effective integration of reasoning and visual tools. The paper addresses an important challenge in video event prediction. Combining causal-transition reasoning with dynamic visual evidence retrieval is a practical approach, particularly when important visual details are missing from sampled frames.

  • Strong experimental results. VepAgent achieves substantial improvements over existing models despite using a relatively small 4B backbone. The ablation studies also demonstrate the benefits of SFT, RL, and individual diagnostic tools.

Critical weaknesses
  • Unfair comparison due to external VLM tools. The STT and VDE tools rely on an external VLM to provide additional visual understanding. Therefore, VepAgent-4B is not directly comparable to standalone MLLM baselines. The paper lacks controlled comparisons under equivalent model capabilities, visual information, and computational budgets.

  • Limited evidence for improved causal reasoning and reduced language shortcuts. The causal-gap reward relies on textual similarity to reference reasoning rather than directly verifying causal correctness. Additionally, the anti-prior reward mainly reweights correctly predicted samples based on textual similarity, without directly identifying language-shortcut behavior. Consequently, the experiments do not convincingly demonstrate that these rewards improve genuine causal reasoning or visual grounding.

  • Missing comparison with a highly relevant recent work. The paper does not compare against Future-L1: Imagine Before You Predict[1], which also focuses on improving causal reasoning for video event prediction. Although the models and training settings differ, the lack of comparison with this closely related and stronger-performing method weakens the paper's state-of-the-art claim and makes it difficult to assess its novelty and relative effectiveness.

[1]Jiang, Tianxiang, et al. "Imagine before you predict: Interleaved latent visual reasoning for video event prediction." EMNLP (2026).

Questions for authors
  • External VLM contribution: How much of the performance improvement comes from the external VLM used by STT and VDE? Could the authors compare against baselines using the same external VLM and additional visual evidence without the proposed RL framework?

  • Anti-prior reward: Since the anti-prior reward depends only on prediction correctness and the fixed textual similarity rank of the ground-truth answer, how does it explicitly discourage language-prior shortcuts? Could the authors provide direct evidence of improved visual grounding?

  • Causal reasoning evaluation: Does higher similarity to reference reasoning necessarily indicate better causal reasoning? Could the authors evaluate intermediate-state prediction or causal consistency independently of final-answer accuracy?

  • Dataset generalization: Could the authors clarify the training-data sources and provide cross-dataset evaluations to rule out potential benchmark-specific learning?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.