acceptodds
Under review as a conference paper at ICLR 2027

Video-Arena: A Scalable Agentic Video Benchmark for Spatiotemporal Capability

Abstract

Understanding and tracking a dynamic world are fundamental to visual intelligence. Existing video benchmarks provide limited insight into whether models can maintain a coherent understanding of evolving world state. Building such a benchmark from real-world videos is challenging because the underlying world state is partially observable, making annotations hard to obtain. To close this gap, we introduce **Video-Arena**, a benchmark that turns competitive games into scalable, verifiable tests of how well models understand evolving world states from videos. Specifically, inspired by how humans understand dynamic events, we evaluate spatiotemporal world state understanding and tracking through three task categories: perceiving critical events, interpreting state updates, and tracking the resulting states. Our pipeline can generate videos and aligned reference labels from game records *at scale*. To keep evaluation affordable and reproducible, we curate a golden set of 180 tasks across 9 games and run evaluations under two protocols: native visual understanding and tool-assisted agentic analysis. The best-performing systems achieve F1 scores of and under the two protocols, respectively. The result highlights that native models struggle to perceive events reliably, while tool-assisted systems improve event recovery but still struggle to interpret state updates and track the resulting states consistently over time.

Primary area:
datasets and benchmarks
Keywords:
Vision-Language Model, Video Benchmark, Agentic Benchmark, Spatiotemporal Intelligence

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_LZmAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a benchmark to evaluate how well video understanding models perceive events, interpret their effects and also track evolving world states over time. It uses game records and simulators to generate videos and automatically derive GT annotations. This reduces need and effort for manual labeling. The models are evaluated under 2 protocols. One of them is the native visual understanding using sampled video frames and the other is agentic evaluation with access to video-processing tools, code execution and also multi-turn analysis.

Critical strengths
  • It addresses an important gap in the video understanding benchmark space.
  • The paper evaluates event perception, state interpretation and temporal tracking together.
  • The automated data generation pipeline is quite good and enables scalable benchmark construction that has deterministic GT labels. This reduces reliance on manual annotations which are expensive and time consuming and also LLM judges which cannot entirely be trusted.
  • The benchmark has diverse environments and covers 9 games across 6 genres.
  • The experiments are detailed and the tier-wise analysis is also quite good. The additional validation of GT labels and temporal sampling also strengthens the paper.
Critical weaknesses
  • The naive and agentic protocols differ substantially in visual coverage, tool access and also interaction budgets. This makes it a bit difficult to isolate benefits of agentic reasoning.
  • Some GT labels rely on game-engine evals rather than the directly observable states. This can make it difficult to distinguish world-state tracking from game-specific strategic reasoning in my opinion.
  • The F1 metric is strict and requires all event and state fields to match. Now this can maybe obscure whether failures arise from event perception. state interpretation or temporal tracking.
  • The eval also uses only 20 cases per game and one effective run per model-case pair. This can limit reliability estimates.
Questions for authors
  • Can the authors please provide a more controlled comparison? They can use equal visual input budgets to isolate benefits of tool use and multi-turn reasoning.
  • How do you distinguish failures in world state tracking from difficulties in approximating game engines that are used to generate T2 and T3 labels?
  • Please also clarify how much of the low native F1 is caused by limited frame sampling and strict tuple matching rather than actual video-understanding failures?
  • Can you also evaluate multiple runs per model-case pair to motivate the reliability of the results and especially under agentic protocol?
4 comments

Dear reviewer LZmA,

Greetings from the authors!

Thank you for your constructive feedback, which helped us strengthen the paper. We have uploaded the revised manuscript, with changes highlighted in blue:

  • Matched visual input: clarified the joint tool/multi-turn gains at the same 256-frame budget (§4.2; Figure 3).
  • State targets: clarified that T2/T3 use coarse labels for who benefits and who leads, without requiring models to reproduce exact engine win probabilities (§3.1; Appendix B.2).
  • Sampling and strict matching: clarified the limits of temporal ceilings and conditional tier-wise diagnostics (§§4.3–4.4; Appendix E.3).
  • Repeatability: added full 180-case results for two systems, with five visual and three agentic runs (Appendix E.1; Table 12).

As the discussion period draws to a close, could you let us know whether any concerns remain? We hope our rebuttal and revisions address your concerns and further strengthen your recognition and confidence in our paper!

Best regards,

The Authors

Thank you for the response. While most of the concerns are resolved, there are still a few remaining concerns:

W1. I appreciate you isolating the effect of tools and multi-turn reasoning together, but what I meant in my question was is there a way you can isolate the effects of the two separately: How much does the tool use contribute and how much the multi-turn?

W2. I appreciate the clarification that exact engine win probabilities are difficult and unnecessary and I agree with that and also that coarse labels make the task more practical. However, I still feel that it is difficult to distinguish failures in world-state tracking from failures in game-specific strategic reasoning. A model might correctly track the game state but still fail T2/T3 because it cannot assess the position as well as the game engine. I understand that the benchmark evaluates these abilities jointly and that is fair and thus I wouldn't push on this further, but I would really appreciate it if the authors can somehow give some further clarification on this distinction.

While I would really appreciate it if the authors could address these remaining concerns, I am fine if they don't considering the limited time available. I would neverthless increase my score.

Thank you for reviewing our revision and for letting us know that it addressed most of your concerns! We appreciate the opportunity to clarify the two remaining points.

W1. Separating the effects of tools and multiple turns

We ran GPT-5.6-Sol on Chess and CS2 under four settings. The first three use the same 256 frames. The fourth gives the full video.

SettingChess F1CS2 F1
Single turn, 256 frames, no tools47.317.8
Multiple turns, 256 frames, no tools49.227.1
Multiple turns, 256 frames, with tools69.835.2
Multiple turns, full video, with tools71.565.9

Chess is reasoning-bound. The model's native visual capabilities recover moves reliably, yielding high T1 scores across all four settings. Extra turns alone add 1.9 F1 points. Tool access lets the model write and run scripts to assess board positions, improving T2/T3 and yielding the largest gain (+20.6 F1).

CS2 is perception-bound. Extra turns (+9.3 F1), tools (+8.1), and full video access (+30.7) each improve performance. Extra turns let the model revisit the sampled frames. With tools, it can write and run image recognition algorithms to detect and compare visual cues across frames. Full video access helps it recover fast events that fall between sampled frames. Once the model identifies the winner of each round, tracking the score requires only a running tally.

W2. Distinguishing state reconstruction from strategic reasoning

Thank you for clarifying this distinction. We agree that a model can reconstruct the chessboard correctly and still misjudge positional advantage. However, different world states vary in how difficult they are to track and how much reasoning they require.We view state tracking as a form of spatiotemporal reasoning. The reasoning needed to infer a target state from events over time is part of tracking that state.

We selected widely played games whose rules the evaluated models already know. The difficulty lies in applying those rules to video evidence and reasoning about the resulting state. Figure 4 illustrates two different bottlenecks:

  • CS2 is perception-bound. Key events unfold rapidly, so recovering them requires precise temporal localization. Once the model determines the winner of each round, tracking the score requires only a running tally.
  • Chess is reasoning-bound. Moves are discrete, and the board remains stable between moves, making the event sequence relatively easy to recover. The rules are straightforward, but evaluating positional advantage requires deeper strategic reasoning, even with coarse labels. A model can therefore reconstruct the board correctly and still lose points on T2/T3. We will clarify in the paper that Chess T2/T3 scores reflect the ability to assess and track positional advantage, while T1 measures how well the model recovers the moves.

Thank you again for your constructive feedback. Hope the further experiments and analysis can address your concern in our work!

Thank you for providing these at such short notice. These extra results have clarified everything. I would like to encourage acceptance of the paper.

Reviewer_mMVVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a scalable agentic video benchmark, Video-Arena, for spatiotemporal capability. Video-Arena turns game replays and simulator traces into video tasks requiring event perception, state-update interpretation, and state tracking. It evaluates 180 cases across nine games under native-video and coding-agent protocols, with deterministic labels from game parsers or evaluators. The substantive contribution is a scalable, verifiable construction and evaluation protocol for game video.

Critical strengths
  1. Controlled game environments are a valid and scalable route to video evaluation across diverse games and temporal horizons. The designs of per-game label generators, human audit, corpus-balance checks, and sandbox controls are solid.

  2. The overall evaluation on native models and coding-agent scaffolds is valuable.

  3. The paper is well-motivated and clearly presented.

Critical weaknesses
  1. Positioning and novelty.

This paper overclaims the gap in prior work: temporal event/state tracking (HumanPCR; VideoReasonBench; CATER; CLEVRER) and controllable generated video as evaluation (VET-Bench; MORSE-500) are already central in related benchmarks.

The contribution should be framed more narrowly as using general game infrastructure to construct verifiable video tasks, including automatic labels and anti-leakage protocols.

  1. Interpretability and scalability of the evaluation.

The paper balances trajectory shape, outcome, and duration, but does not sufficiently establish that future cases retain diverse, meaningful, and visually recoverable event sequences. This is central to the scalability claim.

Agentic scores can reflect video-derived rule programs and game expertise, not only visual tracking. Figure 3 (chess example) achieves 76.3 F1 with a fixed visual decoder and search script without reading images; the paper should clarify how it differs from hidden-information leakage and what it implies for claims about visual tracking. The paper should quantify how often it drives success before interpreting high scores as evidence of visual state tracking.

Questions for authors
  1. Beyond trajectory shape, outcome, and duration, how are future trajectories filtered for event diversity and visual recoverability? Please report rejection rates and selection cost.

  2. How often do agentic solutions rely on video decoders and rule-based scripts, as in Figure 3? How much does it contribute to the reported scores?

  3. Can you add a setting with game rules and terminology supplied, or stronger player baselines, to separate domain knowledge from temporal state tracking?

Reviewer_1UjGOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces a benchmark of scalable and verifiable environments from highly competed games, in order to test how well models understand the evolving world. The evaluations can be run under 2 seperate protocols including native visual understanding and tool-assisted agentic analysis. The substantive results reveal that even for best performing system, it merely achieves F1 score of 13.2 under native visual understanding protocol, and models struggle to capture the state updates consistently over time under tool-assisted protocol.

Critical strengths
  • The paper is well-written and clearly express the motivation, design ideology and evaluation process.
  • Video is an important modality and the work try to construct a scalable environment to test how well models understand the ever-changing world.
Critical weaknesses
  • The authors claim the pipeline can generate videos and references at scale, but only 180 tasks across 9 games are generated, which is not sufficient to cover enough long-tail cases. Why does the author not extend more tasks? otherwise its scalability does seem to be sufficiently supported.
Questions for authors

Please address the weaknesses.

4 comments

Dear reviewer 1UjG,

Greetings from the authors!

Thank you for your constructive feedback, which helped us strengthen the paper. We have uploaded the revised manuscript, with changes highlighted in blue:

  • Evaluation scale: clarified the evaluation-cost rationale for the 180-case, nine-game golden set (§3.3).
  • Scalability and coverage: documented seven implemented extension adapters and measured generation times, while clarifying the remaining long-tail coverage limits (Table 2; Appendix E.2; §5).

As the discussion period draws to a close, could you let us know whether any concerns remain? We hope our rebuttal and revisions address your concerns and further strengthen your recognition and confidence in our paper!

Best regards,

The Authors

Reviewer 1UjG,

Please read the authors' rebuttal and post your comments on your raised concerns.

AC

I thank the authors for their response, and I will maintain my positive rating of the work.

Thank you for taking the time to review our rebuttal and revised manuscript. We sincerely appreciate your recognition in our work and suggestion on extending more tasks. We will release more cases from existing and candidate games to support broader community evaluation.

Thank you again for your thoughtful feedback and valuable time!

Comments on the paper

1 comment
AuthorsOpenReview ↗

The Overview of Rebuttal Period

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and all reviewers for their time and constructive discussion. Two reviewers have now confirmed positive evaluations of 3 (Weak Accept):

  • Reviewer 1UjG reviewed our response and maintained their positive rating.
  • Reviewer LZmA confirmed that the rebuttal addressed their concerns and would like to encourage acceptance of the paper.”

Reviewer mMVV has not joined in the discussion, while we hope our rebuttal could address their concerns:

  • Overclaim: we discussed all six suggested benchmarks and clarified our contribution: using game infrastructure to construct scalable and verifiable spatiotemporal tasks with automatically aligned labels.
  • Selection and scalability: we reported candidate-exclusion rates and generation costs, documented seven implemented extension adapters, and clarified human checks for visual recoverability and the limits of coverage.
  • Scripts, leakage, and tool contributions: we explained that scripts process video pixels while source records, ground truth, and external engines are blocked, and reported 92.8% turn-level code-tool usage. Subsequent controls at the same 256-frame coverage under the multi-turn protocol show that adding tools improves Chess/CS2 F1 by 20.6/8.1 points.
  • Rule knowledge versus application: we clarified that knowing game rules does not ensure applying them correctly to video evidence. T1 measures event recovery, while T2/T3 assess the interpretation and tracking of state changes.

We additionally provided repeated evaluations to strengthen the reliability. The uploaded revision marks changes in blue, and the latest follow-up experiments are reported in the discussion.

We are confident that these revisions, driven by the reviewers' valuable feedback, make our paper more robust, rigorous and clear. We are pleased that the reviewers joining in the rebuttals keep a positive recommendation in light of our responses sufficiently addressing their concerns. We promise to incorporate rebuttals into the final manuscript.

Best regards,

Authors of Paper #13422

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.