- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces a benchmark to evaluate how well video understanding models perceive events, interpret their effects and also track evolving world states over time. It uses game records and simulators to generate videos and automatically derive GT annotations. This reduces need and effort for manual labeling. The models are evaluated under 2 protocols. One of them is the native visual understanding using sampled video frames and the other is agentic evaluation with access to video-processing tools, code execution and also multi-turn analysis.
Critical strengths
- It addresses an important gap in the video understanding benchmark space.
- The paper evaluates event perception, state interpretation and temporal tracking together.
- The automated data generation pipeline is quite good and enables scalable benchmark construction that has deterministic GT labels. This reduces reliance on manual annotations which are expensive and time consuming and also LLM judges which cannot entirely be trusted.
- The benchmark has diverse environments and covers 9 games across 6 genres.
- The experiments are detailed and the tier-wise analysis is also quite good. The additional validation of GT labels and temporal sampling also strengthens the paper.
Critical weaknesses
- The naive and agentic protocols differ substantially in visual coverage, tool access and also interaction budgets. This makes it a bit difficult to isolate benefits of agentic reasoning.
- Some GT labels rely on game-engine evals rather than the directly observable states. This can make it difficult to distinguish world-state tracking from game-specific strategic reasoning in my opinion.
- The F1 metric is strict and requires all event and state fields to match. Now this can maybe obscure whether failures arise from event perception. state interpretation or temporal tracking.
- The eval also uses only 20 cases per game and one effective run per model-case pair. This can limit reliability estimates.
Questions for authors
- Can the authors please provide a more controlled comparison? They can use equal visual input budgets to isolate benefits of tool use and multi-turn reasoning.
- How do you distinguish failures in world state tracking from difficulties in approximating game engines that are used to generate T2 and T3 labels?
- Please also clarify how much of the low native F1 is caused by limited frame sampling and strict tuple matching rather than actual video-understanding failures?
- Can you also evaluate multiple runs per model-case pair to motivate the reliability of the results and especially under agentic protocol?
4 comments
Thank you for the response. While most of the concerns are resolved, there are still a few remaining concerns:
W1. I appreciate you isolating the effect of tools and multi-turn reasoning together, but what I meant in my question was is there a way you can isolate the effects of the two separately: How much does the tool use contribute and how much the multi-turn?
W2. I appreciate the clarification that exact engine win probabilities are difficult and unnecessary and I agree with that and also that coarse labels make the task more practical. However, I still feel that it is difficult to distinguish failures in world-state tracking from failures in game-specific strategic reasoning. A model might correctly track the game state but still fail T2/T3 because it cannot assess the position as well as the game engine. I understand that the benchmark evaluates these abilities jointly and that is fair and thus I wouldn't push on this further, but I would really appreciate it if the authors can somehow give some further clarification on this distinction.
While I would really appreciate it if the authors could address these remaining concerns, I am fine if they don't considering the limited time available. I would neverthless increase my score.
Thank you for reviewing our revision and for letting us know that it addressed most of your concerns! We appreciate the opportunity to clarify the two remaining points.
W1. Separating the effects of tools and multiple turns
We ran GPT-5.6-Sol on Chess and CS2 under four settings. The first three use the same 256 frames. The fourth gives the full video.
| Setting | Chess F1 | CS2 F1 |
|---|---|---|
| Single turn, 256 frames, no tools | 47.3 | 17.8 |
| Multiple turns, 256 frames, no tools | 49.2 | 27.1 |
| Multiple turns, 256 frames, with tools | 69.8 | 35.2 |
| Multiple turns, full video, with tools | 71.5 | 65.9 |
Chess is reasoning-bound. The model's native visual capabilities recover moves reliably, yielding high T1 scores across all four settings. Extra turns alone add 1.9 F1 points. Tool access lets the model write and run scripts to assess board positions, improving T2/T3 and yielding the largest gain (+20.6 F1).
CS2 is perception-bound. Extra turns (+9.3 F1), tools (+8.1), and full video access (+30.7) each improve performance. Extra turns let the model revisit the sampled frames. With tools, it can write and run image recognition algorithms to detect and compare visual cues across frames. Full video access helps it recover fast events that fall between sampled frames. Once the model identifies the winner of each round, tracking the score requires only a running tally.
W2. Distinguishing state reconstruction from strategic reasoning
Thank you for clarifying this distinction. We agree that a model can reconstruct the chessboard correctly and still misjudge positional advantage. However, different world states vary in how difficult they are to track and how much reasoning they require.We view state tracking as a form of spatiotemporal reasoning. The reasoning needed to infer a target state from events over time is part of tracking that state.
We selected widely played games whose rules the evaluated models already know. The difficulty lies in applying those rules to video evidence and reasoning about the resulting state. Figure 4 illustrates two different bottlenecks:
- CS2 is perception-bound. Key events unfold rapidly, so recovering them requires precise temporal localization. Once the model determines the winner of each round, tracking the score requires only a running tally.
- Chess is reasoning-bound. Moves are discrete, and the board remains stable between moves, making the event sequence relatively easy to recover. The rules are straightforward, but evaluating positional advantage requires deeper strategic reasoning, even with coarse labels. A model can therefore reconstruct the board correctly and still lose points on T2/T3. We will clarify in the paper that Chess T2/T3 scores reflect the ability to assess and track positional advantage, while T1 measures how well the model recovers the moves.
Thank you again for your constructive feedback. Hope the further experiments and analysis can address your concern in our work!
Thank you for providing these at such short notice. These extra results have clarified everything. I would like to encourage acceptance of the paper.
Dear reviewer LZmA,
Greetings from the authors!
Thank you for your constructive feedback, which helped us strengthen the paper. We have uploaded the revised manuscript, with changes highlighted in blue:
As the discussion period draws to a close, could you let us know whether any concerns remain? We hope our rebuttal and revisions address your concerns and further strengthen your recognition and confidence in our paper!
Best regards,
The Authors