- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces VSTAT, a benchmark for evaluating visual state tracking in video MLLMs. It contains 834 video clips and 1,500 questions spanning synthetic, recorded, and in-the-wild scenarios. The authors organize tasks by state elements, state structures, and perceptual challenges, and investigate model failures through temporal stretching, comparisons with manually transcribed events, and analysis of generated reasoning traces. I find the evaluation target meaningful and the diagnostic experiments informative. However, the evidence supports a gap between video processing and reasoning over structured observations more clearly than it establishes the specific causes of that gap.
Critical strengths
- It provides a well-motivated evaluation target. Maintaining evolving, query-relevant states is an important aspect of video understanding that is not adequately captured by recognizing isolated actions or describing overall video content. Tasks such as tracking a hidden object, maintaining per-player statistics, and reconstructing event sequences make this distinction concrete.
- Separating state elements from state structures provides a principled way to characterize what information a model must maintain. The additional perceptual categories distinguish difficulties such as occlusion, entity ambiguity, and camera motion.
- The temporal-stretching experiment directly investigates whether brief events are being missed, while the text-transcription experiment tests whether models can perform state updates when accurate event descriptions are provided. The distinction among event recognition, entity association, and state update errors is also informative.
Critical weaknesses
- The text-transcription experiment does not fully isolate the claimed perceptual bottleneck. Providing structured text removes more than visual recognition: it also supplies event boundaries, explicit identities or positions, temporal ordering, and a compact selection of relevant information. Near-perfect performance under this condition demonstrates that the model can track states given these structured observations, but does not identify which part of the original video-processing task caused the failure. Moreover, the experiment covers only three simple synthetic task families, limiting its support for conclusions about the broader benchmark. An additional oracle-visual condition—providing sufficiently dense, clearly localized visual evidence with necessary context—would help distinguish inadequate observation from difficulties in converting visual evidence into usable event representations.
- Temporal stretching reduces the risk of missing events but does not establish that the model receives sufficient dynamic evidence. Extending each event to at least one second does not guarantee that a 1-FPS input resolves its trajectory, direction, or relevant before-and-after states.
- A model answering from a predetermined frame sequence may behave differently from a system that can locate uncertain intervals and request additional frames. This distinction is particularly relevant because human participants are explicitly allowed to watch videos multiple times, whereas an equivalent model-side procedure is not documented. Appendix D mentions agentic frameworks, but the corresponding framework configurations, video-access tools, and results are not clearly presented.
- The open-weight evaluation primarily covers models up to approximately 9B parameters, while the proprietary evaluation and detailed diagnosis center on Gemini. Additional model families and larger open-weight models would strengthen claims about current MLLMs more broadly.
Questions for authors
Please refer to the weakness part.
3 comments
Thank you for the detailed response and additional experiments.However, I was not convinced by the rebuttal. For instance, the stretched-video experiment does not fully isolate the perceptual bottleneck. more frames does not indicate that all necessary event transitions are adequately represented. I would like to keep my score.
Thank you for response. We want to clarify that a key detail of the stretched-video experiment that may not have been clear in our response. In each 5-second analysis video, the single event spans the entire 5 seconds (e.g., one cup swap or one die roll from start to end). Therefore, after stretching, all 25 sampled frames clearly depict the transition itself, covering the event densely from its start to its end at an effective rate of 5 FPS, rather than only a few snapshots of it. Even under this setting, Gemini-3.1 Pro achieves only 53.6%, far below the near-perfect accuracy with text transcriptions. Moreover, on the full benchmark, we remark that doubling the sampling rate of native video input from 4 FPS to 8 FPS does not improve performance (44.4 vs. 42.3). So we think this experiment clearly addresses your concern. We have clarified this detail in Section 3.3 and Appendix C.2 of the revised manuscript.
We deeply appreciate your insightful comments and efforts in reviewing our manuscript. We respond to each of your comments one-by-one in what follows. In the revised draft, we mark our major revisions as “blue”.
[W1] The text-transcription experiment does not fully isolate the perceptual bottleneck.
Thank you for the insightful comment. We agree that the structured text removes more than visual recognition, as it also supplies event boundaries, explicit identities, temporal ordering, and a compact selection of relevant information. We remark that our temporally stretched 5-second setting in Section 3.3 already closely approximates the oracle-visual condition you suggested:
Even under this setting, Gemini-3.1 Pro achieves only 53.6%, far below the near-perfect accuracy with text transcriptions (Table 4). This suggests that the failure does not mainly stem from inadequate observation, but from difficulties in converting clearly visible and localized visual evidence into usable event representations. This is also consistent with our error analysis in Section 3.4, which covers 70 questions from every video category and shows that more than 50% of failures stem from event recognition. We clarified the number of events in each video, the number of sampled frames, and this interpretation in Section 3.3 and Appendix C.2 in the revised manuscript.
[W2] Temporal stretching does not guarantee sufficient dynamic evidence.
This is a good point. In our stretching experiment, each 5-second video contains a single event observed through 25 sampled frames, so the model sees the event’s trajectory as well as the static states before and after it. We revised the expression “fully visible to the model” to state these details explicitly. To further validate this, we evaluate Gemini-3.1 Pro on the full benchmark with native video input at different sampling rates. Increasing the sampling rate does not improve performance (44.4, 44.4, and 42.3 at 1 (Table 2), 4, and 8 FPS, respectively), suggesting that the gap does not mainly stem from insufficient dynamic evidence.
[W3] Fixed frame sequences vs. systems that can request additional frames; agentic framework details.
Thank you for pointing this out. As you mentioned, human participants could watch each video multiple times, while models answer from predetermined frame sequences. Thus, human performance should be interpreted as an upper reference of solvability rather than a strictly matched comparison; we clarified this in Appendix B in the revised manuscript. To examine models that can actively revisit the video, we evaluate an agentic framework using Claude Opus 4.7 (Max) as the backbone. On 200 QA pairs sampled across all task categories, the agentic framework achieves only 37.4%, below the Chance-Level (Frequency) baseline of 57.4% on the same subset. This indicates that the ability to revisit intervals and request additional frames does not resolve the difficulty of VSTAT. We added the framework configuration, video-access tools, and results in Appendix C.7 in the revised manuscript.
[W4] Additional model families and larger open-weight models.
Following your suggestion, we additionally evaluate larger open-weight models from three families, using ordered frames sampled at 1 FPS. Although larger models generally perform better, even the best model (Qwen3-VL-235B-A22B, 39.4) remains close to the Chance-Level (Frequency) baseline (37.8) and far below Gemini-3.1 Pro (44.4) and humans (90.5). We added these results in Table 2 in the revised manuscript.
\begin{array}{lcc} \hline \text{Model} & \text{Params} & \text{Avg.} \newline \hline \text{Chance Level (Frequency)} & - & 37.8 \newline \text{Human Performance} & - & 90.5 \newline \hline \text{Qwen3-VL-8B} & 8\text{B} & 32.7 \newline \text{Qwen3-VL-32B} & 32\text{B} & 35.2 \newline \text{Qwen3-VL-235B-A22B} & 235\text{B} & \textbf{39.4} \newline \text{InternVL3.5-38B} & 38\text{B} & 33.8 \newline \text{InternVL3.5-241B-A28B} & 241\text{B} & 35.4 \newline \text{GLM-4.6V} & 106\text{B} & 37.5 \newline \hline \text{Gemini-3.1 Pro} & - & 44.4 \newline \hline \end{array}