acceptodds
Under review as a conference paper at ICLR 2027

Benchmarking Visual State Tracking in Multimodal Video Understanding

Abstract

Understanding a video requires more than recognizing isolated moments, as humans continuously track entities, states, and events over time. This capacity for visual state tracking is fundamental to video understanding, yet remains underexplored in current evaluations of Multimodal Large Language Models (MLLMs). We introduce Visual STAte Tracking benchmark (VSTAT), a video-based benchmark designed to diagnose visual state tracking in MLLMs. VSTAT consists of 834 clips drawn from both synthetic and real-world videos, paired with 1,500 questions that cannot be answered from any single frame or short segment, requiring continuous perception and integration of events across the entire video stream. Despite their strong performance on existing video benchmarks, we find that state-of-the-art MLLMs perform far below humans and only modestly above answer-prior baselines. To analyze this gap, we compare MLLMs' thinking traces with the underlying video stream to understand why and when MLLMs fail on VSTAT. Through extensive analyzes, we find that MLLMs reason and track correctly in text, but still fail at visually perceiving the events they need to track.

Primary area:
datasets and benchmarks
Keywords:
video understanding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_HopZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VSTAT, a benchmark for evaluating visual state tracking in video MLLMs. It contains 834 video clips and 1,500 questions spanning synthetic, recorded, and in-the-wild scenarios. The authors organize tasks by state elements, state structures, and perceptual challenges, and investigate model failures through temporal stretching, comparisons with manually transcribed events, and analysis of generated reasoning traces. I find the evaluation target meaningful and the diagnostic experiments informative. However, the evidence supports a gap between video processing and reasoning over structured observations more clearly than it establishes the specific causes of that gap.

Critical strengths
  1. It provides a well-motivated evaluation target. Maintaining evolving, query-relevant states is an important aspect of video understanding that is not adequately captured by recognizing isolated actions or describing overall video content. Tasks such as tracking a hidden object, maintaining per-player statistics, and reconstructing event sequences make this distinction concrete.
  2. Separating state elements from state structures provides a principled way to characterize what information a model must maintain. The additional perceptual categories distinguish difficulties such as occlusion, entity ambiguity, and camera motion.
  3. The temporal-stretching experiment directly investigates whether brief events are being missed, while the text-transcription experiment tests whether models can perform state updates when accurate event descriptions are provided. The distinction among event recognition, entity association, and state update errors is also informative.
Critical weaknesses
  1. The text-transcription experiment does not fully isolate the claimed perceptual bottleneck. Providing structured text removes more than visual recognition: it also supplies event boundaries, explicit identities or positions, temporal ordering, and a compact selection of relevant information. Near-perfect performance under this condition demonstrates that the model can track states given these structured observations, but does not identify which part of the original video-processing task caused the failure. Moreover, the experiment covers only three simple synthetic task families, limiting its support for conclusions about the broader benchmark. An additional oracle-visual condition—providing sufficiently dense, clearly localized visual evidence with necessary context—would help distinguish inadequate observation from difficulties in converting visual evidence into usable event representations.
  2. Temporal stretching reduces the risk of missing events but does not establish that the model receives sufficient dynamic evidence. Extending each event to at least one second does not guarantee that a 1-FPS input resolves its trajectory, direction, or relevant before-and-after states.
  3. A model answering from a predetermined frame sequence may behave differently from a system that can locate uncertain intervals and request additional frames. This distinction is particularly relevant because human participants are explicitly allowed to watch videos multiple times, whereas an equivalent model-side procedure is not documented. Appendix D mentions agentic frameworks, but the corresponding framework configurations, video-access tools, and results are not clearly presented.
  4. The open-weight evaluation primarily covers models up to approximately 9B parameters, while the proprietary evaluation and detailed diagnosis center on Gemini. Additional model families and larger open-weight models would strengthen claims about current MLLMs more broadly.
Questions for authors

Please refer to the weakness part.

3 comments

We deeply appreciate your insightful comments and efforts in reviewing our manuscript. We respond to each of your comments one-by-one in what follows. In the revised draft, we mark our major revisions as “blue”.


[W1] The text-transcription experiment does not fully isolate the perceptual bottleneck.

Thank you for the insightful comment. We agree that the structured text removes more than visual recognition, as it also supplies event boundaries, explicit identities, temporal ordering, and a compact selection of relevant information. We remark that our temporally stretched 5-second setting in Section 3.3 already closely approximates the oracle-visual condition you suggested:

  • Clearly localized: each 5-second video contains only a single event, so the model does not need to localize the relevant event within a long video stream.
  • Sufficiently dense: after 5× temporal stretching, sampling at 1 FPS is equivalent to sampling the original video at 5 FPS, so each single-event clip is observed through 25 sampled frames, capturing the trajectory and direction of the event.
  • Necessary context: the initial state (e.g., the initial position of the ball in the shell game) is always visible at the beginning of each video.

Even under this setting, Gemini-3.1 Pro achieves only 53.6%, far below the near-perfect accuracy with text transcriptions (Table 4). This suggests that the failure does not mainly stem from inadequate observation, but from difficulties in converting clearly visible and localized visual evidence into usable event representations. This is also consistent with our error analysis in Section 3.4, which covers 70 questions from every video category and shows that more than 50% of failures stem from event recognition. We clarified the number of events in each video, the number of sampled frames, and this interpretation in Section 3.3 and Appendix C.2 in the revised manuscript.


[W2] Temporal stretching does not guarantee sufficient dynamic evidence.

This is a good point. In our stretching experiment, each 5-second video contains a single event observed through 25 sampled frames, so the model sees the event’s trajectory as well as the static states before and after it. We revised the expression “fully visible to the model” to state these details explicitly. To further validate this, we evaluate Gemini-3.1 Pro on the full benchmark with native video input at different sampling rates. Increasing the sampling rate does not improve performance (44.4, 44.4, and 42.3 at 1 (Table 2), 4, and 8 FPS, respectively), suggesting that the gap does not mainly stem from insufficient dynamic evidence.


[W3] Fixed frame sequences vs. systems that can request additional frames; agentic framework details.

Thank you for pointing this out. As you mentioned, human participants could watch each video multiple times, while models answer from predetermined frame sequences. Thus, human performance should be interpreted as an upper reference of solvability rather than a strictly matched comparison; we clarified this in Appendix B in the revised manuscript. To examine models that can actively revisit the video, we evaluate an agentic framework using Claude Opus 4.7 (Max) as the backbone. On 200 QA pairs sampled across all task categories, the agentic framework achieves only 37.4%, below the Chance-Level (Frequency) baseline of 57.4% on the same subset. This indicates that the ability to revisit intervals and request additional frames does not resolve the difficulty of VSTAT. We added the framework configuration, video-access tools, and results in Appendix C.7 in the revised manuscript.


[W4] Additional model families and larger open-weight models.

Following your suggestion, we additionally evaluate larger open-weight models from three families, using ordered frames sampled at 1 FPS. Although larger models generally perform better, even the best model (Qwen3-VL-235B-A22B, 39.4) remains close to the Chance-Level (Frequency) baseline (37.8) and far below Gemini-3.1 Pro (44.4) and humans (90.5). We added these results in Table 2 in the revised manuscript.

\begin{array}{lcc} \hline \text{Model} & \text{Params} & \text{Avg.} \newline \hline \text{Chance Level (Frequency)} & - & 37.8 \newline \text{Human Performance} & - & 90.5 \newline \hline \text{Qwen3-VL-8B} & 8\text{B} & 32.7 \newline \text{Qwen3-VL-32B} & 32\text{B} & 35.2 \newline \text{Qwen3-VL-235B-A22B} & 235\text{B} & \textbf{39.4} \newline \text{InternVL3.5-38B} & 38\text{B} & 33.8 \newline \text{InternVL3.5-241B-A28B} & 241\text{B} & 35.4 \newline \text{GLM-4.6V} & 106\text{B} & 37.5 \newline \hline \text{Gemini-3.1 Pro} & - & 44.4 \newline \hline \end{array}

Thank you for the detailed response and additional experiments.However, I was not convinced by the rebuttal. For instance, the stretched-video experiment does not fully isolate the perceptual bottleneck. more frames does not indicate that all necessary event transitions are adequately represented. I would like to keep my score.

Thank you for response. We want to clarify that a key detail of the stretched-video experiment that may not have been clear in our response. In each 5-second analysis video, the single event spans the entire 5 seconds (e.g., one cup swap or one die roll from start to end). Therefore, after stretching, all 25 sampled frames clearly depict the transition itself, covering the event densely from its start to its end at an effective rate of 5 FPS, rather than only a few snapshots of it. Even under this setting, Gemini-3.1 Pro achieves only 53.6%, far below the near-perfect accuracy with text transcriptions. Moreover, on the full benchmark, we remark that doubling the sampling rate of native video input from 4 FPS to 8 FPS does not improve performance (44.4 vs. 42.3). So we think this experiment clearly addresses your concern. We have clarified this detail in Section 3.3 and Appendix C.2 of the revised manuscript.

Reviewer_PLPmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes VSTAT, a benchmark for visual state tracking on video MLLMs that can be only answered by watching full video. It organizes questions by state element and structure, and contains 1,500 questions on 834 synthetic, self-recorded, and in-the-wild video clips. Evaluation on proprietary models and open source video MLLMs show that the best model scores 44.4, compared with 90.5 for humans. Their controlled synthetic experiments and failure analysis of model-generated reasoning traces suggest that recognizing the visual events and associating entities across time are major difficulties.

Critical strengths

The benchmark formulates each question by the state element and the structure it requires. Its visualed examples illustrates how different question on the same video can require different visual state representations. Quality check on the datset also makes the intended evaluation understandable. This formulation can be possibly used as a basis for further control analysis.

Critical weaknesses
  1. Contribution relative to VCBench[1] is unclear: VCBench evaluates counting and identity maintenace across multiple timestamp queries and is highly related work, but its not listed in Table 1. VSTAT covers more elements and structures, but the reported results on Table 2 do not make clear which additional model failures this coverage shows. This limits the evidence for VSTAT's distinct diagnostic value.

  2. Temporal requirement analysis: The paper aims to build a question that are not answerable from a few frames, but it doesn't report full video vs. last frame or sparse frame controls by task family, which are broadly used in temporal necessity of video understading. [2,3]


References:

[1] Pengyiang Liu et al. “VCBench: A Streaming Counting Benchmark for Spatial-Temporal State Maintenance in Long Videos.” ECCV 2026.

[2] Ziyao Shangguan et al. “TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models.” ICLR 2025.

[3] Yuanxin Liu et al. “VideoReasonBench: Can MLLMs Perform Vision-Centric Complex Video Reasoning?” arXiv:2505.23359.

Questions for authors
  • The scores in Table 2 exactly match the three source-specific scores in Talbe 9 for Gemini models. Can you provide the corrected values?

Minor copyedit:

Section 3 (p. 5, lines 255–260) introduces proprietary models with “(a)” but gives no “(b)” before the open-source models; please add “(b)” or remove “(a).”

3 comments

We deeply appreciate your insightful comments and efforts in reviewing our manuscript. We respond to each of your comments one-by-one in what follows. In the revised draft, we mark our major revisions as “blue”.


[W1] Contribution relative to VCBench.

Thank you for introducing this relevant work. VCBench (now SVCBench in the latest version) evaluates counting and identity maintenance in a streaming setting, which is closely related to our motivation. VSTAT complements it in two aspects. First, beyond counts, VSTAT covers locations and attributes as state elements, as well as sequence, set, and dictionary structures. Second, VSTAT spans diverse synthetic, self-recorded, and in-the-wild videos with six types of perceptual challenges.

This broader coverage reveals failures that a count-only evaluation would not capture. For instance, Gemini-3.1 Pro (low) performs worst on location (38.5) and dictionary states (38.7), while it performs considerably better on sequence states (60.8). Model rankings also change across element types: LLaVA-OV-2-8B achieves the best location accuracy among open-sourced models (43.0) but only 28.3 on count, while Cambrian-S-7B shows the opposite trend (33.2 on count, 33.6 on location). We added SVCBench to Table 1 and expanded the related work discussion in the revised manuscript, as shown in the table below.

\begin{array}{lcccccc} \hline \text{Benchmark} & \text{Source} & \text{Num Clips} & \text{Num QAs} & \text{State Tracking} & \text{Real-world} & \text{Diverse} \newline \hline \text{VideoMME-v2} & \text{C, R} & 800 & 3{,}200 & \triangle & \checkmark & \checkmark \newline \text{Perception Test} & \text{C, R} & 11{,}620 & 38{,}000 & \triangle & \checkmark & \checkmark \newline \text{CATER} & \text{Y} & 5{,}500 & 5{,}500 & \checkmark & \times & \times \newline \text{CLEVRER} & \text{Y} & 20{,}000 & >300{,}000 & \times & \times & \times \newline \text{VSI-SUPER (Cambrian-S)} & \text{R} & 500 & 500 & \checkmark & \checkmark & \times \newline \text{SVCBench} & \text{R, Y} & 406 & 1{,}000 & \checkmark & \checkmark & \times \newline \text{CP-Bench} & \text{Y} & 101 & 101 & \checkmark & \times & \times \newline \text{VET-Bench} & \text{Y} & 100 & 100 & \checkmark & \times & \times \newline \hline \textbf{VSTAT (ours)} & \text{C, R, Y} & 834 & 1{,}500 & \checkmark & \checkmark & \checkmark \newline \hline \end{array}


[W2] Temporal requirement analysis: full video vs. last frame or sparse frame controls by task family.

Thanks for the suggestion! Following your suggestion, we evaluate models with partial observations of each video and compare them with the full video, as shown in the table below (open-sourced models use ordered frames sampled at 1 FPS). Gemini-3.1 Pro, the only model clearly above the Chance-Level (Frequency) baseline with the full video, drops by 8–13 points under all partial-observation conditions, falling below the baseline. Without video, all models also score below the baseline, indicating that answer-option priors alone do not solve the benchmark. Open-sourced models remain below the baseline even with the full video, consistent with our observation in Section 3.2. We added these results with the per-task breakdown in Appendix C.5 in the revised manuscript.

\begin{array}{lccc} \hline \text{Input} & \text{Gemini-3.1 Pro} & \text{Qwen3VL-8B} & \text{LLaVA-OV-2-8B} \newline \hline \text{Chance Level (Frequency)} & 37.8 & 37.8 & 37.8 \newline \hline \text{No video} & 29.9 & 28.4 & 34.3 \newline \text{Last frame} & 31.7 & 23.0 & 25.4 \newline \text{First + last frames} & 31.7 & 24.9 & 26.2 \newline \text{8 uniform frames} & 36.7 & 29.7 & 30.3 \newline \text{Final quarter (1 FPS)} & 35.0 & 27.1 & 28.6 \newline \hline \textbf{Full video} & \textbf{44.4} & \textbf{32.7} & \textbf{32.4} \newline \hline \end{array}


[Q1] Gemini scores in Table 9 match those in Table 2.

Thank you for your careful reading. This was a copy error when constructing Table 9: the state-element scores of the Gemini models from Table 2 were mistakenly entered. The corrected source-specific scores are reported in the table below, and we fixed Table 9 in the revised manuscript. Note that the overall averages and all other results remain unchanged.

\begin{array}{lcccc} \hline \text{Model} & \text{Avg.} & \text{YouTube} & \text{Synthetic} & \text{Recorded} \newline \hline \text{Gemini-3.1 Pro (low)} & 44.4 & \text{46.2} & \text{42.4} & \text{41.8} \newline \text{Gemini-3.1 Pro (high)} & 43.9 & \text{44.7} & \text{42.1} & \text{45.9} \newline \text{Gemini-3.0 Flash (low)} & 39.8 & \text{45.2} & \text{32.0} & \text{39.7} \newline \text{Gemini-3.0 Flash (high)} & 38.8 & \text{44.2} & \text{31.3} & \text{35.5} \newline \hline \end{array}


[Q2] Editorial comment.

Thanks for the suggestion! We added “(b)” before the open-sourced models in Section 3 (L255-260) in the revised manuscript.

Thank you for the detailed response and the additional experiments. Corrected Table 9 and per-task partial observatino results in Appendix C.5 address my question and my tmeporal requirement concern.

My remaining concern is about the interpretation of category-level results. Since the frequency baseliens differ across categories, raw score differences alone are hard to interpret on their own. I recommend authors to discuss and explain what the location and attribute categories reveal about model failures beyond counting.

Minor edit: SVCBench also includes simulated videos so W1 table should be updated. Considering the responses, I will raise my rating from 2 to 3.

Thanks for your constructive feedback!

1. Interpretation of category-level results

As you mentioned, the frequency baseline results are different across categories, so we compare each model’s score against the baseline of each category. For count category, all models stay close to or below the baseline values. We observed that most failures here come from missing or hallucinating events, as in Figure 12. For location, the gain over the baseline is also small. This is mainly because of difficulty to track identical-looking objects move under occlusion or camera motion (e.g. shell game). This shows that the model loses track of which object moved where, which matches the entity association failures in Figures 15 and 17. Attribute shows the largest improvement compared with the baseline result, since many attributes (e.g., latte art or doll appearance) can be recognized once the relevant moment is seen. However, we remark that the scores are still far below humans. We added this discussion to Appendix C.1.


2. Minor edit.

Thank you! We updated the source of SVCBench to R, Y in Table 1 and the table in [W1].

Reviewer_GHskOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VSTAT, a benchmark for evaluating visual state tracking in multimodal video understanding. Unlike existing benchmarks that can often be solved using a few keyframes, VSTAT focuses on tasks that require tracking entities, events, and changing states throughout a video. The benchmark includes both synthetic and real-world videos, with questions covering different types of state tracking. The authors evaluate several recent MLLMs and find that their performance is considerably below human performance. Through additional experiments using text transcriptions, temporally stretched videos, and error analysis, they suggest that visual perception and event recognition are major bottlenecks for current models.

Critical strengths
  1. A relevant and useful benchmark for an underexplored aspect of video understanding.

I think the paper addresses an important limitation of existing video understanding benchmarks. Tracking how objects and states change over time is different from recognizing individual actions or understanding a few selected frames, and this distinction is not always captured by current evaluations. I particularly like that VSTAT covers different types of tracking tasks rather than focusing on a single synthetic setting. The combination of real-world and synthetic videos, along with the taxonomy of state complexity and perceptual challenges, makes the benchmark potentially useful for future work on temporal video understanding.

  1. Interesting diagnostic experiments beyond standard benchmark evaluation.

Another strength is that the authors do not stop at reporting model accuracy. The experiments comparing video inputs with text transcriptions are particularly interesting, as they help investigate whether models fail because they cannot recognize the relevant visual events or because they cannot reason about them. The analysis of event recognition, entity association, and state update errors also provides useful insights into where current MLLMs struggle. Although I have some concerns about how broadly these findings can be generalized, I find this analysis a valuable part of the paper.

Critical weaknesses
  1. The claim that VSTAT requires continuous state tracking needs stronger validation.

A central claim of the paper is that the questions cannot be answered from a few keyframes or short video segments. However, this property is mainly supported by the dataset construction procedure rather than by a systematic experimental evaluation. The authors mention filtering questions to avoid shortcuts, which is useful, but I would like to see a quantitative comparison using only the first and last frames, sparsely sampled frames, or selected short segments.

This is important because some questions may still be answerable from partial observations or visible end states, even if they were designed to require continuous tracking. Without such an analysis, it is difficult to determine how much the benchmark specifically measures state tracking rather than general visual recognition or counting ability.

  1. The conclusion that perception is the main bottleneck seems somewhat stronger than the experimental evidence.

The comparison between video and text inputs is interesting, but it is conducted on a small set of relatively simple synthetic tasks. The text descriptions explicitly provide the relevant states and events, which removes much of the difficulty of interpreting the original visual stream. Therefore, the large performance gap does not necessarily isolate perception from the other challenges involved in video-based reasoning.

Similarly, the temporal stretching experiment is limited to a specific setting, and the analysis based on model-generated thinking traces may not always accurately reflect the model's actual internal reasoning. I think the results provide good evidence that visual perception is an important bottleneck, but they are not yet sufficient to conclude that reasoning plays only a minor role across the broader benchmark.

Questions for authors
  1. Clarification on the necessity of full-video tracking.

Have the authors evaluated models using only the first and last frames, sparse keyframes, or short video segments? I would be interested in seeing how performance changes compared with using the full video. This would help establish whether VSTAT consistently measures continuous state tracking across its different task categories.

  1. Clarification on the perception versus reasoning analysis.

How many examples were evaluated in the text-transcription experiments, and how representative are they of the full benchmark? Since the text descriptions already contain the relevant events and states, could the authors clarify to what extent the observed performance gap can be attributed specifically to visual perception rather than the additional complexity of reasoning directly over video inputs?

  1. Clarification on human evaluation and error analysis.

Appendix B mentions that human participants could watch the videos multiple times, while the evaluated models operate with limited frame budgets. How many participants were involved, and what proportion of the benchmark did they evaluate? Also, for the manually classified failure modes, was there any independent verification or measurement of agreement between annotators? These details would help assess the reliability of the reported human gap and error breakdown.

  1. Clarification on dataset redistribution.

The benchmark includes videos collected from YouTube, and the authors mention plans to release the video clips. Could the authors clarify whether these clips will be redistributed directly or provided through links, and how the relevant licensing and permissions will be handled?

3 comments

We deeply appreciate your insightful comments and efforts in reviewing our manuscript. We respond to each of your comments one-by-one in what follows. In the revised draft, we mark our major revisions as “blue”.


[W1/Q1] Necessity of full-video tracking: comparison with first/last frames, sparse keyframes, or short segments.

Thanks for the suggestion! Following your suggestion, we evaluate models with partial observations of each video and compare them with the full video, as shown in the table below. Gemini-3.1 Pro, the only model clearly above the Chance-Level (Frequency) baseline with the full video, drops by 8-13 points under all partial-observation conditions, falling below the baseline. Without video, all models also score below the baseline, indicating that answer-option priors alone do not solve the benchmark. Open-sourced models remain below the baseline even with the full video, consistent with our observation in Section 3.2. We added these results with the per-task breakdown in Appendix C.5 in the revised manuscript.

\begin{array}{lccc} \hline \text{Input} & \text{Gemini-3.1 Pro} & \text{Qwen3VL-8B} & \text{LLaVA-OV-2-8B} \newline \hline \text{Chance Level (Frequency)} & 37.8 & 37.8 & 37.8 \newline \hline \text{No video} & 29.9 & 28.4 & 34.3 \newline \text{Last frame} & 31.7 & 23.0 & 25.4 \newline \text{First + last frames} & 31.7 & 24.9 & 26.2 \newline \text{8 uniform frames} & 36.7 & 29.7 & 30.3 \newline \text{Final quarter (1 FPS)} & 35.0 & 27.1 & 28.6 \newline \hline \textbf{Full video} & \textbf{44.4} & \textbf{32.7} & \textbf{32.4} \newline \hline \end{array}


[W2/Q2] The perception-bottleneck conclusion seems stronger than the evidence; size and representativeness of the text-transcription experiment.

This is a good point. The text-transcription experiment uses three Blender tasks (rolling die, shell game, and tilt box) with 150 questions per video duration (5, 10, and 20 seconds), 450 in total, since their events can be exactly transcribed from the scene programs. We agree that the text also provides event boundaries, explicit identities, and a compact selection of relevant information.

To separate perception from reasoning complexity, we note that in our stretched 5-second setting (Table 4), each video contains only a single event observed through 25 sampled frames, so the model neither needs to localize the event nor integrate multiple state updates. Even so, Gemini-3.1 Pro achieves only 53.6%, far below the near-perfect text accuracy. Our error analysis over 70 questions from every video category (Section 3.4) is also consistent, with more than 50% of failures stemming from event recognition. Nevertheless, we agree that this does not establish that reasoning plays only a minor role, and we soften our claim to “visual perception is one of the major bottlenecks” in the revised manuscript.


[Q3] Human evaluation and reliability of the error analysis.

6 participants were involved in the human evaluation, and they evaluated the entire benchmark. Participants were excluded from evaluating any videos or questions they had contributed, and all responses were collected without access to the ground-truth answers (Appendix B). As you pointed out, participants were allowed to watch each video multiple times, unlike models operating with fixed frame budgets. Thus, human performance should be interpreted as an upper reference of solvability rather than a strictly matched comparison. For the error analysis, each failure case was classified by one author and cross-checked by another author who did not perform the initial classification. We clarified this in Appendix B and Appendix C.3 in the revised manuscript.


[Q4, Ethics] Dataset redistribution of YouTube-derived clips.

Thank you for raising this important point. We will not redistribute YouTube-derived clips directly. Instead, we release the video IDs, timestamps, and a download-and-preprocessing script that reproduces the exact clips used in our benchmark, together with the questions, answers, and taxonomy labels. Synthetic Blender videos and self-recorded videos, for which we obtained explicit consent for research use and public release from all identifiable persons, will be released directly. We clarified this release policy in Appendix A.3 in the revised manuscript.

Thanks for the detailed response and additional experiments. The partial-video comparisons are helpful, and the clarifications on the perception analysis and dataset release address most of my concerns.

I still have some reservations about whether all tasks strictly require continuous tracking, especially given the cases discussed in Appendix C.5. Overall, the rebuttal has improved my assessment of the paper, but I'll keep my score at 3.

Thanks for your feedback! We are glad that our response addressed most of your concerns. We also agree that a few tasks in VSTAT may not strictly require continuous tracking, but we still believe most of our tasks are closely related to continuous state tracking. Nevertheless, to make VSTAT more rigorous, we will revise those few tasks and corresponding questions (e.g., redesigning the questions or distractors so that they cannot be answered from the end state or the options alone) in the final version.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Common Response

Dear reviewers and AC,

We sincerely appreciate your valuable time and effort spent reviewing our manuscript.

We propose VSTAT, a benchmark for diagnosing the visual state tracking capability of MLLMs, where answering each question requires continuously perceiving and integrating events throughout the video stream. As reviewers highlighted, our benchmark addresses an important and underexplored aspect of video understanding (GHsk, HopZ), provides a principled taxonomy of state elements and structures (PLPm, HopZ), and offers informative diagnostic experiments beyond standard benchmark evaluation (GHsk, HopZ).

We appreciate your constructive feedback on our manuscript. In response to the comments, we have carefully revised and enhanced the manuscript as follows:

  • Temporal necessity analysis with partial observations, including per-task results (Appendix C.5)
  • Evaluation of larger open-weight models, including Qwen3-VL-235B-A22B, InternVL3.5-241B-A28B, and GLM-4.6V (Table 2, Appendix C.6)
  • Evaluation of an agentic framework that can actively revisit the video (Appendix C.7)
  • Evaluation with different video sampling rates and clarification of the temporal-stretching setting (Section 3.3, Appendix C.2)
  • Revision of the perception-bottleneck claim to “one of the major bottlenecks” (Abstract, Section 1, Section 3.3, Conclusion)
  • Details of human evaluation and the error analysis protocol (Appendix B, Appendix C.3)
  • Data release policy for YouTube-derived clips (Appendix A.3)
  • Comparison with SVCBench (Table 1, Section 4)
  • Correction of Table 9
  • Fixing typos (L255–260)

In the revised manuscript, these updates are temporarily highlighted in blue for your convenience to check.

We sincerely believe that these updates help us better deliver the benefits of the proposed VSTAT to the ICLR community.

Thank you very much,
Authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.