acceptodds
Withdrawn from ICLR 2027

Keep Your Eye on the Cup: Diagnosing Identity and State Tracking in Video Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A wrong answer on a video benchmark can reflect missed motion, confused object identities, or an incorrectly recovered state. We examine these demands with a con- trolled shell game in which cups swap positions while carrying hidden balls. The benchmark contains 650 synthetic videos and 62 exploratory staged real videos. Its central experiment adds persistent identity markers to the same synthetic videos, holding motion, questions, and answers fixed. On monochrome videos, labeled boxes improve target-cup final-position accuracy by 10.6–65.4 percentage points across eight models. The equal-model gain is 37.5 points for final ball position but only 8.6 points for counting a ball’s visits to a position. Accuracy remains lower on longer sequences even with labels. In a separate reconstruction task, only 8.9–30.0% of videos have every reference state correct. Increasing the requested sampling rate from 2 to 4 FPS has mixed effects. These results expose sensitivity to identity information without establishing that every intervening swap was tracked: marked endpoint questions can be answered from initial and final observations

Primary area:
datasets and benchmarks
Keywords:
videollm, vlm, reasoning, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NGPbOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a new benchmark to assess identity and state tracking capabilities of multimodal large language models. In particular, the authors employ the well-known shell game with different settings in a controlled manner to construct their proposed benchmark consisting of synthetic and real videos. The empirical results demonstrate the deficiencies of the evaluated models in tasks involving identifying the final position of a target cup, and the final position, verification of a selected state, position at a selected step, displacement between two steps, and visit count of the ball.

Critical strengths

S1. Multimodal LLMs had become more and more capable throughout the years. As a community, it would be interesting to analyze these models to evaluate where they lack a good performance, which tasks they are not very good at. Personally, I find these kind of benchmarking efforts important, and the submitted paper includes such an effort.

Critical weaknesses

W1. The paper does focus on a single evaluation environment that is built upon the shell game. Even, the authors considered certain controlled settings, the proposed benchmark is too limited. as compared to prior efforts like VSTAT which contains shell games as one of their 39 different tasks.

W2. Experimental evaluation is weak. The authors did not evaluate any specific video large language models (VideoLLMs) like, which are specialized models built to perceive, reason over, and extract temporal data from videos.

W3. I specifically expect to see the results of models that are explicitly designed for object tracking such as Molmo2-SGCoT.

W4. No human evaluation is carried out on the proposed benchmark. Including human performances could definitely strengthen the discussion of the findings. At least, a small scale human study could have been carried out.

Minor points:

  • The way the citations referenced in the text requires corrections in the latex source.
  • There is a missing reference on line 140.
Questions for authors

Q1. How well do humans perform on the evaluated tasks? Q2. Does a model capable of tracking objects, such as Molmo2-SGCoT, perform better than generic multimodal modals?

Reviewer_NqfJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary
  1. Motivation: This paper investigates whether video-language models (VLMs) can reliably track object identities and state changes over time, rather than simply recognizing objects or predicting their final states.

  2. Method: The authors introduce a controlled benchmark inspired by the “Shell Game,” where multiple cups repeatedly swap positions while carrying hidden balls. They further add persistent identity cues, such as text labels, fixed-color bounding boxes, and colored points, while keeping the underlying motion, questions, and answers unchanged. This design helps distinguish object-identity confusion from failures in temporal state tracking.

  3. Key Results: Identity cues substantially improve performance on endpoint questions—for example, final ball-position accuracy increases by an average of 37.5 percentage points. However, history-dependent tasks such as visit counting improve by only 8.6 percentage points. Performance also declines as the number of swaps increases, and only 8.9%–30.0% of videos are correctly reconstructed across all intermediate states.

  4. Takeaway: The results suggest that correctly identifying an object and its final state does not necessarily mean that the model has tracked its full trajectory. Reliable long-horizon temporal state tracking therefore remains a significant weakness of current VLMs.

Critical strengths
  1. Well-motivated problem: Clearly distinguishes object identity recognition from true temporal state tracking.
  2. Controlled design: Persistent identity cues enable clean comparisons while keeping motion, questions, and answers unchanged.
  3. Clear findings: Results convincingly show that strong endpoint performance does not imply reliable history tracking.
Critical weaknesses
  1. Imperfect identity control: Identity markers also change visual salience and localization, making the source of improvement difficult to isolate.
  2. Limited real-world evaluation: The benchmark is mainly synthetic, with only 62 real videos.
  3. Potential confounding factors: More swaps also mean longer videos, so sequence length and input length are not fully disentangled.
Questions for authors
  1. Effect of Identity Cues: The identity markers may not only help distinguish objects but also improve visual salience and localization. Have the authors conducted any controlled experiments to disentangle these effects? How can we determine whether the performance gains primarily come from better identity tracking rather than easier visual recognition?

  2. Generalization to Real-World Videos: The benchmark mainly consists of synthetic videos, with only 62 real-world examples. How consistent are the main findings between synthetic and real videos? Do identity cues provide similar improvements in real-world scenarios?

  3. Impact of Video Length and Swap Complexity: Increasing the number of swaps also increases video duration, which may introduce additional challenges related to longer input sequences. Have the authors conducted experiments that control video length while varying the number of swaps, or vice versa, to better isolate the impact of temporal tracking complexity?

Reviewer_toZxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a controlled shell-game benchmark to evaluate identity and state tracking in video-language models. The benchmark consists of 650 synthetic videos and 62 staged real videos, with questions covering final object positions, intermediate states, displacement, visit counting, and full trajectory reconstruction.

The main experiment compares the same synthetic videos with and without persistent identity markers, including labeled boxes, colored boxes, and colored points. The results show that identity markers substantially improve endpoint prediction but provide much smaller improvements on history-dependent questions. The authors also investigate the effects of sequence length and frame sampling rate, finding that longer sequences remain challenging and that increasing FPS does not consistently improve performance.

Critical strengths
  1. The matched identity-marker experiment provides a useful controlled comparison.

The identity-marker interventions are applied to the same videos while preserving the underlying motions, questions, and reference answers. This is more informative than comparing independently generated videos with different visual appearances or difficulty levels.

The distinction between endpoint and history-dependent questions is also sensible. The reported difference between improvements in final ball position (+37.5 percentage points) and visit counting (+8.6 points) provides concrete evidence that supplying object identity information has different effects across question types. The paired design and uncertainty estimates make this observation reasonably convincing.

  1. The paper reports intermediate-state and trajectory-level results rather than relying only on final-answer accuracy. Including selected-step questions, visit counting, and full trajectory reconstruction is useful because correct final answers do not necessarily imply correct intermediate states.
Critical weaknesses
  1. Limited novelty and unclear significance.

The proposed benchmark overlaps substantially with existing work such as VET-Bench, Video-MME-Logical, VSTAT, and TOC-Bench. The main contribution is the comparison of different identity markers on the same shell-game videos. However, the central finding that explicit identity cues improve endpoint accuracy is rather expected, and the paper does not clearly establish what new insights this evaluation provides beyond prior benchmarks. I find the overall contribution incremental and its broader significance insufficiently demonstrated.

  1. Limited diagnostic value and insufficient experimental controls.

It remains unclear which capabilities the benchmark actually measures. Identity markers change not only object identity information but also visual salience and localization cues. Moreover, endpoint questions can often be answered without tracking intermediate swaps, while some history-dependent results are no better than simple answer-prior baselines. The lack of initial/final-frame controls and vision-based tracking baselines makes it difficult to distinguish identity recognition from temporal reasoning. The small, unmatched real-video evaluation also provides limited evidence of generalizability.

Questions for authors
  1. What specific diagnostic capability does this benchmark offer beyond existing shell-game and temporal state-tracking benchmarks?

  2. How much of the endpoint performance can be achieved using only the initial and final frames, without processing intermediate swaps?

  3. How do the authors establish that the history-dependent questions measure genuine state tracking, particularly when simple answer-prior baselines outperform several models?

  4. Why are the marker effects substantially different on monochrome and multicolored videos (Appendix Table 10), and to what extent do the main conclusions generalize beyond the synthetic setting?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.