- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents a new benchmark to assess identity and state tracking capabilities of multimodal large language models. In particular, the authors employ the well-known shell game with different settings in a controlled manner to construct their proposed benchmark consisting of synthetic and real videos. The empirical results demonstrate the deficiencies of the evaluated models in tasks involving identifying the final position of a target cup, and the final position, verification of a selected state, position at a selected step, displacement between two steps, and visit count of the ball.
Critical strengths
S1. Multimodal LLMs had become more and more capable throughout the years. As a community, it would be interesting to analyze these models to evaluate where they lack a good performance, which tasks they are not very good at. Personally, I find these kind of benchmarking efforts important, and the submitted paper includes such an effort.
Critical weaknesses
W1. The paper does focus on a single evaluation environment that is built upon the shell game. Even, the authors considered certain controlled settings, the proposed benchmark is too limited. as compared to prior efforts like VSTAT which contains shell games as one of their 39 different tasks.
W2. Experimental evaluation is weak. The authors did not evaluate any specific video large language models (VideoLLMs) like, which are specialized models built to perceive, reason over, and extract temporal data from videos.
W3. I specifically expect to see the results of models that are explicitly designed for object tracking such as Molmo2-SGCoT.
W4. No human evaluation is carried out on the proposed benchmark. Including human performances could definitely strengthen the discussion of the findings. At least, a small scale human study could have been carried out.
Minor points:
- The way the citations referenced in the text requires corrections in the latex source.
- There is a missing reference on line 140.
Questions for authors
Q1. How well do humans perform on the evaluated tasks? Q2. Does a model capable of tracking objects, such as Molmo2-SGCoT, perform better than generic multimodal modals?