- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces TOC-Bench, a benchmark for evaluating temporal object consistency in video-language models. Questions are grounded in object tracks and structured event timelines, with candidate answers determined before natural-language realization. The construction pipeline combines model-assisted annotation, shortcut filtering, and expert verification, yielding 2,323 QA pairs over 1,951 videos across ten diagnostic dimensions and four answer formats. Evaluation of 26 models reveals substantial difficulties in repeated-event counting, event ordering, and recognizing unsupported premises. The best reported model achieves 47.1% overall accuracy, compared with a three-person human average of 89.3%. Controlled frame-budget experiments find no consistent improvement from increasing uniformly sampled frames alone.
Critical strengths
A useful object-grounded evaluation design. The contribution is not counting or ordering in isolation, but constructing these questions from the histories of specific object instances (Section 3). Determining candidate answers from structured records before language generation, followed by expert verification, provides a reasonably auditable construction process. The combination of identity, visibility, event chronology, and false-premise questions offers useful coverage beyond general video QA.
Critical weaknesses
The figures and overall presentation need substantial improvement. In my view, several figures, particularly Figures 2 and 8, are purely AI-generated. Their heavily stylized layouts and poorly legible text weaken technical communication. The manuscript is also difficult to follow because key methodological qualifications are scattered across appendices.
Questions for authors
Were the filtering exemptions in the supplied configuration used to construct the final benchmark? Do existing logs or audits provide evidence of shortcut resistance for the exempted ReID and counting subsets?
Hi reviewers #NjJ5 and #CQ1Y, as we are approaching the end of the author rebuttal phase, could you please take a look at the authors' responses and provide your feedback asap? Thx, AC