acceptodds
Under review as a conference paper at ICLR 2027

TOC-Bench: Diagnosing Temporal Object Consistency in Video Large Language Models

Abstract

Video large language models (Video-LLMs) have achieved remarkable progress in general video understanding, yet their ability to maintain temporal object consistency remains insufficiently evaluated. This capability connects the identity, state, and persistence of the same object across occlusion, disappearance, reappearance, and cross-object interactions. We introduce TOC-Bench, a diagnostic benchmark that grounds temporal questions in explicit object tracks and structured event timelines. Candidate answers are fixed by object-event records before language realization. A three-layer temporal-necessity filtering protocol probes text-only, single-frame, and frame-shuffled shortcuts, removing 60.7% of 45,527 candidates. From the resulting pool, expert verification yields 2,323 QA pairs over 1,951 real videos, spanning 10 diagnostic dimensions and four deterministic task formats. Experiments on 26 Video-LLMs show substantial difficulties in repeated-event counting, exact event ordering, and verification of unsupported object/event premises: the strongest evaluated model achieves 47.1% accuracy, versus a three-person human average of 89.3%. Controlled 16/32/64-frame comparisons show that a larger uniform frame budget alone does not consistently improve performance. TOC-Bench offers new instance-linked annotations and a focused platform for diagnosing and improving object-aware temporal reasoning. The resource is available at https://anonymous.4open.science/r/toc_bench-1355.

Primary area:
datasets and benchmarks
Keywords:
video large language models, temporal object consistency, object-centric video understanding, temporal reasoning, video question answering, diagnostic benchmarks, hallucination evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CQ1YOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces TOC-Bench, a benchmark for evaluating temporal object consistency in video-language models. Questions are grounded in object tracks and structured event timelines, with candidate answers determined before natural-language realization. The construction pipeline combines model-assisted annotation, shortcut filtering, and expert verification, yielding 2,323 QA pairs over 1,951 videos across ten diagnostic dimensions and four answer formats. Evaluation of 26 models reveals substantial difficulties in repeated-event counting, event ordering, and recognizing unsupported premises. The best reported model achieves 47.1% overall accuracy, compared with a three-person human average of 89.3%. Controlled frame-budget experiments find no consistent improvement from increasing uniformly sampled frames alone.

Critical strengths

A useful object-grounded evaluation design. The contribution is not counting or ordering in isolation, but constructing these questions from the histories of specific object instances (Section 3). Determining candidate answers from structured records before language generation, followed by expert verification, provides a reasonably auditable construction process. The combination of identity, visibility, event chronology, and false-premise questions offers useful coverage beyond general video QA.

Critical weaknesses

The figures and overall presentation need substantial improvement. In my view, several figures, particularly Figures 2 and 8, are purely AI-generated. Their heavily stylized layouts and poorly legible text weaken technical communication. The manuscript is also difficult to follow because key methodological qualifications are scattered across appendices.

Questions for authors

Were the filtering exemptions in the supplied configuration used to construct the final benchmark? Do existing logs or audits provide evidence of shortcut resistance for the exempted ReID and counting subsets?

Reviewer_jruuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TOC-Bench, a diagnostic benchmark for evaluating temporal object consistency in Video-LLMs. Built upon explicit object tracks and structured event timelines, the benchmark contains 2,323 QA pairs from 1,951 videos, covering ten diagnostic dimensions and four question formats. Through a three layer temporal-necessity filtering protocol and expert verification, the authors aim to ensure evaluation quality. Experiments across 26 Video-LLMs reveal substantial gaps between model and human performance, particularly in event counting, temporal ordering, and false-premise verification, highlighting the limitations of current models in temporal object reasoning.

Critical strengths
  1. Meaningful and focused evaluation objective. The paper investigates an important aspect of video understanding by explicitly connecting temporal reasoning to object identities and their event histories. This provides a useful complement to existing general-purpose video question-answering benchmarks.
  2. Comprehensive empirical evaluation. The TOC-Bench evaluates 26 Video-LLMs across multiple diagnostic dimensions and question formats. The analysis includes human comparisons, hallucination-related performance, frame-budget sensitivity, and statistical uncertainty, providing useful insights into the limitations of current models.
  3. The paper is easy to follow and reproduce.
Critical weaknesses
  1. Concerns Regarding the Reliability of Temporal-Necessity Filtering. I found that a central claim of TOC-Bench is that its three-layer filtering protocol effectively eliminates non-temporal shortcuts. However, I have concerns about whether the retained questions genuinely require temporal reasoning. Specifically, Appendix C.1 indicates that questions answered incorrectly under both ordered and shuffled video inputs are still retained. To the best of my knowledge, such cases do not necessarily establish temporal necessity, as failure under both conditions may reflect the filtering model's limited perception or reasoning capabilities rather than dependence on temporal ordering. Given that temporal necessity is fundamental to the benchmark's validity, the authors should provide more evidence, such as cross-model filtering consistency and human verification of temporal necessity on the representative questions.
  2. Concerns Regarding TOC-Bench specifically evaluates temporal object consistency. TOC-Bench aims to evaluate temporal object consistency, most questions focus on general temporal reasoning, such as event counting, duration, and ordering, rather than explicitly assessing object identity and state persistence. Notably, only 23 questions (approximately 1%) address reappearance identity, despite its central role in the benchmark's definition. Furthermore, with only 1.19 QA pairs per video and independent question evaluation, the benchmark does not directly assess consistency across multiple observations or queries. The authors should further distinguish object-grounded temporal reasoning from temporal object consistency and consider additional identity-sensitive or multi-query consistency evaluations.
  3. Concerns Regarding Temporal Evidence Availability. The substantial performance gap between Video-LLMs: 47.1% and human evaluators: 89.3% may partly reflect insufficient visual evidence rather than limitations in temporal reasoning. Most models receive only 32 uniformly sampled frames, which may miss critical events involving brief actions, occlusions, or object reappearances. Moreover, it remains unclear whether human evaluators and models receive equivalent visual inputs. The authors should consider event-level evidence coverage analysis, evaluations with oracle keyframes or relevant temporal segments, and human comparisons under matched input conditions to better distinguish evidence limitations from temporal reasoning failures.
Questions for authors

Please refer to the Critical weaknesses

Reviewer_NjJ5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TOC-Bench tests whether Video-LLMs track object identity, state and persistence through occlusion, disappearance and reappearance. Object tracks come from Qwen3-VL noun phrases and SAM3. Event timelines are derived by rules, and answers are fixed before GPT-5.4-mini rewrites the questions. A three-layer filter (text-only, single-frame, shuffled-frame probes with Qwen3-VL-8B) cuts 45,527 candidates to 17,900. Experts verify 3,000 of those and keep 2,323 QA pairs over 1,951 videos, covering 10 dimensions and 4 formats, including false-premise variants. The best of 26 models scores 47.1%, against 89.3% for humans.

Critical strengths
  • Answers come from structured object-event records rather than an LLM, and scoring is deterministic.
  • The shortcut filtering and the false-premise variants (HDA-S / HDA-E) are well motivated.
  • The evaluation is broad (26 models, a human baseline, a 16/32/64-frame study), and the paper is candid about its limitations.
Critical weaknesses
  • All filtering uses a single model, so the benchmark is partly adversarial to it. Qwen3-VL-8B scores 25.2%, below the 30.4% random baseline, and several other models are also below random. This suggests filtering also kept items that are noisy or ambiguous, not just hard. Results on the unfiltered pool or with an ensemble of filter models are needed.
  • Label quality is uncertain. "Reappear" and "occlusion" events come from SAM3 visibility heuristics, so tracker dropouts can become ground truth. Each item is checked by one expert with no agreement measure, and the 22.6% rejection rate on the verified sample implies notable noise in the pipeline.
  • Some dimensions are hard to interpret. On Count, always answering "2" gives 45.3%, above every model, so low scores may reflect format or prior effects rather than tracking. ReID has only 23 items, with many models at or near 100%. About 95% of videos come from Charades and Perception Test, so the occlusion-heavy sources (OVIS + MOSE, 90 videos) contribute little.
  • Events are timed on a 3 fps tracking timeline, but models see 32 uniform frames, so short occlusions may not be visible in the input at all. It is unclear whether the human solvers watched the full video or the same frames.
Questions for authors
  1. How do models score on items removed by the filter, or with a different filter model?
  2. Did human solvers see the full video or the same 32 frames?
  3. Can you double-annotate a subset and report agreement and the residual label error rate?

Comments on the paper

1 comment

Hi reviewers #NjJ5 and #CQ1Y, as we are approaching the end of the author rebuttal phase, could you please take a look at the authors' responses and provide your feedback asap? Thx, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.