acceptodds
Under review as a conference paper at ICLR 2027

STRAND: Benchmarking and Improving Object-Centric Spatio-Temporal Monitoring in Video Large Language Models

Abstract

While multimodal large language models (MLLMs) have advanced video understanding, they remain highly prone to hallucinations in dynamic scenes. We argue this stems from a failure in spatio-temporal monitoring, the ability to persistently track object identities, states, and relations over time. Existing benchmarks obscure this deficit by relying on single final-answer evaluations for queries that can often be resolved via local visual cues or statistical priors. To rigorously diagnose this, we introduce STRAND (Spatio-Temporal Reasoning Audited by Necessary Decomposition), a benchmark of human-verified object-centric facts that evaluates intermediate reasoning by decomposing queries into sub-questions, distinguishing genuine temporal understanding from coincidental correctness. Crucially, we score models with Faithful Accuracy, an unconditional joint metric that credits a prediction only when the target answer and every prerequisite sub-question are correct, so that a model cannot inflate its score by being selectively consistent on the small subset of targets it happens to answer correctly. To address failure modes exposed by STRAND, we further propose STRAND-Track, an object-centric framework that explicitly constructs and reasons over structured object trajectories via chunk-wise state extraction and temporal aggregation. Extensive experiments, including comparisons against end-to-end MLLMs and modular video harnesses on a matched backbone, a configuration matched on frames, calls and tokens simultaneously, and a configuration matched on the answering prompt and its decoding settings, demonstrate that our object-centric framework significantly reduces hallucinated answers and improves spatio-temporal reasoning consistency over state-of-the-art MLLMs. Component ablations isolate cross-chunk identity linking as the step the gain rests on, while an oracle-trajectory reference shows that constructing the trajectories and reasoning over them leave comparable shares of the remaining error, so neither stage alone closes the gap.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Large Language Models, Spatio-Temporal Reasoning, Object-Centric Video Understanding, Video Hallucination, Faithful Reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_owiiOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces STRAND, a benchmark designed to evaluate object-centric spatio-temporal reasoning in VideoLLMs. It decomposes complex video questions into atomic sub-questions and introduces Faithful Accuracy to jointly evaluate final answers and supporting facts. The authors also propose STRAND-Track, a framework that constructs structured object trajectories through temporal chunking, state extraction, and cross-chunk identity linking. Experiments across multiple VideoLLMs show improvements in reasoning accuracy and consistency.

Critical strengths
  1. Relatively reliable benchmark construction. The benchmark is built upon human-annotated object-centric facts, followed by automatic question generation and human verification. The authors also report annotation agreement to support data quality.

  2. Relatively comprehensive experimental controls. The paper includes resource-matched comparisons, prompt-matched evaluations, and component ablations, providing useful evidence for assessing the contributions of individual components.

Critical weaknesses
  1. Limited Dataset Scale. STRAND contains only 88 videos and 977 target questions. Although the dataset is human-verified, its relatively small scale raises concerns about the diversity and representativeness of the evaluation. A larger and more diverse benchmark would provide stronger evidence for the generalizability of the findings.

  2. Overreliance on Binary Questions and Insufficient Differentiation from Existing Benchmarks. STRAND relies primarily on Yes/No questions, which may oversimplify complex spatio-temporal reasoning. Moreover, the authors should better clarify its unique contributions compared with existing benchmarks such as RTV-Bench [1], particularly in terms of diagnostic capabilities.

  3. Limitations of Faithful Accuracy. Although the authors acknowledge that Faithful Accuracy does not certify the model's internal reasoning process, a fundamental concern remains: correctly answering the main question does not necessarily require following the predefined sub-question decomposition. A model may directly identify the relevant visual evidence and reach the correct answer without relying on all the intermediate facts. Therefore, requiring every sub-question to be answered correctly may penalize valid reasoning strategies. It remains unclear to what extent Faithful Accuracy measures faithful reasoning rather than the joint correctness of separately evaluated questions.

Reference

[1] Xun, S., Tao, S., Li, J., et al. (2025). RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Video. Advances in Neural Information Processing Systems (NeurIPS).

Questions for authors
  1. Dataset Scale and Generalizability. Given that STRAND contains only 88 videos, how do the authors justify its representativeness across diverse video scenarios? Would a larger evaluation be necessary to establish the generalizability of the findings?

  2. Benchmark Design and Novelty. What are the unique diagnostic advantages of STRAND compared with existing benchmarks such as RTV-Bench [1]? How does the reliance on binary Yes/No questions affect its ability to evaluate complex spatio-temporal reasoning?

  3. Validity of Faithful Accuracy. Since a model may directly answer the main question without following the predefined sub-question decomposition, how can Faithful Accuracy reliably reflect faithful reasoning rather than joint answer correctness? Can the authors provide further empirical evidence to support this distinction?

[1] Xun, S., Tao, S., Li, J., et al. (2025). RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Video. Advances in Neural Information Processing Systems (NeurIPS).

2 comments

Dear Reviewer owii,

We would like to kindly follow up on our previous message, as the discussion phase is nearing its end, with less than 6 hours remaining. We would be grateful to hear whether our clarifications and additional experiments have addressed your concerns. If so, we would sincerely appreciate it if you could consider revisiting your assessment in light of these updates.

Thank you again for your time and consideration.

Best regards,

Authors of Paper 32847

Dear Reviewer owii,

We would like to kindly follow up on our previous message, as the discussion phase is nearing its end, with less than 1 hour remaining. We would be grateful to hear whether our clarifications and additional experiments have addressed your concerns. If so, we would sincerely appreciate it if you could consider revisiting your assessment in light of these updates.

Thank you again for your time and consideration.

Best regards,

Authors of Paper 32847

Reviewer_YN8gOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces STRAND, a benchmark for evaluating object-centric spatio-temporal monitoring in video large language models, and STRAND-Track, a framework that explicitly constructs and reasons over object trajectories.

Critical strengths

The paper identifies an important limitation of conventional video-understanding benchmarks. A correct final answer does not necessarily indicate that a model has correctly tracked the relevant objects and events throughout a video. And the benchmark is built from human-annotated object-centric facts rather than relying entirely on automatically generated video descriptions.

Critical weaknesses
  1. Although STRAND is carefully constructed, it contains limited number of videos, which limits the strength of conclusions about generalization across different video domains. Also, although the paper includes contrastive and open-form probes to investigate potential answer-prior effects, these supplementary evaluations remain limited in scale and do not constitute a comprehensive evaluation of open-ended video understanding.

  2. STRAND-Track introduces additional preprocessing to construct object trajectories before answering questions. While this cost can be amortized across multiple queries about the same video, the benefit is less clear for one-off queries or latency-sensitive applications.A more explicit characterization of the workloads may better support the paper.

Questions for authors

same as weakness.

2 comments

Dear Reviewer YN8g,

We would like to kindly follow up on our previous message, as the discussion phase is nearing its end, with less than 6 hours remaining. We kindly request to let us know if the above clarifications and the previously added experiments have addressed your concerns.

We thank the reviewer for engaging with us in the discussion.

Best regards,

Authors of Paper 32847

Dear Reviewer YN8g,

We would like to kindly follow up on our previous message, as the discussion phase is nearing its end, with less than 1 hour remaining. We kindly request to let us know if the above clarifications and the previously added experiments have addressed your concerns.

We thank the reviewer for engaging with us in the discussion.

Best regards,

Authors of Paper 32847

Reviewer_BKK2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

STRAND evaluates final answers and intermediate facts through 977 human-verified yes/no targets. Faithful Accuracy requires both the target and all sub-questions to be correct. STRAND-Track answers from explicitly tracked object trajectories, reaching 59.4 Faithful Accuracy versus 38.9 for the best end-to-end model, though matched controls substantially reduce the gain.

Critical strengths
  1. Faithful Accuracy improves on conditional consistency, supported by checks for language priors, label balance, and single-frame shortcuts.

  2. The controls cover prompts, compute budgets, generic scaffolding, identity linking, and oracle trajectories.

  3. Annotation agreement is high (κ = 0.88), and extracted trajectories are evaluated against gold annotations.

Critical weaknesses
  1. Under the strictest control, the gain is only 2.1 points, close to the paper’s 2.0-point tie threshold. Prompting alone recovers most of Gemini’s target-accuracy gain. The method also closely matches the annotation format, while Table 1 uses an unmatched baseline prompt.

  2. Requiring every answer to be correct naturally compounds independent errors; low Faithful Accuracy need not imply lucky final answers. With only 88 videos and yes/no questions containing presuppositions, the metric’s interpretation as faithfulness is less convincing than its use for ranking systems.

Questions for authors

Please see the weaknesses section.

2 comments

Dear Reviewer BKK2,

We would like to kindly follow up on our previous message, as the discussion phase is nearing its end, with less than 6 hours remaining. We kindly request to let us know if the above clarifications and the previously added experiments have addressed your concerns.

We thank the reviewer for engaging with us in the discussion.

Best regards,

Authors of Paper 32847

Dear Reviewer BKK2,

We would like to kindly follow up on our previous message, as the discussion phase is nearing its end, with less than 1 hour remaining. We kindly request to let us know if the above clarifications and the previously added experiments have addressed your concerns.

We thank the reviewer for engaging with us in the discussion.

Best regards,

Authors of Paper 32847

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.