- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces STRAND, a benchmark designed to evaluate object-centric spatio-temporal reasoning in VideoLLMs. It decomposes complex video questions into atomic sub-questions and introduces Faithful Accuracy to jointly evaluate final answers and supporting facts. The authors also propose STRAND-Track, a framework that constructs structured object trajectories through temporal chunking, state extraction, and cross-chunk identity linking. Experiments across multiple VideoLLMs show improvements in reasoning accuracy and consistency.
Critical strengths
-
Relatively reliable benchmark construction. The benchmark is built upon human-annotated object-centric facts, followed by automatic question generation and human verification. The authors also report annotation agreement to support data quality.
-
Relatively comprehensive experimental controls. The paper includes resource-matched comparisons, prompt-matched evaluations, and component ablations, providing useful evidence for assessing the contributions of individual components.
Critical weaknesses
-
Limited Dataset Scale. STRAND contains only 88 videos and 977 target questions. Although the dataset is human-verified, its relatively small scale raises concerns about the diversity and representativeness of the evaluation. A larger and more diverse benchmark would provide stronger evidence for the generalizability of the findings.
-
Overreliance on Binary Questions and Insufficient Differentiation from Existing Benchmarks. STRAND relies primarily on Yes/No questions, which may oversimplify complex spatio-temporal reasoning. Moreover, the authors should better clarify its unique contributions compared with existing benchmarks such as RTV-Bench [1], particularly in terms of diagnostic capabilities.
-
Limitations of Faithful Accuracy. Although the authors acknowledge that Faithful Accuracy does not certify the model's internal reasoning process, a fundamental concern remains: correctly answering the main question does not necessarily require following the predefined sub-question decomposition. A model may directly identify the relevant visual evidence and reach the correct answer without relying on all the intermediate facts. Therefore, requiring every sub-question to be answered correctly may penalize valid reasoning strategies. It remains unclear to what extent Faithful Accuracy measures faithful reasoning rather than the joint correctness of separately evaluated questions.
Reference
[1] Xun, S., Tao, S., Li, J., et al. (2025). RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Video. Advances in Neural Information Processing Systems (NeurIPS).
Questions for authors
-
Dataset Scale and Generalizability. Given that STRAND contains only 88 videos, how do the authors justify its representativeness across diverse video scenarios? Would a larger evaluation be necessary to establish the generalizability of the findings?
-
Benchmark Design and Novelty. What are the unique diagnostic advantages of STRAND compared with existing benchmarks such as RTV-Bench [1]? How does the reliance on binary Yes/No questions affect its ability to evaluate complex spatio-temporal reasoning?
-
Validity of Faithful Accuracy. Since a model may directly answer the main question without following the predefined sub-question decomposition, how can Faithful Accuracy reliably reflect faithful reasoning rather than joint answer correctness? Can the authors provide further empirical evidence to support this distinction?
[1] Xun, S., Tao, S., Li, J., et al. (2025). RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Video. Advances in Neural Information Processing Systems (NeurIPS).
2 comments
Dear Reviewer owii,
We would like to kindly follow up on our previous message, as the discussion phase is nearing its end, with less than 1 hour remaining. We would be grateful to hear whether our clarifications and additional experiments have addressed your concerns. If so, we would sincerely appreciate it if you could consider revisiting your assessment in light of these updates.
Thank you again for your time and consideration.
Best regards,
Authors of Paper 32847
Dear Reviewer owii,
We would like to kindly follow up on our previous message, as the discussion phase is nearing its end, with less than 6 hours remaining. We would be grateful to hear whether our clarifications and additional experiments have addressed your concerns. If so, we would sincerely appreciate it if you could consider revisiting your assessment in light of these updates.
Thank you again for your time and consideration.
Best regards,
Authors of Paper 32847