- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates distributed evidence composition in long-video understanding and introduces DECBench, a benchmark containing 926 videos and 1,123 open-ended questions across four reasoning categories: aggregation, temporal composition, entity-state continuity, and causal reasoning/summarization. The authors introduce Minimum Evidence Count (MEC) to estimate per-instance evidence requirements and analyze the effects of evidence acquisition and composition difficulty. Based on these findings, they propose SEEK, a training-free framework that decomposes questions into evidence requirements, jointly selects complementary frames, and adaptively allocates visual capacity.
Critical strengths
- The paper highlights an important distinction between retrieving relevant frames and composing temporally distributed evidence, providing useful motivation for more challenging long-video evaluation.
- DECBench covers several complementary reasoning categories and combines real and programmatically generated videos, offering a potentially valuable diagnostic resource.
- The matched-MEC, oracle-evidence, and fixed-budget experiments provide useful preliminary insights into evidence acquisition, composition difficulty, and temporal coverage versus visual fidelity.
Critical weaknesses
- The paper uses Qwen3-8B as an LLM judge for open-ended answer correctness, but does not report its agreement with independent human judgments.
- DECBench relies substantially on existing annotations, MLLM-assisted evidence annotation, automated QA construction, and model-based verification. Although the authors describe a quality-control procedure, the manuscript does not sufficiently document independent human annotation or validation, including annotator qualifications, annotation protocols, inter-annotator agreement, and the proportion of samples manually inspected.
- Although SEEK consistently outperforms AKS and Q-Frame in Table 4, most improvements are below one percentage point. Moreover, the method is evaluated only on Qwen2.5-VL-7B, Qwen3-VL-8B, and Qwen3.5-9B, all from closely related model families and within a narrow parameter range.
- The main comparisons primarily use a 64-frame budget. Although Appendix Table 6 investigates 16-, 32-, and 64-frame configurations under a fixed preprocessing-level patch budget, this analysis is restricted to Qwen3-VL-8B and is not a comprehensive sensitivity study of SEEK. The paper does not establish whether the proposed selection strategy retains its advantage at smaller or larger visual budgets, different video durations, or different evidence-density regimes.
- Some main tables should be placed in the main cotent rather than in the appendix. It would be more round when they stayed in the main part.
Questions for authors
- Could the authors clarify whether DECBench underwent independent human annotation or validation?
- The LLM judge prompt in Appendix B.2 does not appear to include video evidence. How can the judge reliably assess whether predicted answers are visually grounded?
- Could the authors evaluate SEEK on additional model families and larger backbones?