acceptodds
Withdrawn from ICLR 2027

Diagnosing and Advancing Long-Video Understanding for Distributed Evidence Composition

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models can process increasingly long videos, but answering questions that depend on multiple temporally distributed observations remains challenging. We investigate how evidence demand, composition structure, and visual-context allocation affect long-video understanding. We introduce DECBench, a diagnostic benchmark comprising 926 videos and 1,123 open-ended questions spanning aggregation, temporal composition, entity-state continuity, and video-grounded explanatory reasoning and summarization. To quantify per-instance evidence demand, we introduce Minimum Evidence Count (MEC), which estimates the size of a minimum sufficient evidence set and enables comparisons across composition requirements at comparable evidence-demand levels. Our diagnostic analyses reveal two complementary bottlenecks: models often fail to acquire the necessary distributed evidence, while substantial errors persist even when relevant evidence is directly provided, indicating additional difficulty in composing the available observations. We further identify a task-dependent trade-off between temporal coverage and per-observation visual fidelity. Motivated by these findings, we propose SEEK, a training-free, query-conditioned framework that selects complementary observations according to query-implied evidence requirements and adaptively allocates visual capacity under a fixed budget. Experiments across three video MLLMs and three benchmarks show consistent improvements over query-aware frame-selection baselines. Together, DECBench and SEEK provide tools for diagnosing distributed-evidence failures and improving evidence acquisition in long-video understanding.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Large Language Models, Video Understanding, Long-Context Learning, Benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1vwiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates distributed evidence composition in long-video understanding and introduces DECBench, a benchmark containing 926 videos and 1,123 open-ended questions across four reasoning categories: aggregation, temporal composition, entity-state continuity, and causal reasoning/summarization. The authors introduce Minimum Evidence Count (MEC) to estimate per-instance evidence requirements and analyze the effects of evidence acquisition and composition difficulty. Based on these findings, they propose SEEK, a training-free framework that decomposes questions into evidence requirements, jointly selects complementary frames, and adaptively allocates visual capacity.

Critical strengths
  • The paper highlights an important distinction between retrieving relevant frames and composing temporally distributed evidence, providing useful motivation for more challenging long-video evaluation.
  • DECBench covers several complementary reasoning categories and combines real and programmatically generated videos, offering a potentially valuable diagnostic resource.
  • The matched-MEC, oracle-evidence, and fixed-budget experiments provide useful preliminary insights into evidence acquisition, composition difficulty, and temporal coverage versus visual fidelity.
Critical weaknesses
  • The paper uses Qwen3-8B as an LLM judge for open-ended answer correctness, but does not report its agreement with independent human judgments.
  • DECBench relies substantially on existing annotations, MLLM-assisted evidence annotation, automated QA construction, and model-based verification. Although the authors describe a quality-control procedure, the manuscript does not sufficiently document independent human annotation or validation, including annotator qualifications, annotation protocols, inter-annotator agreement, and the proportion of samples manually inspected.
  • Although SEEK consistently outperforms AKS and Q-Frame in Table 4, most improvements are below one percentage point. Moreover, the method is evaluated only on Qwen2.5-VL-7B, Qwen3-VL-8B, and Qwen3.5-9B, all from closely related model families and within a narrow parameter range.
  • The main comparisons primarily use a 64-frame budget. Although Appendix Table 6 investigates 16-, 32-, and 64-frame configurations under a fixed preprocessing-level patch budget, this analysis is restricted to Qwen3-VL-8B and is not a comprehensive sensitivity study of SEEK. The paper does not establish whether the proposed selection strategy retains its advantage at smaller or larger visual budgets, different video durations, or different evidence-density regimes.
  • Some main tables should be placed in the main cotent rather than in the appendix. It would be more round when they stayed in the main part.
Questions for authors
  • Could the authors clarify whether DECBench underwent independent human annotation or validation?
  • The LLM judge prompt in Appendix B.2 does not appear to include video evidence. How can the judge reliably assess whether predicted answers are visually grounded?
  • Could the authors evaluate SEEK on additional model families and larger backbones?
Reviewer_xftcOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper investigates distributed evidence composition in long-video understanding, distinguishing between acquiring relevant observations and combining them to answer questions. It introduces DECBench, containing 1,123 open-ended questions across four composition categories, and Minimum Evidence Count (MEC), which estimates the minimum evidence required to answer a question. The authors analyze performance across composition categories at matched MEC levels and investigate evidence acquisition using oracle frames. They also propose SEEK, a training-free frame-selection framework combining query-conditioned evidence planning, temporal coverage, relations, redundancy suppression, and adaptive visual-budget allocation. SEEK achieves consistent but relatively modest improvements over two baselines across three models and benchmarks.

Critical strengths

Relevant diagnostic perspective. Understanding whether video MLLMs fail because of missing evidence or an inability to compose available observations is an important problem. In particular, investigating composition requirements under comparable evidence demand could provide useful insights into existing models' limitations.

Critical weaknesses

1. Limited novelty and unclear advantages over existing work. The distinction from HERBench [1] is not sufficiently established. HERBench already investigates multi-evidence integration, introduces MRFS to quantify evidence requirements, and analyzes retrieval and fusion failures. Several DECBench categories also overlap with its existing task taxonomy. It remains unclear when researchers should prefer DECBench over HERBench, or MEC over MRFS, and what additional insights the new benchmark and metric provide. A direct comparison of both metrics across several datasets, including HERBench, seems important.

Similarly, SEEK overlaps with prior query-aware frame-selection approaches, particularly HiMu [2]. Earlier methods such as VideoTree [3], M-LLM Based Video Frame Selection [4], and VideoAgent [5] also explore query-guided selection, temporal information, or iterative evidence gathering. Although their mechanisms differ, the paper does not convincingly establish what fundamentally distinguishes SEEK.

Insufficient validation of the benchmark and its central claims. Matching MEC does not necessarily isolate composition difficulty, since questions with the same evidence count can still differ substantially in visual complexity, temporal spacing, distractors, and reasoning requirements. Consequently, the differences across categories cannot be confidently attributed to composition structure alone. Moreover, the matched-MEC analysis uses only 40 samples per category and MEC group, which is relatively small for drawing strong conclusions, particularly without uncertainty estimates or significance tests. MEC itself is also estimated through model-dependent procedures, with limited validation of its reliability.

I am also concerned about the quality control of DECBench. The paper describes several filtering and validation criteria, but mostly at a rather generic level, without clearly documenting a rigorous validation procedure or providing quantitative results on annotation quality. In particular, I could not find human evaluation results, either in terms of human performance on the benchmark or agreement on the correctness and sufficiency of the annotated evidence. This makes it difficult to assess the quality, ambiguity, and actual difficulty of the questions, especially given the reliance on automatically generated or annotated data. These concerns are consequential because the benchmark and MEC-based analysis constitute a central part of the contribution. Finally, SEEK's accuracy improvements are relatively small and their statistical significance is not established.

[1] Ben Ami, D., Serussi, G., Cohen, K., and Baskin, C. "HERBench: A Benchmark for Multi-Evidence Integration in Video Question Answering." CVPR, 2026.

[2] Ben Ami, D., Serussi, G., Cohen, K., and Baskin, C. "HiMu: Hierarchical Multimodal Frame Selection for Long Video Question Answering." arXiv:2603.18558, 2026.

[3] Wang, Z., Yu, S., Stengel-Eskin, E., Yoon, J., Cheng, F., Bertasius, G., and Bansal, M. "VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos." CVPR, 2025.

[4] Hu, K., Gao, F., Nie, X., Zhou, P., Tran, S., Neiman, T., Wang, L., Shah, M., Hamid, R., Yin, B., and Chilimbi, T. "M-LLM Based Video Frame Selection for Efficient Video Understanding." CVPR, 2025.

[5] Wang, X., Zhang, Y., Zohar, O., and Yeung-Levy, S. "VideoAgent: Long-form Video Understanding with Large Language Model as Agent." ECCV, 2024.

Questions for authors
  1. Could the authors clarify when DECBench should be preferred over HERBench, and when MEC should be preferred over MRFS? A direct comparison of MEC and MRFS across multiple datasets, including HERBench and DECBench, using comparable evaluation conditions would be particularly valuable. What new conclusions does MEC enable that MRFS cannot provide?

  2. How can the authors establish that performance differences at matched MEC levels are actually caused by composition structure rather than other task characteristics? Could they provide more controlled experiments, potentially with identical visual evidence or ground-truth textual descriptions? Given the small sample size of 40 questions per category/MEC group, how robust are the reported differences?

  3. Could the authors provide more details and quantitative results regarding the quality-control process? In particular, was the dataset independently validated by human annotators, and what were the agreement rates on answer correctness, evidence sufficiency, and task categorization? Human performance results would also help establish the difficulty and reliability of the benchmark.

  4. What are the main methodological differences between SEEK and HiMu, VideoTree, and earlier query-aware frame-selection approaches? Could the authors provide stronger comparisons under matched visual and computational budgets, especially considering the relatively small improvements over the evaluated baselines?

Reviewer_v3wsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work introduces DECBench for long-video understanding, a diagnostic benchmark comprising 926 videos and 1,123 openended questions spanning aggregation, temporal composition, entity-state continuity, and video-grounded explanatory reasoning and summarization. The diagnostic analyses reveal two interesting points: models often fail to acquire the necessary distributed evidence and substantial errors persist even when relevant evidence is directly provided. Based on these finding, this work introduces the SEEK for selecting complementary observations and adaptively allocating visual capacity under a fixed budget.

Critical strengths
  • The proposed benchmark is well motivated. The paper asks whether the relational structure of distributed evidence matters beyond the amount of evidence, which is a useful extension of prior multi-evidence work such as HERBench.
  • SEEK is formalized carefully and requires no additional training. Experiment results show SEEK slightly outperforms across three backbones and three benchmarks.
  • Table 3 is informative and interesting: It shows that going from 32 to 64 uniformly sampled frames helps little, while oracle frames help substantially but leave accuracy low.
Critical weaknesses
  • The paper does not report how many questions come from each source (curated benchmarks, Ego4D/MovieNet, programmatic). Game videos might be biased.
  • This work uses Qwen3-8B as an LLM judge only, which need further judge validation
  • SEEK’s contribution is incremental, and its gains are relatively small. Considering the current QA sizes, these gains around 0.5 points might be within plausible noise.
  • Missing baselines. Table 4 does not include a uniform 64-frame baseline. Other relevant selectors discussed in related work are also absent.
Questions for authors

Refer to weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.