acceptodds
Withdrawn from ICLR 2027

Beyond Frame Selection: Rethinking Long-Video Understanding with MLLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal Large Language Models (MLLMs) have made strong progress in video understanding, yet long videos remain difficult: the visual token budget grows with video length, so temporally sparse evidence is easily lost. Existing methods, whether frame selection or agent-based exploration, ultimately rely on a single frame set that must trade off broad temporal coverage against fine-grained, question-relevant evidence: mixing both dilutes decisive evidence, while focusing on either sacrifices the other. We propose VideoRouter (VR), which rethinks long-video understanding as coordinating complementary evidence views rather than selecting a single subset of frames. VideoRouter first organizes each video into a question-agnostic temporal hierarchy that partitions it into coarse-to-fine temporally coherent segments. Upper-level nodes capture broad storyline context and event progression, while lower-level nodes preserve fine-grained local details and evidence-bearing moments. This gives rise to two complementary views: a global view for coverage-oriented reasoning and a local view for detail-oriented evidence recovery. We further introduce a verification-guided router that judges which view is better supported by its own selected evidence and decides the final answer. Across six backbones, routing improves over both views in all settings, and the choice of view is shown to be dataset-dependent, confirming that no single evidence granularity is universally preferable. On VideoMME, our method outperforms state-of-the-art frame selection methods by 2.5 points, under the LLaVA-Video-7B backbone. We will release the code.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long Video Understanding, Multimodal Large Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_hdeGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose VideoRouter, a training-free framework based on temporal hierarchical representation in videos and the MLLM-based self-verification mechanism. VideoRouter first constructs a global answer and a local answer, and then leverages a voting router to decide the final response (or fallback if the requirement is not met). The experiments demonstrate the effectiveness of the framework across different MLLMs, and it also outperforms previous video agents. In the ablation studies, the authors also discuss the impact of temporal hierarchy and the routing mechanism.

Critical strengths
  1. The authors use the global-local hierarchical representation in long-form videos, which grasps the challenge that the downstream tasks in such videos do either relate to overall comprehension or local fine-grained perception/reasoning.
  2. The routing mechanism looks reasonable: first assess the soundness of the initial prediction, and then merge them into the final response. This design makes the model adaptively take advantage of the evidence distribution in the video sequence based on the task itself (global or local).
  3. The ablation discusses the main modules carefully (hierarchy and routing).
  4. The effectiveness of the VideoRouter is demonstrated in the main experiments.
Critical weaknesses
  1. The evaluation setting is not clear. What is the base model performance? In table 1, only Local and Global is provided, but on videomme, the performance for qwen2.5-VL-7B is 65.1(w/o subs)/71.6(w/ subs) from the its tech report; and on LongVideoBench validation set the performance is 56.0. In this case, the improvement on Videomme is only around 0.4 and on longvideobench is as high as 8.23.

  2. The inference cost. The authors provide this in the appendix D. And compared with AKS, VideoRouter is more efficient in token usage. However, what about compared with the base model itself, e.g., Qwen2.5-VL-7B? It is also of great value to see the trade of between performance and efficiency when equipped with such a strategy. If, as mentioned in weakness 1, the improvemtn is limited (e.g., in videomme), the extra computation cost will be viewed as useless.

  3. How to demonstrate the effectiveness in the verification process since the same model is used here? How many correct prediction can score 2 and how many wrong prediction can score 1 or 0?

  4. After fallback, is there any further improvement? Only fallback rate is provided in the appendix.

  5. Is there any evidence to illustrate that the two branches indeed capture the global story and local details?

Questions for authors

See above in Critical Weakness.

Reviewer_Ak1vOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VideoRouter, a framework for long-video understanding that constructs a hierarchical video representation and selects visual evidence from complementary global and local views. It then employs MLLM-based verification-guided routing to determine the final answer.

Critical strengths

[S1] Consistent Performance Improvements. The proposed method demonstrates consistent performance improvements across multiple MLLM backbones on VideoMME and LongVideoBench.

[S2] Training-Free Framework. The proposed framework is training-free and can be applied to different MLLM backbones without additional model training.

Critical weaknesses

[W1] Limited Novelty in Hierarchical Evidence Selection. The core ideas of hierarchical video representation and coarse-to-fine evidence selection have already been explored in prior work, such as VideoTree. The proposed approach offers limited methodological novelty beyond these existing techniques.

[W2] Limited Novelty in Verification-Guided Routing. The proposed routing mechanism is essentially an LLM-as-a-Judge approach with rule-based answer selection. It does not introduce a novel decision mechanism that explicitly considers the temporal structure of videos.

[W3] Unclear Distinction Between Global and Local Views. The distinction between Global and Local views appears to be primarily based on different frame-selection strategies (query-agnostic representative sampling vs. query-guided relevance sampling), rather than explicitly defined temporal granularity. Moreover, the Local view may select multiple relevant segments distributed across the entire video, rather than a single localized temporal region. Therefore, it remains unclear whether the observed complementarity truly arises from differences in temporal granularity or simply from employing different frame-selection strategies.

[W4] Marginal Performance Gains Despite Additional Inference Cost. Despite requiring multiple MLLM inference calls, the proposed method achieves relatively modest improvements over existing frame-selection methods, raising concerns about its practical efficiency.

Questions for authors

[Q1] Inference Efficiency. A comparison of inference latency and GPU memory usage with existing methods would be helpful to better assess the computational efficiency of VideoRouter.

[Q2] Missing Q-Frame Results on VideoMME. Table 3 reports Q-Frame results only on LongVideoBench. Could the authors clarify why the results on VideoMME are missing and provide the corresponding performance using LLaVA-Video-7B?

[Q3] Additional Comparisons on Qwen Backbones. Although the paper includes extensive experiments with Qwen-based models, comparisons with AKS and Q-Frame are limited to LLaVA-Video-7B. Could the authors provide additional results comparing these methods on Qwen backbones to further validate the effectiveness of VideoRouter?

[Q4] Alternative Frame Selection for the Global View. The Global view selects representative frames based on temporal clustering. Would using an existing frame-selection method such as AKS, which considers both temporal coverage and question relevance, provide more effective global evidence? Could the authors evaluate the impact of replacing the current Global frame selection with AKS on the final routing performance?

Reviewer_e9jvOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • This paper introduces VideoRouter, a training-free framework for long-video question answering that constructs complementary global and local evidence views from a shared temporal hierarchy.
  • A verification-guided router evaluates the evidence supporting each branch's prediction, either selecting the more reliable answer or triggering joint reasoning over the combined views.
  • Evaluations across six backbones on VideoMME and LongVideoBench show consistent gains over standalone branches and existing frame-selection baselines under matched inference call budgets.
Critical strengths
  • Well-motivated global–local complementarity. Across all six main backbones, the global branch excels on VideoMME whereas the local branch excels on LongVideoBench. Pruning analyses further show that a weaker standalone local branch can provide stronger complementarity, empirically justifying the joint coordination of both views.

  • Consistent gains across diverse backbones. Verification-guided routing outperforms the stronger standalone branch across all twelve primary model–dataset pairs and additional backbones, boosting Qwen2.5-VL-7B from 61.63 to 65.52 on VideoMME and from 60.32 to 64.23 on LongVideoBench.

Critical weaknesses
  • Moderate conceptual novelty over prior hierarchical retrieval and routing. Coarse-to-fine video hierarchies (A) and similar routing mechanism (B) are already well established. Because the core distinction lies in verifying complementary global and local predictions before selection or joint reasoning, the contribution is better framed as a practical inference pipeline rather than a fundamentally new paradigm.

  • Insufficient comparison against recent frame-selection baselines. Table 3 evaluates only AKS and Q-Frame on LongVideoBench and only AKS on VideoMME, omitting competitive recent methods such as BOLT(C), MDP³(D), FOCUS(E), and DIG(B) which also adapts frame selection to global versus localized queries. Comparing against these baselines under matched backbones and budgets (reporting both accuracy and end-to-end latency) is needed to substantiate the advantage of global-local routing.

  • unfair comparison with repeated-run AKS. To match VideoRouter's inherently multi-call routing and verification ceiling in Table 14, the authors wrap AKS (a native single-pass method) in five independent inference runs with majority voting, artificially inflating its frame and visual-token consumption. A fair comparison should evaluate both methods in their native configurations and report accuracy alongside end-to-end latency (including feature extraction and hierarchy construction).

  • Limited benchmark coverage. The method is evaluated on only two video QA benchmarks, VideoMME and LongVideoBench. Evaluation on additional video QA benchmarks covering diverse domains and question types is needed to establish the robustness and generalizability of the reported gains.

[A] VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos. CVPR2025

[B] Divide, then Ground: Adapting Frame Selection to Query Types for Long-Form Video Understanding. CVPR 2026

[C] BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding. CVPR 2025

[D] MDP³: A Training-free Approach for List-wise Frame Selection in Video-LLMs. ICCV 2025

[E] FOCUS: Efficient Keyframe Selection for Long Video Understanding. ICLR 2026

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.