acceptodds
Withdrawn from ICLR 2027

REASON FIRST, RETRIEVE WHAT MATTERS: HYPOTHESIS-GUIDED KEYFRAME SELECTION FOR LONG VIDEO UNDERSTANDING

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long video understanding requires models to recover and reason over sparse, temporally distributed evidence under limited frame budgets. Existing keyframe selection (KFS) methods typically retrieve query-relevant frames from a densely sampled candidate set before inference. Despite recent progress, existing KFS methods still have two key limitations. Global image–text matching introduces unnecessary computation when a small set of uniformly sampled frames already provides sufficient evidence. Moreover, selecting frames mainly by query relevance may miss evidence that distinguishes competing answers, and one-shot frame selection provides no opportunity to correct selection errors. We propose InferKFS, a training-free and plug-and-play framework that couples keyframe selection with iterative MLLM inference. InferKFS first reasons over sparsely sampled frames to construct an initial hypothesis space. The resulting answer distribution determines whether further frame selection is required; when activated, the retained hypotheses guide discriminative frame selection, followed by renewed inference and hypothesis refinement. This iterative process focuses additional visual computation on unresolved hypotheses while avoiding unnecessary global frame matching. Experiments on Video-MME, LongVideoBench, and MLVU demonstrate consistent improvements across different MLLM backbones. InferKFS improves accuracy by up to 6.7 percentage points and achieves 72.9% accuracy on Video-MME with only 1.3% relative frame–hypothesis matching cost.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long Video Understanding, Multimodal Large Language Models, Keyframe Selection, Belief-Guided Evidence Acquisition, Training-Free Inference, Efficient Video Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bF3mOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HypoKFS, a training-free framework for long-video question answering that integrates keyframe selection with iterative inference. It first constructs an answer distribution from sparsely sampled frames and selectively retrieves additional evidence for ambiguous queries. The retrieved frames are scored using hypothesis-conditioned CLIP similarity, discriminative utility, and temporal diversity. Experiments on Video-MME, LongVideoBench, and MLVU demonstrate improvements over existing keyframe selection methods with reduced image–text matching computation.

Critical strengths
  1. Reasoning-guided frame selection. Integrating answer uncertainty into keyframe retrieval is a practical approach that can avoid unnecessary retrieval and prioritize discriminative visual evidence.

  2. Consistent empirical improvements. The method achieves promising accuracy gains across three long-video benchmarks and two open-source MLLM backbones, with component ablations supporting the effectiveness of its design.

Critical weaknesses
  1. Insufficient comparisons with agent-based methods. HypoKFS is essentially an iterative video reasoning and retrieval framework, but the experiments mainly compare against conventional keyframe selection methods. Closely related training-free agentic approaches, such as VideoAgent, VideoTree, DrVideo, VideoAgent2, and VideoChat-A1, are missing. This makes it difficult to assess the novelty and benefits of the proposed iterative reasoning strategy.

  2. Efficiency and generalizability are insufficiently validated. The reported efficiency mainly concerns image–text matching cost rather than end-to-end latency. Multiple MLLM inference rounds may introduce substantial additional overhead. Moreover, HypoKFS is evaluated only with open-source backbones and a fixed CLIP matching encoder, leaving its claimed plug-and-play generalizability insufficiently supported.

Questions for authors

Q1. Comparisons with training-free video agents. Could the authors compare HypoKFS with representative agent-based methods, such as VideoTree, VideoAgent2, and VideoChat-A1, under comparable backbone and computation settings? These methods also perform iterative evidence acquisition and reasoning.

Q2. End-to-end inference efficiency. Could the authors report end-to-end inference latency against existing methods, including CLIP matching, video processing, and all MLLM inference rounds? The reduced matching cost does not necessarily translate into lower overall latency.

Q3. Generalization to proprietary MLLMs. Since HypoKFS is presented as a training-free, plug-and-play framework, could the authors evaluate it with at least one closed-source MLLM? How would the hypothesis distribution be obtained when candidate-token logits are unavailable through the API?

Q4. Dependence on CLIP. The proposed frame selection relies on CLIP ViT-B/32 to measure frame–hypothesis similarity, which may limit its ability to capture complex visual semantics and temporal evidence. Could the authors investigate using a lightweight MLLM for frame–hypothesis relevance scoring instead of CLIP? How would this affect selection accuracy and inference efficiency?

Q5. Generalization to open-ended VideoQA. All evaluations are conducted on multiple-choice benchmarks, where candidate hypotheses are predefined and their probabilities can be directly extracted from answer-option logits. Can HypoKFS be extended to open-ended VideoQA without predefined answer candidates? How would the authors construct and update the hypothesis space, estimate confidence, and perform hypothesis-guided frame selection in this setting? An evaluation on an open-ended VideoQA benchmark would be important to validate the general applicability of the framework.

2 comments
AuthorsOpenReview ↗

Response to bF3m

Q1. Novelty and Comparison with Video Agents

We thank the reviewer for suggesting agent-based comparisons and welcome further questions. Our motivation differs from agentic retrieval: Fig. 3(b) shows that the correct answer already appears among the top two hypotheses inferred from sparse frames for 89.4% of Video-MME questions, suggesting that initial answer probabilities contain useful information about competing answers. Our contribution is coupled hypothesis-discriminative frame scoring and numerical iterative evidence acquisition, rather than agentic planning. The score Dk(ft)=∑i<jwikwjk∑u<vwukwvk∣st,i−st,j∣D_k(f_t)=\sum_{i<j}\frac{w_i^kw_j^k}{\sum_{u<v}w_u^kw_v^k}|s_{t,i}-s_{t,j}| couples hypothesis-pair importance with frame-level separability, complemented by temporal diversity. Whereas VideoAgent relies on LLM judgments of evidence sufficiency and VideoHV-Agent converts competing hypotheses into a shared verbal clue for agentic verification, HypoKFS scores each frame directly against multiple hypothesis pairs and updates acquisition weights after each selection without further LLM calls. Renewed inference then reconstructs the hypothesis space to focus on unresolved ambiguity.

To illustrate the cost of agentic retrieval, we compare with WorldMM-8B, a Qwen3-VL-8B-based agent using iterative multimodal memory retrieval, on Video-MME Long:

MethodBackboneVideo-MME Long (%)End-to-end latency (s/question)
WorldMM-8BQwen3-VL-8B66.060.3
HypoKFS (Ours)Qwen3-VL-8B63.99.1

WorldMM's accuracy is from its paper; both latencies are from our measurements. WorldMM gains 2.1 accuracy points but takes 6.6× as long (60.3 vs. 9.1 s), illustrating the latency cost of agentic retrieval without implying superior HypoKFS accuracy.

Q2. End-to-End Inference Efficiency

We appreciate this important suggestion and will include the end-to-end analysis in the main paper. The Qwen3-VL-8B / Video-MME measurements below were obtained on identical H20-3e hardware and include initial inference, frame encoding/matching, and all subsequent model passes:

MethodInitial answer attemptFrame encoding & matchingSubsequent model passesTotalAccuracy
HypoKFS32 frames (3.2 h)1.3% (1.4 h)16.8 frames (2.2 h)6.8 h67.9%
AKS64 frames (6.6 h)3.7% (4.0 h)None10.6 h65.1%

HypoKFS reduces total runtime by approximately 35.9% (6.8 vs. 10.6 h), while improving accuracy by 2.8 percentage points in this efficiency setting. This efficiency setting differs from Q1.

Q3. Generalization to Stronger and Proprietary MLLMs

Due to server VPN restrictions, we cannot reliably access overseas proprietary APIs. We therefore evaluate plug-and-play transfer using a strong domestically accessible model with available answer-token logits, Qwen3.6-27B, under a 128-frame Video-MME budget:

MethodFrame budgetVideo-MME (%)
Qwen3.6-27B (our reproduction)12873.3
Qwen3.6-27B + HypoKFS12876.8

The 3.5-point improvement supports transfer to a stronger backbone without additional training. The method still requires candidate-token probabilities; this does not establish compatibility with proprietary text-only APIs.

Q4. Dependence on CLIP

We thank the reviewer for suggesting a multimodal scoring alternative. With Qwen3-VL-8B fixed as the answer backbone, we replace CLIP frame–hypothesis similarity with Qwen3-VL-4B prompted to rate each frame–hypothesis pair from 1 to 4, followed by normalization:

MethodVideo-MME accuracy
Uniform sampling (Qwen3-VL-8B)67.6%
HypoKFS + Qwen3-VL-4B scorer70.3%
HypoKFS + CLIP ViT-B/3273.8%

The MLLM scorer improves over uniform sampling but underperforms CLIP in this setting. One plausible limitation is that our heuristic acquisition parameters were calibrated for CLIP similarities and were not retuned for the 4B scorer's discrete rating distribution, which changes both pairwise score gaps and numerical weight updates. We will acknowledge this matcher-dependent calibration requirement.

Q5. Open-Ended VideoQA

We additionally evaluate the entire ActivityNet-QA dataset, which requires free-form answers rather than predefined multiple-choice options. HypoKFS generates candidate hypotheses from sparse initial evidence, estimates their relative support, and applies the same discriminative selection and numerical acquisition updates. With LLaVA-Video-7B, we obtain:

MethodActivityNet-QA (full dataset)
Uniform (our reproduction)56.3%
HypoKFS (Ours)57.9%

The 1.6-point improvement supports applicability to open-ended QA. The smaller gain may reflect shorter videos, with less need for targeted long-range retrieval.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Remaining Concerns

Dear Reviewer,

Thank you again for your valuable comments and for taking the time to review our work. We would greatly appreciate it if you could let us know whether our rebuttal has adequately addressed your concerns, or if there are any remaining issues that we could further clarify. We would be happy to provide additional explanations if needed.

Thank you for your time and consideration.

Reviewer_GqxgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HypoKFS, a training-free, plug-and-play method for selecting keyframes to enable a vision-language model to answer multiple choice questions about long videos. HypoKFS first gives the model a small set of regularly spaced frames and checks the most likely answer options. If the model is sufficiently confident, it returns the answer. Otherwise, it searches for frames that help distinguish the remaining possible answers. It also encourages selection from different times to avoid repetitive evidence. The model then answers again using the selected frames with up to 2 rounds of additional selection. No additional model training is required.

The paper evaluates Qwen3-VL-8B and LLaVA-Video-7B on Video-MME, LongVideoBench and MLVU. Reported results show better performance compared to uniform sampling and other frame selection methods while processing fewer frames on average. The ablations show better results when the method retrieves more frames only when needed. Accuracy also improves when frames distinguish possible answers. More gains come from updates that reduce repeated evidence for the same answer.

Critical strengths

1.The proposed method connects frame selection to the uncertainty in the model’s answer. It first checks whether the initial frames are enough to answer the question. When several answers are possible, it searches for frames that could distinguish them. This provides a framework to when to retrieve more frames and which frames to choose without addititonal model training.

  1. HypoKFS improves performance with Qwen3-VL and LLaVA-Video on Video-MME, LongVideoBench and MLVU. It does so with less frames processed than other methods. The ablations also show gains from frames that distinguish possible answers and also from updates to later choices of frames for short, medium and long videos. On Video-MME, the correct answer is among the model’s two most likely initial answers in about 89% of cases. This shows that initial predictions can narrow the search.
Critical weaknesses
  1. Prior relevant works are not discussed. ReQuest already uses answer uncertainty to trigger additional frame selection. VQOS matches video content against the question combined with each possible answer. VideoHV-Agent identifies possible answers and searches for visual evidence that can distinguish them. The paper needs to explain what it adds compared to relevant prior works to justify novelty.

  2. HypoKFS aims to reduce the cost of finding useful frames in long videos. The paper reports less frame matching and fewer processed frames on average. However, the method also requires an initial model pass and up to 2 more passes. There is no total inference time or computational cost reported.

  3. The paper also does not show how well retrieval recovers when the correct answer initially receives a low probability.

Questions for authors
  1. Could you compare your frame scoring rule with VQOS’s approach, which selects evidence that matches any possible answer? Please keep the model, encoder, retrieval rounds and frame budget unchanged. This would show whether selecting frames that distinguish answers provides an advantage over answer-based retrieval.

  2. What is the total inference time when including the initial answer attempt, frame encoding and matching, and all subsequent model passes? How does this compare with the baselines on the same hardware? .

  3. Among questions answered immediately because the model is confident, how often is that answer wrong? For questions that trigger retrieval, how often does the initial set of retained answers exclude the correct answer. Also, how often do later rounds recover it?

3 comments
AuthorsOpenReview ↗

Response to Gqxg

Q1. Novelty and Controlled Comparison with VQOS

We thank the reviewer, particularly for the valuable Q2 efficiency suggestion. We will incorporate both the latency analysis and VQOS comparison into the main paper and welcome further questions. Despite overlapping terminology, our insight and methodology differ.

Motivation/Insight: ReQuest targets missing evidence and unreliable frame–question relevance; VQOS improves option-conditioned relevance; VideoHV-Agent addresses semantic drift. We instead experimentally study the initial predictions (Introduction, Figs. 1 and 3(b)): for 89.4% of Video-MME questions, the correct answer is among the top two sparse-frame hypotheses. Thus, answer probabilities can guide selection of informative, discriminative evidence by coupling hypothesis importance and frame-level separability, beyond merely triggering retrieval.

Methodology: Our contribution lies in leveraging the informative hypothesis distribution obtained from initial inference to guide frame selection through coupled frame scoring and numerical iterative evidence acquisition, rather than merely combining existing pipeline stages. The score Dk(ft)=∑i<jwikwjk∑u<vwukwvk∣st,i−st,j∣D_k(f_t)=\sum_{i<j}\frac{w_i^kw_j^k}{\sum_{u<v}w_u^kw_v^k}|s_{t,i}-s_{t,j}| couples each competing pair's importance with its frame-level separability, complemented by explicit temporal diversity. Unlike VQOS’s maximum option similarity or VideoHV-Agent’s shared verbal clue for agentic verification, HypoKFS scores each frame directly against multiple hypothesis pairs. Within a selection round, numerical acquisition weights update after every selected frame without LLM calls; across rounds, renewed inference reconstructs the hypothesis space to resolve remaining ambiguity.

Numerical iterative evidence acquisition refers to within-round score–select–update operations without LLM/tool reasoning. Iterative reasoning refers to a recurrent evidence–inference loop; a single rethink is not counted.

MethodContemporaneousInsightInitial decisionOpen-ended QACoupled hypothesis discriminationTemporal diversityNumerical iterative evidence acquisitionIterative reasoning
ReQuest (ECCV 2026)✓ (Sep. 2026)①Answer entropy✗✗✓✗✗
VQOS (ICLR 2026)✗②No early gate✓✗ (max similarity)✗✗✗
VideoHV-Agent (CVPR 2026)✗③Semantic hypotheses✗✗ (verbal clue)✗✗✓
HypoKFS (Ours)—④Answer-token distribution✓✓✓✓✓

Insights (①–④): ① ReQuest: missing evidence and unreliable relevance motivate trained selection and rethinking. ② VQOS: answer options refine video–text relevance. ③ VideoHV-Agent: shared verbal clues support semantic verification. ④ HypoKFS: initial answer competition motivates probability-weighted, discrimination-driven retrieval.

Published LLaVA-Video-7B accuracy (%) with at most 64 frames per inference:

MethodVideo-MMELongVideoBench
Uniform (our evaluation)64.458.9
VQOS (published, Ours-PO)65.561.0
HypoKFS (ours)67.964.8

VQOS values are from its ICLR 2026 Table 1 (provided options); Uniform and HypoKFS results are from our submission. Uniform uses 64 frames per video, while HypoKFS uses at most 32 frames per inference round, processing an average of 48.8 frames per video in total on Video-MME.

Q2. End-to-End Inference Time on Video-MME

The Qwen3-VL-8B / Video-MME timings on identical hardware include all inference passes and frame encoding/matching:

MethodInitial answer attemptFrame encoding & matchingSubsequent model passesTotalAccuracy
HypoKFS32 frames (3.2 h)1.3% (1.4 h)16.8 frames (2.2 h)6.8 h67.9%
AKS64 frames (6.6 h)3.7% (4.0 h)None10.6 h65.1%

HypoKFS is 35.9% faster (6.8 vs. 10.6 h).

Q3. Early-Exit Reliability and Recovery

Qwen3-VL-8B / Video-MME reliability statistics:

DiagnosticCorrect / totalAccuracy
Confident direct answers1,644 / 1,97183.2%
Routed queries, initial inference238 / 72932.6%
Routed queries, after renewed inference324 / 72944.4%

Renewed inference increases correct routed answers from 238 to 324, an 11.8-percentage-point accuracy gain.

ReQuest Chronology and Additional Comparisons

ReQuest's formal ECCV 2026 publication in September falls within the ICLR 2027 guidelines' contemporaneous-work window, for which missing comparisons cannot independently justify rejection. HypoKFS also originated from an earlier conference submission; timestamped records can be provided confidentially to the AC if necessary. Our novelty claim nevertheless rests on the methodological differences above, not chronology.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Remaining Concerns

Dear Reviewer,

Thank you again for your valuable comments and for taking the time to review our work. We would greatly appreciate it if you could let us know whether our rebuttal has adequately addressed your concerns, or if there are any remaining issues that we could further clarify. We would be happy to provide additional explanations if needed.

Thank you for your time and consideration.

Reviewer_GqxgOpenReview ↗

Response to authors from Gqxg

Thank you for responding to my comments an and reporting more results to address my concerns. However, I asked you to replace HypoKFS frame scoring rule with VQOS rule while keeping the same model, encoder, retrieval rounds and frame budget. This would show if choosing frames that separate the answer options works better than choosing frames that match any possible answer. Comparing published results does not isolate this difference. Nevertheless, the contribution is clearer now and the overall results are good. I will thus update my score.

Reviewer_sWEPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors present HypoKFS, a training-free strategy for frame selection for video question-answering. The general approach is to generate a distribution across potential answers by first running a forward pass with uniformly sampled frames. Then the method uses an efficient search mechanism to find additionally relevant frames depending on the question text and answers. The authors show good performance across several long-video understanding datasets.

Critical strengths

The authors propose a well-motivated strategy for key-frame selection by iteratively searching for frames based on the answer probability distribution. This formulation is intuitive, relatively straightforward, and well motivated which should be seen as major strengths for this work.

Critical weaknesses

The paper currently omits several relevant related works including VideoAgent, ReQuest, and VideoHV-Agent that all have similar mechanisms for first reasoning about the video contents before attempting to retrieve additional frames. This omission is particularly relevant since these approaches are quite similar to the proposed method and there are no experiments comparing these similar methods to HypoKFS. Thus, in my opinion, this paper has insufficient positioning against prior related work and also incomplete baseline comparisons, making it difficult to understand what this strategy contributes beyond existing approaches.

Questions for authors

Could the authors clarify how HypoKFS differs from the other closely related works that do reasoning before frame selection (e.g., VideoAgent Wang et al., ECCV 2024; ReQuest Kim et al 2026 ECCV; and VideoHV-Agent CVPR 2026)? VideoAgent uses confidence levels for question answering by iteratively and selectively retrieving additional frames, ReQuest uses initial uncertainty for frame selection and continued inference, and VideoHV-Agent uses competing hypotheses to find evidence. Many similar aspects of these methods appear to also be present for HypoKFS, and the current manuscript would greatly benefit by explaining this directly.

2 comments
AuthorsOpenReview ↗

Novelty: Distinct Motivation and Methodology

We sincerely thank the reviewer for recognizing HypoKFS's motivation and highlighting relevant work. We agree that reasoning before retrieval is not itself nove, and we have sought to clarify how our central insight and methodology differ from prior approaches. We welcome any further questions or concerns and will respond promptly.

Motivation/Insight: ReQuest is motivated by insufficient initial evidence and the mismatch between frame–question similarity and actual answer contribution, leading to uncertainty-triggered rethinking. In contrast, we experimentally investigate what the initial MLLM predictions themselves reveal (Introduction, Fig. 1 and Fig. 3(b)). The correct answer already appears among the two highest-ranked hypotheses inferred from sparse uniform frames for 89.4% of Video-MME questions. This motivates using the answer distribution not merely to trigger retrieval, but to select frames containing the most useful and discriminative evidence by coupling hypothesis-pair importance with frame-level separability.

Methodology: We do not claim that combining initial decisions, frame scoring, evidence acquisition, and renewed inference constitutes novelty. Our key contributions are coupled frame scoring and numerical iterative evidence acquisition. For each candidate frame, we aggregate all competing hypothesis pairs into a single numerical score, Dk(ft)=∑i<jwikwjk∑u<vwukwvk∣st,i−st,j∣D_k(f_t)=\sum_{i<j}\frac{w_i^k w_j^k}{\sum_{u<v}w_u^k w_v^k}|s_{t,i}-s_{t,j}|, which couples the probability-derived importance of each hypothesis pair with how strongly that frame distinguishes the pair; explicit temporal diversity further complements this score. Although VideoHV-Agent also considers competing hypotheses, it first distills them into a shared verbal discriminative clue and relies on multi-agent retrieval and verification. HypoKFS instead directly scores each frame against multiple hypothesis pairs through a lightweight numerical objective, without generating intermediate verbal clues. Similarly, VideoAgent iterates primarily through LLM-driven sufficiency assessments and tool calls, rather than repeatedly exploiting a probability-weighted hypothesis space for frame selection. Crucially, HypoKFS operates at two iterative levels: within-round numerical acquisition updates hypothesis weights after each selected frame without LLM calls, while across-round renewed inference reconstructs the hypothesis space to focus subsequent retrieval on unresolved alternatives. The full-pipeline comparison is provided below.

Numerical iterative evidence acquisition refers to within-round scoring, frame selection, and acquisition-state updates without LLM/tool reasoning. Iterative reasoning refers to a recurrent evidence–inference loop that can revise predictions; a single rethinking pass is not counted.

MethodContemporaneousInsightInitial decisionOpen-ended QAFrame scoring: coupled hypothesis discriminationFrame scoring: temporal diversityNumerical iterative evidence acquisitionIterative reasoning
VideoAgent (ECCV 2024)✗①LLM sufficiency✓✗✗✗✓
ReQuest (ECCV 2026)✓ (Sep. 2026)②Answer entropy✗✗ (max similarity)✓✗✗
VideoHV-Agent (CVPR 2026)✗③Semantic hypotheses✗✗ (verbal clue)✗✗✓
HypoKFS (Ours)—④Answer-token distribution✓✓✓✓✓

Insights (①–④). ① VideoAgent uses LLM judgments of evidence sufficiency to request additional tool-based observations. ② ReQuest identifies missing decisive evidence and the mismatch between frame–question similarity and answer contribution, motivating uncertainty-triggered rethinking with a trained selector. ③ VideoHV-Agent addresses semantic drift through testable hypotheses and shared verbal discriminative clues. ④ HypoKFS experimentally shows that sparse-frame predictions expose informative competition among plausible answers, motivating probability-aware discriminative frame selection.

ReQuest Chronology and Additional Comparisons

ReQuest's formal ECCV 2026 publication in September falls within the ICLR 2027 guidelines' contemporaneous-work window, for which missing comparisons cannot independently justify rejection. HypoKFS also originated from an earlier conference submission; timestamped records can be provided confidentially to the AC if necessary. Our novelty claim nevertheless rests on the methodological differences above, not chronology.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Remaining Concerns

Dear Reviewer,

Thank you again for your valuable comments and for taking the time to review our work. We would greatly appreciate it if you could let us know whether our rebuttal has adequately addressed your concerns, or if there are any remaining issues that we could further clarify. We would be happy to provide additional explanations if needed.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.