- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes HypoKFS, a training-free framework for long-video question answering that integrates keyframe selection with iterative inference. It first constructs an answer distribution from sparsely sampled frames and selectively retrieves additional evidence for ambiguous queries. The retrieved frames are scored using hypothesis-conditioned CLIP similarity, discriminative utility, and temporal diversity. Experiments on Video-MME, LongVideoBench, and MLVU demonstrate improvements over existing keyframe selection methods with reduced image–text matching computation.
Critical strengths
-
Reasoning-guided frame selection. Integrating answer uncertainty into keyframe retrieval is a practical approach that can avoid unnecessary retrieval and prioritize discriminative visual evidence.
-
Consistent empirical improvements. The method achieves promising accuracy gains across three long-video benchmarks and two open-source MLLM backbones, with component ablations supporting the effectiveness of its design.
Critical weaknesses
-
Insufficient comparisons with agent-based methods. HypoKFS is essentially an iterative video reasoning and retrieval framework, but the experiments mainly compare against conventional keyframe selection methods. Closely related training-free agentic approaches, such as VideoAgent, VideoTree, DrVideo, VideoAgent2, and VideoChat-A1, are missing. This makes it difficult to assess the novelty and benefits of the proposed iterative reasoning strategy.
-
Efficiency and generalizability are insufficiently validated. The reported efficiency mainly concerns image–text matching cost rather than end-to-end latency. Multiple MLLM inference rounds may introduce substantial additional overhead. Moreover, HypoKFS is evaluated only with open-source backbones and a fixed CLIP matching encoder, leaving its claimed plug-and-play generalizability insufficiently supported.
Questions for authors
Q1. Comparisons with training-free video agents. Could the authors compare HypoKFS with representative agent-based methods, such as VideoTree, VideoAgent2, and VideoChat-A1, under comparable backbone and computation settings? These methods also perform iterative evidence acquisition and reasoning.
Q2. End-to-end inference efficiency. Could the authors report end-to-end inference latency against existing methods, including CLIP matching, video processing, and all MLLM inference rounds? The reduced matching cost does not necessarily translate into lower overall latency.
Q3. Generalization to proprietary MLLMs. Since HypoKFS is presented as a training-free, plug-and-play framework, could the authors evaluate it with at least one closed-source MLLM? How would the hypothesis distribution be obtained when candidate-token logits are unavailable through the API?
Q4. Dependence on CLIP. The proposed frame selection relies on CLIP ViT-B/32 to measure frame–hypothesis similarity, which may limit its ability to capture complex visual semantics and temporal evidence. Could the authors investigate using a lightweight MLLM for frame–hypothesis relevance scoring instead of CLIP? How would this affect selection accuracy and inference efficiency?
Q5. Generalization to open-ended VideoQA. All evaluations are conducted on multiple-choice benchmarks, where candidate hypotheses are predefined and their probabilities can be directly extracted from answer-option logits. Can HypoKFS be extended to open-ended VideoQA without predefined answer candidates? How would the authors construct and update the hypothesis space, estimate confidence, and perform hypothesis-guided frame selection in this setting? An evaluation on an open-ended VideoQA benchmark would be important to validate the general applicability of the framework.
2 comments
Follow-up on Our Rebuttal and Remaining Concerns
Dear Reviewer,
Thank you again for your valuable comments and for taking the time to review our work. We would greatly appreciate it if you could let us know whether our rebuttal has adequately addressed your concerns, or if there are any remaining issues that we could further clarify. We would be happy to provide additional explanations if needed.
Thank you for your time and consideration.
Response to bF3m
Q1. Novelty and Comparison with Video Agents
We thank the reviewer for suggesting agent-based comparisons and welcome further questions. Our motivation differs from agentic retrieval: Fig. 3(b) shows that the correct answer already appears among the top two hypotheses inferred from sparse frames for 89.4% of Video-MME questions, suggesting that initial answer probabilities contain useful information about competing answers. Our contribution is coupled hypothesis-discriminative frame scoring and numerical iterative evidence acquisition, rather than agentic planning. The score Dk(ft)=∑i<j∑u<vwukwvkwikwjk∣st,i−st,j∣ couples hypothesis-pair importance with frame-level separability, complemented by temporal diversity. Whereas VideoAgent relies on LLM judgments of evidence sufficiency and VideoHV-Agent converts competing hypotheses into a shared verbal clue for agentic verification, HypoKFS scores each frame directly against multiple hypothesis pairs and updates acquisition weights after each selection without further LLM calls. Renewed inference then reconstructs the hypothesis space to focus on unresolved ambiguity.
To illustrate the cost of agentic retrieval, we compare with WorldMM-8B, a Qwen3-VL-8B-based agent using iterative multimodal memory retrieval, on Video-MME Long:
WorldMM's accuracy is from its paper; both latencies are from our measurements. WorldMM gains 2.1 accuracy points but takes 6.6× as long (60.3 vs. 9.1 s), illustrating the latency cost of agentic retrieval without implying superior HypoKFS accuracy.
Q2. End-to-End Inference Efficiency
We appreciate this important suggestion and will include the end-to-end analysis in the main paper. The Qwen3-VL-8B / Video-MME measurements below were obtained on identical H20-3e hardware and include initial inference, frame encoding/matching, and all subsequent model passes:
HypoKFS reduces total runtime by approximately 35.9% (6.8 vs. 10.6 h), while improving accuracy by 2.8 percentage points in this efficiency setting. This efficiency setting differs from Q1.
Q3. Generalization to Stronger and Proprietary MLLMs
Due to server VPN restrictions, we cannot reliably access overseas proprietary APIs. We therefore evaluate plug-and-play transfer using a strong domestically accessible model with available answer-token logits, Qwen3.6-27B, under a 128-frame Video-MME budget:
The 3.5-point improvement supports transfer to a stronger backbone without additional training. The method still requires candidate-token probabilities; this does not establish compatibility with proprietary text-only APIs.
Q4. Dependence on CLIP
We thank the reviewer for suggesting a multimodal scoring alternative. With Qwen3-VL-8B fixed as the answer backbone, we replace CLIP frame–hypothesis similarity with Qwen3-VL-4B prompted to rate each frame–hypothesis pair from 1 to 4, followed by normalization:
The MLLM scorer improves over uniform sampling but underperforms CLIP in this setting. One plausible limitation is that our heuristic acquisition parameters were calibrated for CLIP similarities and were not retuned for the 4B scorer's discrete rating distribution, which changes both pairwise score gaps and numerical weight updates. We will acknowledge this matcher-dependent calibration requirement.
Q5. Open-Ended VideoQA
We additionally evaluate the entire ActivityNet-QA dataset, which requires free-form answers rather than predefined multiple-choice options. HypoKFS generates candidate hypotheses from sparse initial evidence, estimates their relative support, and applies the same discriminative selection and numerical acquisition updates. With LLaVA-Video-7B, we obtain:
The 1.6-point improvement supports applicability to open-ended QA. The smaller gain may reflect shorter videos, with less need for targeted long-range retrieval.