- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SEEK is a training-free, query-aware keyframe selection framework for long-video question answering. It partitions 1-FPS candidate frames using local and contextual visual changes, allocates a frame budget across events using peak/top-fraction/mean BLIP relevance and a coverage prior, and performs entropy-adaptive relevance/diversity selection within each event. Evaluation covers three benchmarks and three MLLM backbones, with reported improvements of 2.6–6.7 percentage points over uniform sampling and 0.8–2.9 points over QCA. Component ablations, parameter and frame-budget studies, qualitative examples, and a frame-replacement analysis provide additional validation.
Critical strengths
-
The framework addresses a concrete failure mode of global relevance ranking: a few high-scoring regions can consume the budget while evidence elsewhere is missed. Event-level allocation and within-event selection provide a coherent way to combine relevance with coverage without fine-tuning the downstream MLLM. Table 1 (p. 7) reports gains over QCA in all nine settings and the best result among the listed methods in eight, including comparisons with three methods reproduced by the authors. This breadth supports the potential practical value of the approach.
-
The evaluation goes beyond a main accuracy table. Table 2 (p. 8) replaces event partitioning with fixed segments, replaces the sparse-evidence utility with mean relevance, and removes entropy adaptation, with the full method improving across all three benchmarks. Appendix A.6–A.8 (pp. 14–17) explores boundary thresholds, top-relevance ratios, and budgets of 32/64/128 frames. These analyses provide useful evidence for the design, and the query-independent partition can be reused across questions.
Critical weaknesses
-
The central comparison is insufficiently specified to verify that the reported gains arise at a matched evaluation budget. Section 4.2 (p. 6) imports Uniform, QCA, and several other baseline results from QCA, while Appendix A.2 (p. 12) states that checkpoints and frame budgets match but does not give the Table 1 budget, exact evaluation splits, or the image preprocessing and inference settings needed to check that match. Section 4.1 only says that K follows each experiment; the 32/64/128 sweep in Appendix A.8 does not explicitly identify the main-table configuration. Moreover, Appendix A.6–A.7 selects the reported boundary and relevance-ratio settings by their average performance on the three named benchmarks without identifying a separate tuning split. These are unresolved protocol questions, rather than evidence that the comparison is actually unfair, but they matter for interpreting gains as small as 0.8 points and for independent reproduction.
-
The specific claim of preserving sparse evidence across semantic events is supported less directly than the overall QA improvement. Proposition 1 (p. 5; proof on pp. 13–14) correctly bounds retained mass under the softmax of BLIP ITM scores, which need not equal retention of decisive answer evidence. The qualitative examples are suggestive, but the counterfactual study in Section 4.6/Figure 5 (p. 9) does not specify how alternative frames are chosen or compare the same intervention on another selector. Its accuracy drop from 64.3% to 61.1% after 75% replacement shows that the chosen frames are useful; it does not isolate sparse-evidence preservation from general relevance or coverage. A controlled explanation of that intervention and its relationship to the proposed mechanism would materially strengthen the principal interpretation.
Questions for authors
-
What exact K, benchmark splits, visual preprocessing, and decoding settings produce Table 1, and how were these matched to the imported QCA results? Were the hyperparameters in Appendix A.6–A.7 chosen on examples disjoint from the reported evaluations? A clear protocol and a matched rerun of the principal QCA/uniform comparisons, where needed, would resolve the main assessment concern.
-
How are the replacement frames in Figure 5 sampled, and are temporal coverage and the number of input frames held fixed? Could the authors compare the same intervention on QCA, or provide another controlled result that distinguishes sparse-evidence retention across events from a general improvement in relevance or coverage?
6 comments
Response to Sparse-Evidence Preservation and Counterfactual Analysis
We thank the reviewer for this insightful comment. We agree that the relationship between sparse-evidence preservation and downstream QA performance deserves further clarification. To address this concern, we clarify the scope of Proposition 1 and extend the counterfactual analysis to two closely related methods, QCA and WFS-SB, under matched evaluation settings.
1. Scope of Proposition 1
Proposition 1 establishes theoretical lower and upper bounds on the relevance probability mass retained by a candidate pool, based on the softmax distribution of BLIP image--text matching scores. These bounds provide theoretical motivation for adapting candidate-pool size according to normalized relevance entropy.
We acknowledge that relevance probability mass is not equivalent to decisive answer evidence. Accordingly, Proposition 1 motivates the entropy-guided candidate retention mechanism but does not establish a formal guarantee of answer-evidence preservation.
2. Counterfactual comparison across different frame-selection methods
To further examine the contribution of selected keyframes, we extend the counterfactual analysis in Section 4.6 to QCA and WFS-SB, using LongVideoBench, InternVL3.5-8B, and a fixed keyframe budget of K = 64.
Following the progressive replacement setting, we evaluate each method by replacing 0%, 25%, 50%, and 75% of its selected keyframes with alternative frames. The resulting QA accuracies are presented below.
| Method | 0% | 25% | 50% | 75% |
|---|---|---|---|---|
| QCA | 63.5 | 62.1 | 61.7 | 60.2 |
| WFS-SB | 63.8 | 62.6 | 62.0 | 60.9 |
| SEEK (Ours) | 64.3 | 63.7 | 63.0 | 61.1 |
SEEK achieves the highest accuracy across all four replacement ratios. At a 25% replacement ratio, SEEK obtains 63.7%, outperforming QCA and WFS-SB by 1.6 and 1.1 percentage points, respectively. At a 50% replacement ratio, SEEK maintains an accuracy of 63.0%, exceeding QCA and WFS-SB by 1.3 and 1.0 percentage points, respectively.
To further examine the sensitivity of each method to progressive replacement, we report the absolute accuracy reduction relative to its original selected frames.
| Method | 25% Drop | 50% Drop | 75% Drop |
|---|---|---|---|
| QCA | 1.4 | 1.8 | 3.3 |
| WFS-SB | 1.2 | 1.8 | 2.9 |
| SEEK (Ours) | 0.6 | 1.3 | 3.2 |
At replacement ratios of 25% and 50%, SEEK exhibits smaller accuracy reductions than both QCA and WFS-SB. At 75%, SEEK retains the highest absolute accuracy, although WFS-SB exhibits a slightly smaller reduction relative to its original performance.
These results extend the original counterfactual analysis beyond a single selector and provide comparative evidence that SEEK maintains stronger downstream QA performance under progressive keyframe replacement.
Response to Question 1
We thank the reviewer for emphasizing the importance of a clearly specified and reproducible evaluation protocol. We agree that the main comparison should explicitly report the frame budget, benchmark splits, visual preprocessing, inference settings, and sources of baseline results. We provide the following clarifications and additional experimental evidence.
1. Evaluation protocol and baseline settings
We clarify that Table 1 uses a fixed keyframe budget of K = 64 for all methods. All evaluations follow the official benchmark-defined evaluation splits of LongVideoBench, Video-MME, and MLVU, with the multiple-choice subset used for MLVU. The evaluation settings are summarized below.
| Setting | Configuration |
|---|---|
| Keyframe budget | K = 64 for all methods in Table 1 |
| Candidate frame sampling | 1 FPS |
| Evaluation data | Official benchmark-defined evaluation splits |
| Visual preprocessing | Resize to 334 × 334, without additional custom preprocessing |
| MLLM backbones | LLaVA-Video-7B, InternVL3.5-8B, Qwen3-VL-8B |
| Inference configuration | Following QCA, including the same prompt templates, temperature, and maximum output tokens |
| Reproduced baselines | WFS-SB, FOCUS, KFC |
| Imported results | Uniform, Top-k, AKS, Q-Frame, OneClip-RAG, BOLT, FRAG, E-VRAG, QCA |
For our reproduced methods, we use the same evaluation examples, MLLM checkpoints, image preprocessing, inference configurations, and keyframe budgets as SEEK. For baselines whose results are adopted from QCA, we retain the published scores and explicitly distinguish them from our reproduced results. The imported results therefore share the reported frame budget and benchmark setting, while their exact implementation-level equivalence should be distinguished from that of methods evaluated within our controlled pipeline.
We will expand Sections 4.1–4.2 and Appendix A.2 to document the exact evaluation split identifiers, sample counts, model checkpoints, prompt templates, and decoding parameters.
2. Additional matched-budget comparisons
To provide further evidence under a controlled evaluation setting, we additionally evaluate QCA and WFS-SB using InternVL3.5-8B with a reduced frame budget of K = 32, alongside SEEK. The results are presented below together with the main K = 64 comparison.
| Budget | Method | LVB | Video-MME | MLVU |
|---|---|---|---|---|
| 32 | QCA | 60.5 | 62.2 | 68.3 |
| 32 | WFS-SB | 61.4 | 63.3 | 69.0 |
| 32 | SEEK | 61.6 | 63.8 | 69.2 |
| 64 | QCA | 63.5 | 63.9 | 71.3 |
| 64 | WFS-SB | 63.8 | 63.5 | 71.2 |
| 64 | SEEK | 64.3 | 64.8 | 72.5 |
Under K = 32, SEEK outperforms QCA by 1.1, 1.6, and 0.9 percentage points on LongVideoBench, Video-MME, and MLVU, respectively. It also exceeds WFS-SB by 0.2, 0.5, and 0.2 percentage points. These results provide additional evidence that SEEK maintains its advantage under a smaller visual budget.
We note that the K = 64 QCA results in Table 1 are adopted from the published QCA comparison, whereas the additional K = 32 comparison is evaluated under our controlled setting. We will make this distinction explicit in the revised manuscript.
3. Hyperparameter selection and sensitivity
Regarding the selection of the boundary percentile and top-relevance ratio, we clarify that the sensitivity analyses in Appendices A.6–A.7 were conducted on the fixed official benchmark evaluation splits. No separate held-out tuning split was used. The selected configurations were informed by the reported benchmark-level sensitivity results.
For transparency, we reproduce the average accuracies obtained using InternVL3.5-8B.
| Parameter | Setting | Avg. Accuracy |
|---|---|---|
| Boundary percentile | 0.65 | 66.6 |
| 0.75 | 67.0 | |
| 0.85 | 67.2 | |
| 0.95 | 67.1 | |
| Top-relevance ratio | 0.25 | 67.0 |
| 0.30 | 67.2 | |
| 0.35 | 67.1 | |
| 0.40 | 66.8 |
The results show that the average accuracy varies only from 67.0% to 67.2% when the boundary percentile ranges from 0.75 to 0.95, and from 67.0% to 67.2% when the top-relevance ratio ranges from 0.25 to 0.35. Thus, performance remains similar across neighboring parameter configurations.
We acknowledge that sensitivity analysis conducted on the evaluation splits does not constitute independent held-out hyperparameter validation. Accordingly, we interpret these results as evidence of limited sensitivity within the tested ranges, rather than as an unbiased estimate of hyperparameter-selection performance.
Response to Evaluation Protocol and Hyperparameter Selection
We thank the reviewer for highlighting the importance of evaluation protocol transparency and reproducibility. We agree that the current manuscript should provide more explicit details regarding the frame budget, evaluation settings, baseline provenance, and hyperparameter selection procedure. We provide the following clarifications and additional experimental evidence.
1. Frame budget and evaluation protocol
We clarify that the total keyframe budget in Table 1 is fixed at 64 frames for all methods, including the results adopted from QCA and our reproduced results for WFS-SB, FOCUS, and KFC. The evaluation follows the official benchmark splits for LongVideoBench, Video-MME, and MLVU.
For our reproduced baselines and SEEK, input frames are resized to 334 × 334, without additional image preprocessing. We follow the inference configuration adopted in QCA, including the same prompt templates, temperature, and maximum output tokens. We also use the same MLLM checkpoints and evaluation examples when comparing the reproduced methods with SEEK.
The sources of the reported results are summarized below.
We distinguish the results adopted from QCA from those obtained through our implementation, as the former retain their original reported evaluation procedures. We will expand Sections 4.1–4.2 and Appendix A.2 to document the official evaluation split identifiers, sample counts, image preprocessing settings, model checkpoints, prompt templates, and decoding parameters.
2. Additional comparisons under matched frame budgets
To further examine the consistency of SEEK's improvements under different visual budgets, we additionally evaluate QCA and WFS-SB with 32 selected frames using InternVL3.5-8B on LongVideoBench, Video-MME, and MLVU. The results are presented alongside the main comparison using 64 frames.
Under the 32-frame budget, SEEK outperforms QCA by 1.1, 1.6, and 0.9 percentage points on LongVideoBench, Video-MME, and MLVU, respectively. Compared with WFS-SB, SEEK achieves improvements of 0.2, 0.5, and 0.2 percentage points. These improvements are consistent with the results under the 64-frame budget, where SEEK also achieves higher accuracy across all three benchmarks.
Furthermore, Appendix A.8 (Figure 7) already evaluates SEEK under 32, 64, and 128 frames across all three MLLM backbones and benchmarks. The results show that accuracy consistently improves as the available frame budget increases across all nine model–benchmark combinations.
3. Hyperparameter selection and sensitivity
We thank the reviewer for raising this concern. We clarify that all methods are evaluated using the official benchmark-defined evaluation splits of LongVideoBench, Video-MME, and MLVU. These splits are fixed and identical across the compared methods. We did not construct or selectively choose additional validation subsets to favor particular parameter configurations.
The sensitivity analyses in Appendices A.6 and A.7 evaluate the boundary percentile Pb and the top-relevance ratio ρ, respectively, on these fixed benchmark splits while keeping the remaining settings unchanged. The results are summarized below.
The default settings Pb=0.85 and ρ=0.30 achieve the highest average accuracy among the evaluated configurations. However, the performance remains similar across neighboring parameter values. Specifically, the average accuracy varies only from 67.0% to 67.2% for Pb between 0.75 and 0.95, and from 67.0% to 67.2% for ρ between 0.25 and 0.35.
We acknowledge that these analyses are conducted on the official evaluation splits rather than on a separate held-out tuning set. Accordingly, they provide evidence of limited sensitivity to neighboring parameter configurations, rather than independent validation of the selected hyperparameters.