acceptodds
Under review as a conference paper at ICLR 2027

SEEK: Preserving Sparse Evidence across Semantic Events for Long-Video Understanding

Abstract

Long-video understanding challenges Multimodal Large Language Models (MLLMs) to identify sparse yet decisive evidence from temporal contexts under limited visual budgets. Keyframe selection provides an efficient way to reduce the visual input while preserving task-relevant information. However, existing methods may miss sparse query-relevant evidence, over-select redundant high-relevance frames, or struggle to handle heterogeneous events with different durations, relevance distributions, and evidence densities, leading to less reliable event-level keyframe allocation. We propose SEEK, an event-centric keyframe selection framework for preserving sparse evidence across semantic events. SEEK first partitions a video into variable-length events based on local and contextual visual changes. It then evaluates each event by jointly considering peak relevance, top- mean relevance, and average relevance, and employs the resulting utility to guide event-wise keyframe allocation. Finally, relevance entropy is used within each event to adapt the candidate pool and balance query relevance, visual diversity, and temporal diversity. Experiments on LongVideoBench, Video-MME, and MLVU with LLaVA-Video-7B, InternVL3.5-8B, and Qwen3-VL-8B show that SEEK consistently outperforms representative frame selection methods, with improvements of – over uniform temporal sampling.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Long Video Question Answering, Keyframe Selection, Sparse Evidence Preservation, Event-Aware Sampling, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cXraOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SEEK is a training-free, query-aware keyframe selection framework for long-video question answering. It partitions 1-FPS candidate frames using local and contextual visual changes, allocates a frame budget across events using peak/top-fraction/mean BLIP relevance and a coverage prior, and performs entropy-adaptive relevance/diversity selection within each event. Evaluation covers three benchmarks and three MLLM backbones, with reported improvements of 2.6–6.7 percentage points over uniform sampling and 0.8–2.9 points over QCA. Component ablations, parameter and frame-budget studies, qualitative examples, and a frame-replacement analysis provide additional validation.

Critical strengths
  1. The framework addresses a concrete failure mode of global relevance ranking: a few high-scoring regions can consume the budget while evidence elsewhere is missed. Event-level allocation and within-event selection provide a coherent way to combine relevance with coverage without fine-tuning the downstream MLLM. Table 1 (p. 7) reports gains over QCA in all nine settings and the best result among the listed methods in eight, including comparisons with three methods reproduced by the authors. This breadth supports the potential practical value of the approach.

  2. The evaluation goes beyond a main accuracy table. Table 2 (p. 8) replaces event partitioning with fixed segments, replaces the sparse-evidence utility with mean relevance, and removes entropy adaptation, with the full method improving across all three benchmarks. Appendix A.6–A.8 (pp. 14–17) explores boundary thresholds, top-relevance ratios, and budgets of 32/64/128 frames. These analyses provide useful evidence for the design, and the query-independent partition can be reused across questions.

Critical weaknesses
  1. The central comparison is insufficiently specified to verify that the reported gains arise at a matched evaluation budget. Section 4.2 (p. 6) imports Uniform, QCA, and several other baseline results from QCA, while Appendix A.2 (p. 12) states that checkpoints and frame budgets match but does not give the Table 1 budget, exact evaluation splits, or the image preprocessing and inference settings needed to check that match. Section 4.1 only says that K follows each experiment; the 32/64/128 sweep in Appendix A.8 does not explicitly identify the main-table configuration. Moreover, Appendix A.6–A.7 selects the reported boundary and relevance-ratio settings by their average performance on the three named benchmarks without identifying a separate tuning split. These are unresolved protocol questions, rather than evidence that the comparison is actually unfair, but they matter for interpreting gains as small as 0.8 points and for independent reproduction.

  2. The specific claim of preserving sparse evidence across semantic events is supported less directly than the overall QA improvement. Proposition 1 (p. 5; proof on pp. 13–14) correctly bounds retained mass under the softmax of BLIP ITM scores, which need not equal retention of decisive answer evidence. The qualitative examples are suggestive, but the counterfactual study in Section 4.6/Figure 5 (p. 9) does not specify how alternative frames are chosen or compare the same intervention on another selector. Its accuracy drop from 64.3% to 61.1% after 75% replacement shows that the chosen frames are useful; it does not isolate sparse-evidence preservation from general relevance or coverage. A controlled explanation of that intervention and its relationship to the proposed mechanism would materially strengthen the principal interpretation.

Questions for authors
  1. What exact K, benchmark splits, visual preprocessing, and decoding settings produce Table 1, and how were these matched to the imported QCA results? Were the hyperparameters in Appendix A.6–A.7 chosen on examples disjoint from the reported evaluations? A clear protocol and a matched rerun of the principal QCA/uniform comparisons, where needed, would resolve the main assessment concern.

  2. How are the replacement frames in Figure 5 sampled, and are temporal coverage and the number of input frames held fixed? Could the authors compare the same intervention on QCA, or provide another controlled result that distinguishes sparse-evidence retention across events from a general improvement in relevance or coverage?

6 comments

Response to Evaluation Protocol and Hyperparameter Selection

We thank the reviewer for highlighting the importance of evaluation protocol transparency and reproducibility. We agree that the current manuscript should provide more explicit details regarding the frame budget, evaluation settings, baseline provenance, and hyperparameter selection procedure. We provide the following clarifications and additional experimental evidence.

1. Frame budget and evaluation protocol

We clarify that the total keyframe budget in Table 1 is fixed at 64 frames for all methods, including the results adopted from QCA and our reproduced results for WFS-SB, FOCUS, and KFC. The evaluation follows the official benchmark splits for LongVideoBench, Video-MME, and MLVU.

For our reproduced baselines and SEEK, input frames are resized to 334 × 334, without additional image preprocessing. We follow the inference configuration adopted in QCA, including the same prompt templates, temperature, and maximum output tokens. We also use the same MLLM checkpoints and evaluation examples when comparing the reproduced methods with SEEK.

The sources of the reported results are summarized below.

MethodsResult SourceFrame Budget
Uniform, Top-k, AKS, Q-Frame, OneClip-RAG, BOLT, FRAG, E-VRAG, QCAAdopted from QCA64
WFS-SB, FOCUS, KFCReproduced by us64
SEEK (Ours)Our implementation64

We distinguish the results adopted from QCA from those obtained through our implementation, as the former retain their original reported evaluation procedures. We will expand Sections 4.1–4.2 and Appendix A.2 to document the official evaluation split identifiers, sample counts, image preprocessing settings, model checkpoints, prompt templates, and decoding parameters.

2. Additional comparisons under matched frame budgets

To further examine the consistency of SEEK's improvements under different visual budgets, we additionally evaluate QCA and WFS-SB with 32 selected frames using InternVL3.5-8B on LongVideoBench, Video-MME, and MLVU. The results are presented alongside the main comparison using 64 frames.

BudgetMethodLVBVideo-MMEMLVU
32QCA60.562.268.3
32WFS-SB61.463.369.0
32SEEK61.663.869.2
64QCA63.563.971.3
64WFS-SB63.863.571.2
64SEEK64.364.872.5

Under the 32-frame budget, SEEK outperforms QCA by 1.1, 1.6, and 0.9 percentage points on LongVideoBench, Video-MME, and MLVU, respectively. Compared with WFS-SB, SEEK achieves improvements of 0.2, 0.5, and 0.2 percentage points. These improvements are consistent with the results under the 64-frame budget, where SEEK also achieves higher accuracy across all three benchmarks.

Furthermore, Appendix A.8 (Figure 7) already evaluates SEEK under 32, 64, and 128 frames across all three MLLM backbones and benchmarks. The results show that accuracy consistently improves as the available frame budget increases across all nine model–benchmark combinations.

3. Hyperparameter selection and sensitivity

We thank the reviewer for raising this concern. We clarify that all methods are evaluated using the official benchmark-defined evaluation splits of LongVideoBench, Video-MME, and MLVU. These splits are fixed and identical across the compared methods. We did not construct or selectively choose additional validation subsets to favor particular parameter configurations.

The sensitivity analyses in Appendices A.6 and A.7 evaluate the boundary percentile PbP_b and the top-relevance ratio ρ\rho, respectively, on these fixed benchmark splits while keeping the remaining settings unchanged. The results are summarized below.

ParameterSettingLVBVideo-MMEMLVUAvg.
PbP_b0.6563.864.271.766.6
0.7564.264.572.267.0
0.8564.364.872.567.2
0.9564.164.672.767.1
ρ\rho0.2564.264.672.267.0
0.3064.364.872.567.2
0.3564.364.772.367.1
0.4064.164.371.966.8

The default settings Pb=0.85P_b=0.85 and ρ=0.30\rho=0.30 achieve the highest average accuracy among the evaluated configurations. However, the performance remains similar across neighboring parameter values. Specifically, the average accuracy varies only from 67.0% to 67.2% for PbP_b between 0.75 and 0.95, and from 67.0% to 67.2% for ρ\rho between 0.25 and 0.35.

We acknowledge that these analyses are conducted on the official evaluation splits rather than on a separate held-out tuning set. Accordingly, they provide evidence of limited sensitivity to neighboring parameter configurations, rather than independent validation of the selected hyperparameters.

Response to Sparse-Evidence Preservation and Counterfactual Analysis

We thank the reviewer for this insightful comment. We agree that the relationship between sparse-evidence preservation and downstream QA performance deserves further clarification. To address this concern, we clarify the scope of Proposition 1 and extend the counterfactual analysis to two closely related methods, QCA and WFS-SB, under matched evaluation settings.

1. Scope of Proposition 1

Proposition 1 establishes theoretical lower and upper bounds on the relevance probability mass retained by a candidate pool, based on the softmax distribution of BLIP image--text matching scores. These bounds provide theoretical motivation for adapting candidate-pool size according to normalized relevance entropy.

We acknowledge that relevance probability mass is not equivalent to decisive answer evidence. Accordingly, Proposition 1 motivates the entropy-guided candidate retention mechanism but does not establish a formal guarantee of answer-evidence preservation.

2. Counterfactual comparison across different frame-selection methods

To further examine the contribution of selected keyframes, we extend the counterfactual analysis in Section 4.6 to QCA and WFS-SB, using LongVideoBench, InternVL3.5-8B, and a fixed keyframe budget of K = 64.

Following the progressive replacement setting, we evaluate each method by replacing 0%, 25%, 50%, and 75% of its selected keyframes with alternative frames. The resulting QA accuracies are presented below.

Method0%25%50%75%
QCA63.562.161.760.2
WFS-SB63.862.662.060.9
SEEK (Ours)64.363.763.061.1

SEEK achieves the highest accuracy across all four replacement ratios. At a 25% replacement ratio, SEEK obtains 63.7%, outperforming QCA and WFS-SB by 1.6 and 1.1 percentage points, respectively. At a 50% replacement ratio, SEEK maintains an accuracy of 63.0%, exceeding QCA and WFS-SB by 1.3 and 1.0 percentage points, respectively.

To further examine the sensitivity of each method to progressive replacement, we report the absolute accuracy reduction relative to its original selected frames.

Method25% Drop50% Drop75% Drop
QCA1.41.83.3
WFS-SB1.21.82.9
SEEK (Ours)0.61.33.2

At replacement ratios of 25% and 50%, SEEK exhibits smaller accuracy reductions than both QCA and WFS-SB. At 75%, SEEK retains the highest absolute accuracy, although WFS-SB exhibits a slightly smaller reduction relative to its original performance.

These results extend the original counterfactual analysis beyond a single selector and provide comparative evidence that SEEK maintains stronger downstream QA performance under progressive keyframe replacement.

Response to Question 1

We thank the reviewer for emphasizing the importance of a clearly specified and reproducible evaluation protocol. We agree that the main comparison should explicitly report the frame budget, benchmark splits, visual preprocessing, inference settings, and sources of baseline results. We provide the following clarifications and additional experimental evidence.

1. Evaluation protocol and baseline settings

We clarify that Table 1 uses a fixed keyframe budget of K = 64 for all methods. All evaluations follow the official benchmark-defined evaluation splits of LongVideoBench, Video-MME, and MLVU, with the multiple-choice subset used for MLVU. The evaluation settings are summarized below.

SettingConfiguration
Keyframe budgetK = 64 for all methods in Table 1
Candidate frame sampling1 FPS
Evaluation dataOfficial benchmark-defined evaluation splits
Visual preprocessingResize to 334 × 334, without additional custom preprocessing
MLLM backbonesLLaVA-Video-7B, InternVL3.5-8B, Qwen3-VL-8B
Inference configurationFollowing QCA, including the same prompt templates, temperature, and maximum output tokens
Reproduced baselinesWFS-SB, FOCUS, KFC
Imported resultsUniform, Top-k, AKS, Q-Frame, OneClip-RAG, BOLT, FRAG, E-VRAG, QCA

For our reproduced methods, we use the same evaluation examples, MLLM checkpoints, image preprocessing, inference configurations, and keyframe budgets as SEEK. For baselines whose results are adopted from QCA, we retain the published scores and explicitly distinguish them from our reproduced results. The imported results therefore share the reported frame budget and benchmark setting, while their exact implementation-level equivalence should be distinguished from that of methods evaluated within our controlled pipeline.

We will expand Sections 4.1–4.2 and Appendix A.2 to document the exact evaluation split identifiers, sample counts, model checkpoints, prompt templates, and decoding parameters.

2. Additional matched-budget comparisons

To provide further evidence under a controlled evaluation setting, we additionally evaluate QCA and WFS-SB using InternVL3.5-8B with a reduced frame budget of K = 32, alongside SEEK. The results are presented below together with the main K = 64 comparison.

BudgetMethodLVBVideo-MMEMLVU
32QCA60.562.268.3
32WFS-SB61.463.369.0
32SEEK61.663.869.2
64QCA63.563.971.3
64WFS-SB63.863.571.2
64SEEK64.364.872.5

Under K = 32, SEEK outperforms QCA by 1.1, 1.6, and 0.9 percentage points on LongVideoBench, Video-MME, and MLVU, respectively. It also exceeds WFS-SB by 0.2, 0.5, and 0.2 percentage points. These results provide additional evidence that SEEK maintains its advantage under a smaller visual budget.

We note that the K = 64 QCA results in Table 1 are adopted from the published QCA comparison, whereas the additional K = 32 comparison is evaluated under our controlled setting. We will make this distinction explicit in the revised manuscript.

3. Hyperparameter selection and sensitivity

Regarding the selection of the boundary percentile and top-relevance ratio, we clarify that the sensitivity analyses in Appendices A.6–A.7 were conducted on the fixed official benchmark evaluation splits. No separate held-out tuning split was used. The selected configurations were informed by the reported benchmark-level sensitivity results.

For transparency, we reproduce the average accuracies obtained using InternVL3.5-8B.

ParameterSettingAvg. Accuracy
Boundary percentile0.6566.6
0.7567.0
0.8567.2
0.9567.1
Top-relevance ratio0.2567.0
0.3067.2
0.3567.1
0.4066.8

The results show that the average accuracy varies only from 67.0% to 67.2% when the boundary percentile ranges from 0.75 to 0.95, and from 67.0% to 67.2% when the top-relevance ratio ranges from 0.25 to 0.35. Thus, performance remains similar across neighboring parameter configurations.

We acknowledge that sensitivity analysis conducted on the evaluation splits does not constitute independent held-out hyperparameter validation. Accordingly, we interpret these results as evidence of limited sensitivity within the tested ranges, rather than as an unbiased estimate of hyperparameter-selection performance.

Reviewer_JBoMOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SEEK, an event-centric keyframe selection framework for long-video understanding. It first partitions the video into variable-length semantic events, then performs event-wise frame budget allocation based on sparse-evidence utility and coverage, and finally conducts adaptive keyframe selection within each event by considering query relevance, visual diversity, and temporal diversity. Experiments on LongVideoBench, Video-MME, and MLVU with multiple MLLM backbones demonstrate competitive performance against recent keyframe selection methods.

Critical strengths

1. Clear Motivation and Problem Formulation. The paper clearly motivates the limitations of existing sampling strategies and effectively distinguishes SEEK from prior approaches. Figure 1 is particularly helpful in illustrating how existing methods may miss sparse evidence, select redundant high-relevance frames, or lead to unreliable event-wise allocation, thereby motivating the use of variable-length semantic events for keyframe selection.

2. Competitive Quantitative Performance. The paper provides competitive comparisons with recent keyframe selection methods. In particular, it includes QCA as a strong recent baseline and reproduces three additional methods under its evaluation setting. The results show that SEEK achieves competitive performance across multiple benchmarks and MLLM backbones.

Critical weaknesses

1. Limited Novelty. The paper presents a clear pipeline for long-video understanding, but the methodological novelty is not sufficiently distinguished from prior work. The overall hierarchical framework of event partitioning, event-wise budget allocation, and intra-event keyframe selection has already been explored in recent keyframe selection methods, and SEEK appears to mainly integrate several existing ideas, including semantic event partitioning, relevance-based scoring, adaptive budget allocation, and diversity-aware selection. The paper does not clearly establish what constitutes a substantial methodological advance within each stage beyond combining and adapting these existing techniques.

2. Insufficient Experimental Validation of Key Design Choices. The experimental analysis does not sufficiently disentangle or justify several key components in Stage 2 and Stage 3. In Stage 2, the Sparse-Evidence Utility combines multiple relevance statistics, while in Stage 3, several mechanisms in Adaptive Selection are evaluated only through grouped ablations, making it difficult to determine their individual contributions. In addition, some important design choices, such as the weighting of different relevance statistics, lack adequate justification or sensitivity analysis. This limits the empirical evidence supporting the necessity and robustness of the proposed components.

Questions for authors

1. Unclear Frame Budget and Evaluation Settings. Could the authors clarify the specific frame budget used for the evaluations in Table 1? Although the implementation details state that the total keyframe budget k follows the setting of each experiment, the exact budget for the main comparison is not explicitly specified. Since several results are adopted from QCA, one may infer a 64-frame budget, but this should be clearly stated. In particular, the frame budget used when reproducing WFS-SB, FOCUS, and KFC should also be reported to ensure a fair comparison. In addition, could the authors provide results under different frame budgets to assess whether SEEK’s advantage remains consistent across varying visual budgets?

2. Insufficiently Disentangled Ablation of Adaptive Selection. In Stage 3, could the authors clarify the individual contribution of each component? Table 2 jointly ablates query relevance and temporal diversity, while the contribution of visual diversity is not independently evaluated. Moreover, entropy adaptation appears to affect two distinct mechanisms: adaptive candidate-pool sizing and adaptive relevance–diversity balancing. However, the “w/o Entropy Adaptation” variant disables them jointly, making it difficult to determine the contribution of each mechanism. Could the authors provide more disentangled ablations for these components?

3. Insufficient Parameter Analysis of Sparse-Evidence Utility. Could the authors clarify how the weights 0.45/0.45/0.10 for peak relevance, top-k mean relevance, and mean relevance are determined? While the paper provides sensitivity analysis for the top-relevance ratio, the choice of these three aggregation weights does not appear to be justified or analyzed. Since these weights directly determine the Sparse-Evidence Utility, could the authors provide further justification or sensitivity analysis to demonstrate the robustness of this design?

6 comments

Response to Limited Novelty

We acknowledge that hierarchical frame-selection frameworks have been explored in previous studies. However, the main contribution of SEEK is not the hierarchical structure itself, but the explicit modeling of sparse evidence across heterogeneous events and the entropy-guided adaptation of within-event keyframe selection. We clarify the methodological distinctions and provide supporting theoretical and experimental evidence below.

1. Motivation and distinctions from existing methods

As illustrated in Figure 1, existing frame-selection strategies face several limitations. Uniform sampling may miss sparse query-relevant evidence, while global relevance ranking can repeatedly select temporally adjacent and visually redundant frames. Moreover, coarse temporal segmentation may divide coherent events or group unrelated content within the same segment, resulting in less reliable keyframe allocation. These limitations motivate SEEK to jointly consider event structure, sparse evidence, and within-event relevance distributions.

The methodological differences between SEEK and two closely related approaches, QCA and WFS-SB, are summarized below.

AspectQCAWFS-SBSEEK (Ours)
Temporal partitionFixed temporal segmentsQuery-similarity-based boundariesQuery-independent variable-length events based on local and contextual visual changes
Frame allocationQuery relevance and content variationClip-level allocationSparse-Evidence Utility with a coverage prior
Within-event selectionDiversity-aware selectionMaximal Marginal Relevance (MMR)Entropy-adaptive candidate pool and relevance--diversity weights
Relevance distribution modelingNo explicit entropy-based adaptationNo explicit entropy-based adaptationNormalized relevance entropy controls both candidate pool size and selection weights

2. Experimental evidence for individual methodological contributions

The existing ablation study in Table 2 evaluates different design choices using InternVL3.5-8B across all three benchmarks.

VariantLVBVideo-MMEMLVUAvg.
w/o Semantic Event Partition64.264.371.466.6
w/o Sparse-Evidence Utility63.164.471.366.3
w/o Rel. & Temp. Diversity63.964.672.066.8
w/o Entropy Adaptation64.064.272.266.8
SEEK (Full)64.364.872.567.2

Replacing Semantic Event Partition with fixed temporal segmentation decreases average accuracy by 0.6 percentage points. Replacing Sparse-Evidence Utility with mean relevance produces the largest average decrease of 0.9 percentage points. Removing the relevance and temporal diversity terms from within-event selection reduces average accuracy by 0.4 percentage points, while replacing entropy-dependent candidate pool sizes and relevance weights with fixed settings also causes a 0.4-point decrease.

To further examine the contribution of Sparse-Evidence Utility beyond a combination of existing relevance statistics, we additionally compare individual allocation utilities under matched keyframe budgets, keeping the remaining components unchanged.

Allocation UtilityLVBVideo-MMEMLVUAvg.
Mean relevance63.164.471.366.3
Peak relevance63.364.571.766.5
Top-k mean relevance63.764.972.166.9
Sparse-Evidence Utility64.364.872.567.2

The combined utility achieves the highest average accuracy, exceeding mean relevance, peak relevance, and top-k mean relevance by 0.9, 0.7, and 0.3 percentage points, respectively. Although top-k mean relevance achieves a slightly higher score on Video-MME, the proposed utility performs better on LongVideoBench and MLVU, indicating the benefit of jointly characterizing sparse and overall event relevance.

3. Theoretical motivation for entropy-guided adaptation

Importantly, Adaptive Selection is supported by Proposition 1 (Entropy Bounds on Candidate Retention) in Section 3.4, with its proof provided in Appendix A.4.

The proposition establishes both lower and upper bounds on the relevance probability mass retained by the top-ranked candidate pool as functions of the candidate pool size, event length, and normalized relevance entropy.

4. Performance improvements over closely related methods

The methodological differences are also reflected in the results across three MLLM backbones and three long-video benchmarks. The following table reports SEEK's accuracy improvements over QCA and WFS-SB in percentage points.

Backbonevs. QCA (LVB / Video-MME / MLVU)vs. WFS-SB (LVB / Video-MME / MLVU)
LLaVA-Video-7B+1.3 / +1.6 / +0.8+0.7 / +1.4 / -0.5
InternVL3.5-8B+0.8 / +0.9 / +1.2+0.5 / +1.3 / +1.3
Qwen3-VL-8B+2.9 / +1.6 / +1.2+3.8 / +2.3 / +1.6

Response to Insufficient Experimental Validation of Key Design Choices

We thank the reviewer for this constructive comment. We agree that further disentangling the contributions of individual mechanisms would provide stronger empirical support for the design of SEEK. To address this concern, we provide additional experiments on InternVL3.5-8B across LongVideoBench, Video-MME, and MLVU, focusing on the relevance statistics and weighting strategy in Stage 2, as well as the independent contributions of the two entropy-guided mechanisms in Stage 3.

1. Individual contributions of relevance statistics in Stage 2

The proposed Sparse-Evidence Utility combines peak relevance, top-k mean relevance, and mean relevance to characterize different aspects of event-level evidence. Peak relevance captures the strongest response, top-k mean relevance measures the high-relevance portion of an event, and mean relevance reflects its overall relevance.

We additionally compare the proposed utility against each individual statistic while keeping the event partition, within-event selection procedure, and total keyframe budget unchanged.

Allocation UtilityLVBVideo-MMEMLVUAvg.
Mean relevance63.164.471.366.3
Peak relevance63.364.571.766.5
Top-k mean relevance63.764.972.166.9
Sparse-Evidence Utility64.364.872.567.2

The proposed utility achieves the highest average accuracy of 67.2%, outperforming mean relevance, peak relevance, and top-k mean relevance by 0.9, 0.7, and 0.3 percentage points, respectively. These results support the benefit of combining different relevance statistics rather than relying on a single event-level relevance measure.

2. Sensitivity analysis of the relevance-statistic weights

To further justify the weighting strategy, we conduct additional experiments by varying the weights assigned to peak relevance, top-k mean relevance, and mean relevance, while keeping all other settings unchanged.

PeakTop-k MeanMeanLVBVideo-MMEMLVUAvg.
0.300.300.4063.564.070.966.1
0.500.500.0063.864.472.266.8
0.600.300.1064.164.272.166.8
0.450.450.1064.364.872.567.2

The default configuration achieves the highest average accuracy among the evaluated settings. Increasing the mean-relevance weight to 0.40 decreases the average accuracy from 67.2% to 66.1%, suggesting that excessive reliance on event-level averaging can weaken the contribution of sparse decisive evidence. Removing the mean-relevance term or assigning a larger weight to peak relevance also reduces the average accuracy to 66.8%.

3. Independent contributions of entropy-guided mechanisms in Stage 3

The original ablation in Table 2 evaluates entropy adaptation by simultaneously replacing the adaptive candidate pool size and relevance weight with fixed settings. To further disentangle their individual contributions, we conduct additional experiments that independently enable or disable these two mechanisms.

Candidate PoolRelevance WeightAvg. Accuracy
FixedFixed66.8
Entropy-adaptiveFixed66.9
FixedEntropy-adaptive67.0
Entropy-adaptiveEntropy-adaptive67.2

Compared with the fully fixed configuration, independently adapting the candidate pool size improves average accuracy by 0.1 percentage points, while independently adapting the relevance weight improves accuracy by 0.2 percentage points. Jointly adapting both mechanisms further increases average accuracy to 67.2%, exceeding either individual adaptation.

These results demonstrate that the two mechanisms make distinct contributions to Adaptive Selection. Entropy-guided candidate pool adaptation adjusts the range of frames considered for selection, whereas entropy-guided relevance-weight adaptation adjusts the balance between query relevance and visual and temporal diversity. Their joint application achieves the strongest performance.

Response to Question 1

We thank the reviewer for pointing out the need to clarify the frame budget and evaluation settings.

Frame budget in Table 1. We clarify that the total keyframe budget is fixed at K = 64 for all methods in Table 1, including the results adopted from QCA and our reproduced results for WFS-SB, FOCUS, and KFC. The same frame budget is applied across all three MLLM backbones and benchmarks to ensure a fair comparison. We will explicitly state this setting in the revised manuscript.

Evaluation under different frame budgets. The current appendix already includes a frame-budget analysis in Appendix A.8 (Figure 7), where SEEK is evaluated under K = 32, 64, and 128 across three MLLM backbones and three benchmarks. The results show that SEEK consistently achieves higher accuracy as the frame budget increases across all nine model--benchmark combinations.

To further examine whether SEEK maintains its advantage over competing methods under different visual budgets, we additionally evaluate QCA and WFS-SB under K = 32 using InternVL3.5-8B on LongVideoBench, Video-MME, and MLVU. The following table summarizes the results under K = 32 and K = 64.

BudgetMethodLVBVideo-MMEMLVUAvg.
32QCA60.562.268.363.7
32WFS-SB61.463.369.064.6
32SEEK61.663.869.264.9
64QCA63.563.971.366.2
64WFS-SB63.863.571.266.2
64SEEK64.364.872.567.2

Under K = 32, SEEK outperforms QCA by 1.1, 1.6, and 0.9 percentage points on LongVideoBench, Video-MME, and MLVU, respectively. Compared with WFS-SB, SEEK achieves improvements of 0.2, 0.5, and 0.2 percentage points, respectively. These improvements are consistent with the results under K = 64, where SEEK also outperforms both methods across all three benchmarks.

We will incorporate the additional comparisons and clarify the evaluation settings in the revised manuscript.

Reviewer_B2VQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses the challenge of selecting keyframes that preserve sparse, query-relevant evidence across long videos. It proposes an event-centric framework SEEK that combines variable-length event partitioning, sparse-evidence-aware frame allocation, and entropy-guided adaptive selection. Experiments are conducted on three long-video benchmarks using three MLLM backbones, with comparisons against existing frame-selection methods and component ablations. Results show consistent improvements over existing methods, particularly under limited keyframe budgets.

Critical strengths
  1. The event-centric design accounts for heterogeneous event durations and evidence distributions, addressing limitations of fixed temporal segmentation.
  2. The sparse-evidence utility and entropy-guided selection jointly balance query relevance, visual diversity, and temporal coverage.
  3. Extensive experiments across three benchmarks and three backbones demonstrate consistent improvements, supported by ablation and counterfactual analyses.
Critical weaknesses
  1. The novelty is limited, as the partition-then-allocate paradigm is already common in frame selection, and SEEK mainly combines existing relevance and diversity metrics with heuristic adaptations.
  2. Experimental validation is insufficient, with limited ablation studies across backbones and insufficient analysis of the proposed components' individual contributions.
  3. The evaluation lacks end-to-end latency comparisons with competing methods, leaving the practical efficiency gains unclear.
Questions for authors
  1. Can the authors demonstrate that the sparse-evidence utility provides consistent gains over simpler relevance-based allocation under matched keyframe budgets?
  2. How robust is SEEK to its hyperparameter choices and the quality of BLIP-based relevance scores across different benchmarks?
  3. Can the authors provide end-to-end latency comparisons with competing methods under matched keyframe budgets, including feature extraction and relevance-scoring overhead?
5 comments
  1. Novelty and Distinctions from Existing Methods

We appreciate the reviewer's concern regarding novelty. While the general partition-then-select paradigm has been explored in previous studies, the novelty of SEEK lies in how sparse evidence is modeled across events and how keyframe selection is adapted to different within-event relevance distributions.

As illustrated in Figure 1, existing frame selection strategies exhibit different limitations. Uniform sampling may miss sparse query-relevant evidence, while global relevance ranking may repeatedly select redundant high-relevance frames. Moreover, coarse temporal segmentation may group distinct events under a shared local keyframe budget, resulting in less reliable event-wise allocation. These limitations motivate SEEK to construct variable-length events, allocate keyframes according to sparse-evidence utility, and adapt within-event selection to the relevance distribution.

Compared with QCA, which partitions videos into fixed temporal segments and allocates frames based on query relevance and content variation, SEEK constructs query-independent variable-length events based on local and contextual visual changes. This allows the event partition to better accommodate heterogeneous video content and to be precomputed and reused across different queries.

Compared with WFS-SB, which identifies event boundaries from query--frame similarity signals and performs Maximal Marginal Relevance (MMR)-based selection within the resulting clips, SEEK explicitly models event-level sparse evidence using peak relevance, top-(k) mean relevance, and mean relevance. This prevents the contribution of a few decisive frames from being diluted by lower-relevance frames during event-wise allocation. Furthermore, SEEK uses normalized relevance entropy to jointly adapt the candidate pool size and the relevance--diversity balance within each event, rather than applying a fixed selection strategy across different relevance distributions.

Importantly, the proposed entropy-guided adaptation is supported by Proposition 1 (Entropy Bounds on Candidate Retention), which establishes theoretical lower and upper bounds on the relevance mass retained by the candidate pool as a function of its size and normalized relevance entropy. Specifically, for a fixed event length and candidate pool size, lower entropy yields a stronger lower bound on retained relevance mass. As relevance becomes more dispersed, a larger candidate pool is motivated to preserve the distributed evidence.

These differences are also reflected in the experimental results. SEEK outperforms QCA across all nine model--benchmark combinations by 0.8--2.9 percentage points. Compared with WFS-SB, SEEK achieves higher accuracy in eight of the nine settings, with improvements of up to 3.8 percentage points, while performing only 0.5 percentage points lower in the remaining setting. These results demonstrate the effectiveness of sparse-evidence-aware allocation and entropy-guided adaptive selection beyond the general partition-then-select paradigm.

  1. Response to Question 1

We thank the reviewer for this question. The motivation for Sparse-Evidence Utility is that different relevance statistics capture different aspects of event-level evidence. Mean relevance reflects the overall relevance of an event but can dilute sparse decisive evidence when only a few frames are strongly related to the query. Peak relevance preserves the strongest evidence but depends on a single frame and can be sensitive to isolated high-relevance responses. Top-k mean relevance provides a more stable estimate of the high-relevance portion of an event, but does not reflect the overall event relevance. We therefore combine peak relevance, top-k mean relevance, and mean relevance to obtain a more robust event-level utility.

To isolate this design, we compare different relevance-based allocation strategies under the same keyframe budget while keeping the event partition and within-event selection procedure unchanged.

Allocation UtilityLongVideoBenchVideo-MMEMLVUAvg.
Mean relevance63.164.471.366.3
Peak relevance63.364.571.766.5
Top-kk mean relevance63.764.972.166.9
Sparse-Evidence Utility64.364.872.567.2

As already shown in Table 2, replacing Sparse-Evidence Utility with mean relevance decreases the average accuracy from 67.2% to 66.3%. The additional comparisons further show that peak relevance and top-k mean relevance achieve average accuracies of 66.5% and 66.9%, respectively, both below the 67.2% achieved by the full utility. The proposed utility outperforms mean and peak relevance on all three benchmarks, and improves over top-k mean relevance on LongVideoBench and MLVU while being only 0.1 point lower on Video-MME. Overall, the combined utility achieves the best average performance, indicating that jointly modeling the strongest evidence, the high-relevance portion, and the overall event relevance provides a more reliable criterion for event-wise keyframe allocation than using any single relevance statistic.

  1. Response to Question 2

We thank the reviewer for this question. We evaluate robustness from two perspectives: sensitivity to key hyperparameters and dependence on the relevance estimator.

For hyperparameter sensitivity, the current appendix already analyzes two key parameters controlling different stages of SEEK. For Semantic Event Partition, we vary the boundary percentile PbP_b from 0.65 to 0.95; for Sparse-Evidence Utility, we vary the top-relevance ratio ρ\rho from 0.25 to 0.40. The results are reported on all three benchmarks. In particular, the average accuracy varies only from 67.0 to 67.2 when Pb∈[0.75,0.95]P_b\in[0.75,0.95], and from 67.0 to 67.2 when ρ∈[0.25,0.35]\rho\in[0.25,0.35], indicating that SEEK is stable within reasonably broad parameter ranges.

To further examine whether SEEK depends on the specific BLIP-based relevance estimator, we additionally replace BLIP-large with CLIP on the InternVL3.5-8B backbone, while keeping the remaining pipeline and keyframe budget unchanged.

Relevance ScorerLongVideoBenchVideo-MMEMLVUAvg.
CLIP64.164.271.666.6
BLIP-large64.364.872.567.2

The results above are obtained with InternVL3.5-8B on all three benchmarks. Although BLIP-large provides the strongest overall performance, replacing it with CLIP causes only a moderate degradation. More importantly, the CLIP-based SEEK still outperforms QCA on all three benchmarks by 0.6, 0.3, and 0.3 percentage points, respectively. This suggests that the effectiveness of SEEK does not depend on a particular relevance estimator, although higher-quality relevance scores can further improve performance.

In the revised manuscript, we will further expand the sensitivity analysis to cover the remaining parameters.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revision Summary

Revision Summary

We sincerely thank all reviewers for their constructive feedback and valuable suggestions. In response to their comments, we have conducted additional experiments and provided further methodological and experimental clarifications. The main updates are summarized below.

1. Clarification of methodological contributions. We further distinguish SEEK from closely related approaches, particularly QCA and WFS-SB, by explaining the roles of query-independent variable-length event partitioning, sparse-evidence-aware allocation, and entropy-guided adaptive selection. We also clarify the motivation illustrated in Figure 1 and the theoretical scope of Proposition 1, which establishes bounds on retained relevance mass rather than a formal guarantee of decisive answer-evidence retention.

2. Additional component-level ablations. We extend the evaluation of Sparse-Evidence Utility by comparing mean relevance, peak relevance, and top-k mean relevance, achieving average accuracies of 66.3%, 66.5%, and 66.9%, respectively, compared with 67.2% for SEEK. We also provide a sensitivity analysis of the aggregation weights. For Adaptive Selection, we independently evaluate visual diversity, entropy-guided candidate-pool adaptation, and entropy-guided relevance-weight adaptation, providing more detailed evidence for their individual contributions.

3. Clarification of evaluation settings and frame-budget robustness. We explicitly clarify that the main comparisons use K = 64, follow the official benchmark evaluation splits, and adopt QCA-aligned inference settings. We distinguish imported baseline results from reproduced results and provide further preprocessing details. Additionally, we evaluate QCA, WFS-SB, and SEEK under K = 32, where SEEK maintains higher accuracy across LongVideoBench, Video-MME, and MLVU. We also clarify the parameter-selection procedure and the interpretation of the sensitivity analyses.

4. Additional robustness and efficiency analyses. We evaluate an alternative CLIP-based relevance scorer and examine the sensitivity of key design parameters. We further report keyframe-selection latency for QCA, WFS-SB, FOCUS, KFC, and SEEK under a matched 64-frame budget on Video-MME. SEEK completes selection in 432 ms per sample, with shared feature extraction and relevance scoring treated separately.

5. Extended counterfactual evaluation. We extend the progressive keyframe-replacement analysis to QCA and WFS-SB under the same evaluation setting. SEEK maintains the highest QA accuracy across replacement ratios of 0%, 25%, 50%, and 75%, and exhibits smaller accuracy reductions at 25% and 50% replacement. We further clarify the interpretation and limitations of the counterfactual analysis in relation to sparse-evidence preservation.

We will incorporate the relevant analyses and implementation details into the revised manuscript.

Dear Reviewers and Area Chair,

Thank you for your time and careful consideration. We sincerely apologize for the delay in preparing our responses; we have run additional experiments to thoroughly address your comments and verify the results. We would greatly appreciate it if you could review our rebuttal when convenient, and we welcome any further feedback.

Thank you again.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.