acceptodds
Under review as a conference paper at ICLR 2027

Broad Coverage, Focused Resolution: Hypothesis-Discriminative Evidence Acquisition for Long Video Question Answering

10 views

Abstract

Long video question answering requires selecting evidence that determines the answer under a limited visual budget. Uniform sampling can miss brief events, while existing selection methods mainly organize evidence by query relevance and temporal coverage. This paper focuses on selecting a compact and complementary evidence set based on the distinctions among candidate answers. We propose HDEA, a two stage evidence acquisition framework driven by competition among answer candidates. Broad Competition Coverage considers all candidate pairs, selects complementary discriminative evidence, and builds an evidence core through temporally constrained submodular coverage. Focused Competition Resolution uses the core prediction to reweight candidate pairs, selects supplementary evidence while preserving the core, and fuses the predictions from the core and two nested expanded visual contexts. The two stages follow one principle: broad coverage before answering and focused resolution after answering. HDEA requires no additional training, subtitles, or external language models. On VideoMME, LongVideoBench, and MLVU, HDEA reaches 67.44%, 65.22%, and 74.25%, respectively, and achieves competitive performance among representative long video methods. Across four frozen video MLLMs and three benchmarks, HDEA demonstrates strong cross model transferability, outperforming uniform sampling in all 12 model and benchmark combinations. These results support candidate answer competition as a common structure for evidence acquisition and refinement in long video question answering. Code is available at https://anonymous.4open.science/r/HDEA.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
long video question answering, multimodal large language models, video understanding, frame selection, evidence acquisition

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VhS4OpenReview ↗
Rating
1(−2)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1(−1)
Read review
Paper summary

This paper studies how to select more effective video evidence under a limited visual budget for long-video multiple-choice question answering. The authors propose HDEA, which leverages the competition among candidate answers to guide evidence selection. In the first stage, the method selects a complementary set of video clips according to the discrimination needs of different candidate-answer pairs, forming an evidence core. In the second stage, it uses the candidate-answer probabilities obtained from the initial prediction to acquire more targeted supplementary evidence, and combines predictions obtained under different visual contexts. The method requires no additional training and is evaluated on three long-video QA benchmarks with multiple frozen video MLLMs.

Critical strengths
  • The paper organizes video evidence around the distinctions among candidate answers and further considers complementarity among different pieces of evidence. Compared with frame selection methods that rely mainly on question relevance, this design is better aligned with the decision-making structure of multiple-choice question answering.

  • Comparisons with variants such as OptionPool and PairwisePointwise help disentangle the contributions of candidate-answer information and set-level complementary selection. In particular, Broad Coverage substantially outperforms pointwise selection while using the same pairwise discrimination information, providing useful evidence for the importance of modeling complementarity at the set level.

Critical weaknesses
  • Stage 2 introduces additional supplementary evidence and extra answer-model inference, but the resulting performance gains are relatively limited. The current results do not yet clearly demonstrate how much of the improvement comes specifically from the competition-aware focused selection mechanism, rather than simply from using more video evidence and additional answer-model inference.
  • The full HDEA pipeline requires three answer-model inference calls, while the three-way prediction fusion provides almost no additional improvement over directly using E48. Therefore, the trade-off between the additional computational cost introduced by Stage 2 and its practical benefit is not yet sufficiently clear.
Questions for authors
  • Could the authors further clarify how much of the performance improvement in Stage 2 can be attributed specifically to competition-aware focused selection, as opposed to simply using more video evidence and additional answer-model inference?
  • The three-way prediction fusion in the full HDEA pipeline appears to provide almost no improvement over directly using E48. Could the authors further clarify the necessity and practical value of this fusion step?
13 comments
AuthorsOpenReview ↗

Response to Reviewer VhS4

We thank the reviewer for recognizing the decision aligned selection and the controlled support for complementarity. We address the two concerns with additional selection controls and a two-call default output for the revision.

All new controls use frozen Qwen3-VL-8B and the complete VideoMME / LongVideoBench / MLVU-Dev question sets: 2,700 / 1,337 / 2,174 questions, including fallback cases. MLVU is evaluated using M-AVG across seven task categories. Scores are percentages; gains are percentage points computed from unrounded scores. Mean is the descriptive, unweighted average of the three displayed benchmark scores.

Reply to W1 & Q1: Contribution of focused selection

Concern: Does Stage 2 help through competition aware selection, or mainly through additional evidence and inference?

Reply: With Qwen3-VL-8B, expanding Core32 to E48E_{48} improves accuracy by +0.96 / +0.52 / +0.74 pp across the three benchmarks. Across four answer models, it improves ten of twelve settings, ties one, and decreases one. Within Stage 2, competition weighting yields positive E48E_{48} differences over uniform pair weights on all three benchmarks (+0.19 / +0.37 / +0.51 pp). These results separate the overall expansion benefit from the smaller weighting contribution, whose confidence intervals include zero.

The controls fix the parent Core32, eight non-overlapping four frame groups outside the core, expansion budgets, prompt, visual processing, and option-token scoring. Uniform pair weights replace πaπb\pi_a\pi_b with a constant. With the groups numbered chronologically as A1 to A8, the new chronological-prefix control uses A1/A2 for E40E_{40} and A1/A2/A3/A4 for E48E_{48}.

Visual context / selection ruleVideoMMELongVideoBenchMLVU M-AVGMean
Core3266.5664.6273.6268.27
Competition ranked E4067.4465.0773.7768.76
Uniform pair E4867.3364.7773.8568.65
Chronological prefix E4867.4465.2274.4769.04
Competition ranked E4867.5265.1574.3569.01

For competition minus uniform pair weights at E48E_{48}, the 95% bootstrap confidence intervals clustered by video are [−0.52, 0.85] / [−0.53, 1.28] / [−0.40, 1.40]. These unadjusted intervals use 10,000 paired physical-video-cluster replicates. At E40E_{40}, the weighting differences are +0.44 / 0.00 / −0.06 pp. The chronological prefix control is also competitive, with slightly higher E48E_{48} scores on LongVideoBench and MLVU.

The new chronological prefix control supplements the original temporal control reported in Table F.3. All original results are retained, including the MLVU fusion accuracy of 74.77.

The existing comparison with single pass Broad Coverage 48 also gives +0.89 / +0.60 / −0.52 pp for core preserving E48E_{48} on the three benchmarks (original Table 3). We will distinguish these overall expansion results from the smaller contribution of posterior pair weighting. For the E48E_{48} comparisons, posterior free controls retain the same Stage 1 Broad Coverage core and 48-frame budget, but require only one answer model call; they match the evidence budget, not the inference cost.

AuthorsOpenReview ↗

Response to Reviewer VhS4

Reply to W2 & Q2: Practical value of fusion

Concern: Does combining three predictions justify a third call to the answering model?

Reply: We will use the direct E48E_{48} prediction as the default HDEA output in the revision. This retains both evidence acquisition stages with two calls to the answering model, while the original three-call fusion remains an optional configuration.

OutputNominal callsVideoMMELongVideoBenchMLVU M-AVGMean
Direct E48 prediction267.5265.1574.3569.01
Original three-call fusion367.4465.2274.2568.97

Fusion minus E48E_{48} is −0.07 / +0.07 / −0.10 pp for Qwen3-VL-8B. Across all twelve model and benchmark settings, E48E_{48} is higher in seven, tied in one, and lower in four, with an unweighted mean difference of +0.07 pp. The omitted call evaluates E40E_{40}; E48E_{48} construction depends on the core prediction, not the E40E_{40} prediction.

For Qwen3-VL-8B, archived cumulative answering stage times for E48E_{48} versus fusion are 9.95 vs. 14.63 / 10.16 vs. 14.64 / 14.29 vs. 21.42 seconds, corresponding to reductions of 32.0% / 30.6% / 33.3%. These sums retain actual repeated fallback evaluations and exclude SigLIP feature extraction and selector overhead.

The revised default therefore removes one answer model call and about 32% of the recorded answering stage time on the primary model. We will apply this output choice uniformly across the four models and three benchmarks, retain every original fusion result, and identify the default change as a revision during the review period.

  1. Regarding Stage 2, the additional experiments show that competition-aware focused selection provides very limited gains over uniform pair weighting, none of which reach statistical significance. More importantly, under the same visual budget, simply selecting the first four supplementary evidence groups in chronological order achieves comparable or even slightly better average performance at a lower inference cost. Although the authors report this result, they do not sufficiently explain why additional model inference is still necessary to obtain candidate-answer probabilities and use them to guide supplementary evidence selection. Therefore, my concern regarding the actual contribution of Stage 2 remains unresolved.

  2. In light of the experimental results for Stage 2, I have also reconsidered the design of Stage 1 and its supporting experimental evidence. In my initial review, the substantial performance improvements of Broad Coverage over PairwisePointwise were an important reason for my inclination toward acceptance. However, the newly added matched-fill control shows that these improvements become considerably smaller when the evidence-filling strategy is controlled. Furthermore, the simpler OptionMax32 achieves comparable performance without explicitly modeling candidate-pair discrimination or set-level complementarity. These results weaken the evidence supporting what I originally considered a critical strength of the paper and raise further concerns about the actual contribution of the core mechanism in Stage 1.

I appreciate the additional controlled experiments provided by the authors and hope they can further clarify the necessity and independent contributions of the core mechanisms in both stages in light of these new experimental results, thereby addressing the above concerns.

Reviewer_qcRiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HDEA is a training-free method for long-video multiple-choice question answering. It first selects a 32-frame evidence core using differences between candidate-answer matching scores. It then uses the initial answer probabilities to select additional evidence, expands the context to 40 and 48 frames, and combines the three predictions. Experiments cover three benchmarks and four frozen video-language models.

Critical strengths
  1. The first stage has useful controlled support. Table 2 shows that selecting complementary clips as a set outperforms independent ranking using the same pairwise scores, with the model, candidate clips, constraints, and frame budget fixed.

  2. The method requires no additional training and improves over uniform sampling across all twelve model–benchmark combinations. The authors report matched-budget controls, inference costs, and negative results.

Critical weaknesses
  1. The advantage over closely related methods remains unclear. Table 1 uses different answering models and frame budgets. In Table 2, the reported gains over OptionPool are 1.74, 0.45, and 0.38 points on VideoMME, LongVideoBench, and MLVU. Table D.1 provides paired prediction changes, but uncertainty estimates would help assess the smaller gains. Comparisons with established selectors under matched settings are still needed to establish the specific benefit of candidate-pair coverage.

  2. The second stage gives mixed results. On MLVU, competition-weighted ranking helps the 48-frame prediction, but temporal ranking gives better fused accuracy: 74.77 versus 74.25 (Table F.3). Fusion also underperforms the 48-frame prediction in seven of twelve settings, while requiring three model calls instead of two (Appendix F.1). The benefit of the proposed weighting and fusion therefore remains unclear beyond adding supplementary evidence.

Questions for authors
  1. Can Broad Coverage be compared with MDP³ or VQOS using the same answering model, candidate frames, and frame budget? A maximum question–option similarity baseline under the Table 2 setup would also help test its advantage over simpler option-aware scoring.

  2. With the core, candidate supplementary groups, budgets, and output rule fixed, how does competition-weighted ranking compare with uniform candidate-pair weights across all three benchmarks?

  3. What supports using three-prediction fusion as the default rather than returning the 48-frame prediction, given its additional cost and mixed accuracy results?

3 comments
AuthorsOpenReview ↗

Response to Reviewer qcRi

We thank the reviewer for recognizing the controlled support for complementary selection. We have completed the requested comparisons with MDP³, OptionMax, and uniform pair weights on all three full benchmarks.

New controls use frozen Qwen3-VL-8B on 2,700 VideoMME / 1,337 LongVideoBench / 2,174 MLVU-Dev questions, including predefined fallbacks. For MLVU, we report M-AVG across seven task categories. Scores are percentages; gains are percentage points computed from unrounded scores. Mean is the descriptive, unweighted average of the three displayed benchmark scores.

Reply to W1 & Q1: Matched selectors and uncertainty

Concern: Under a shared answerer, candidate catalog, and budget, does coverage across candidate pairs outperform simpler scoring based on answer options?

Reply: With a shared budget of 32 frames and one model call, Broad Coverage improves over AKS32 on all three benchmarks (+3.81 / +2.99 / +2.71 pp) and has the highest observed mean across the three benchmarks (68.27%) among the compared selectors. Its largest advantage over the MDP³ adaptation is on LongVideoBench, at +4.04 pp; individual benchmark leaders differ.

All selectors use the same saved catalog of candidate frames, a budget of 32 frames, the frozen answering model, its native visual processing, the same prompt, and the same option token scoring. OptionMax uses si=max⁡azi(ca)s_i=\max_a z_i(c_a) under Core's feasible domain. MDP³ retains the authors' kernel and dynamic programming procedure, default λ=0.2\lambda=0.2, and its own constraints. This is an adaptation using cached SigLIP features, with bfloat16 extraction, float16 storage, and float32 selection.

SelectorVideoMMELongVideoBenchMLVU M-AVGMean
Uniform3263.5958.2663.9561.93
PairwisePointwise3262.2256.3263.1360.56
AKS32, matched setting62.7461.6370.9165.09
MDP³-32, matched adaptation67.4160.5872.5066.83
OptionPool3264.8164.1773.2367.40
OptionMax3266.3763.7273.8967.99
Broad Coverage / HDEA-Core3266.5664.6273.6268.27

MDP³ has the highest VideoMME score; OptionMax is slightly higher on MLVU. We will base selector comparisons on this shared evaluation and present the table of previously published systems as context.

Core minus baseline [95% CI clustered by video], in percentage points:

BaselineVideoMMELongVideoBenchMLVU M-AVG
OptionPool32+1.74 [0.33, 3.11]+0.45 [−1.51, 2.43]+0.38 [−1.16, 1.92]
OptionMax32+0.19 [−1.15, 1.52]+0.90 [−1.10, 2.92]−0.28 [−1.83, 1.23]
MDP³-32, matched adaptation−0.85 [−2.33, 0.63]+4.04 [1.72, 6.34]+1.12 [−0.58, 2.79]

We also added a control with pointwise selection and matched relevance filling. It fixes each question's number of discriminative selections outside the required set, kqk_q, along with the required clips, temporal constraints, and remaining budget for relevance filling. Broad Coverage improves over this control on all three benchmarks by +1.07 [0.00, 2.15] / +1.65 [0.07, 3.32] / +1.22 [0.12, 2.33] pp (95% CIs clustered by video). These results support the contribution of selection based on the current set, beyond the amount of relevance filling.

Uncertainty: All intervals are unadjusted paired bootstrap CIs clustered by physical video, using 10,000 replicates. VideoMME is stratified by duration; MLVU resamples global video clusters and recomputes M-AVG. In the separate family of twelve McNemar tests on individual questions, only LongVideoBench Core versus MDP³ remains significant after Holm correction (p=0.00605p=0.00605); the matched filling comparisons do not. The intervals clustered by video and the McNemar tests use different statistical units. For MLVU, M-AVG also differs from accuracy across individual questions.

AuthorsOpenReview ↗

Response to Reviewer qcRi

Reply to W2 & Q2: Independent contribution of posterior weights

Concern: What does πaπb\pi_a\pi_b add when the core, groups, budgets, and readout are fixed?

Reply: Competition weighting gives positive E48E_{48} point differences over uniform pair weights on all three benchmarks (+0.19 / +0.37 / +0.51 pp), with confidence intervals spanning zero. The mean across the three benchmarks is 69.01% versus 68.65% at E48E_{48} and 68.97% versus 68.77% after fusion.

Both variants rank the same eight disjoint groups, each containing four frames, A={A1,…,A8}\mathcal A=\{A_1,\ldots,A_8\}, outside the same parent core. The discrimination scores, frame budgets, and settings for the answering model are fixed. Uniform weighting replaces πaπb\pi_a\pi_b with a constant. E40E_{40} and E48E_{48} are evaluated directly; Full uses the same geometric pooling rule for both variants.

OutputPair weightingVideoMMELongVideoBenchMLVU M-AVGMean
E40Uniform67.0065.0773.8368.63
E40Competition67.4465.0773.7768.76
E48Uniform67.3364.7773.8568.65
E48Competition67.5265.1574.3569.01
FullUniform67.3365.0073.9768.77
FullCompetition67.4465.2274.2568.97

The 95% confidence intervals for the E48E_{48} weighting differences, clustered by video, are [−0.52, 0.85] / [−0.53, 1.28] / [−0.40, 1.40]. The overall E48E_{48} gains over Core are +0.96 / +0.52 / +0.74 pp, before fusion.

For the temporal controls, groups A1 to A8 are ordered chronologically. The original control based on temporal blocks uses A2/A6 for E40E_{40} and A1/A3/A5/A7 for E48E_{48}. The new control using chronological prefixes uses A1/A2 and A1/A2/A3/A4, respectively. Both controls remain reported, including the original 74.77 MLVU Full score versus 74.25 for competition weighting.

E48E_{48} scores using chronological prefixes are 67.44 / 65.22 / 74.47 (mean 69.04%), compared with 67.52 / 65.15 / 74.35 (mean 69.01%) for competition weighting. Thus, temporal supplementation remains competitive. We will distinguish the overall benefit of expansion from the smaller contribution of posterior weighting, which varies across configurations. For the E48E_{48} comparisons, controls without posterior weighting retain the same Stage 1 Broad Coverage core and budget of 48 frames, but require only one call to the answering model; they match the evidence budget, not the inference cost.

Reply to W2 & Q3: Output choice and the third call

Concern: Why prefer fusion to E48 given its extra cost and mixed accuracy?

Reply: We will use the direct E48E_{48} prediction as the default HDEA output in the revision and retain the original fusion with three calls as an optional configuration. Both evidence acquisition stages are retained.

OutputNominal callsVideoMMELongVideoBenchMLVU M-AVGMean
Direct E48 prediction267.5265.1574.3569.01
Original fusion367.4465.2274.2568.97

Across twelve model and benchmark settings, E48E_{48} is higher in seven, tied in one, and lower in four, with an unweighted mean difference of +0.07 pp. E48E_{48} construction uses the core prediction and does not require evaluating E40E_{40}; fusion itself invokes no model.

For Qwen3-VL-8B, archived cumulative answering times for E48E_{48} and fusion are 9.95 vs. 14.63 / 10.16 vs. 14.64 / 14.29 vs. 21.42 seconds, or 32.0% / 30.6% / 33.3% lower. These sums include actual repeated fallback evaluations and exclude SigLIP feature extraction and selector overhead.

This change to the default output during review will apply uniformly across models and benchmarks. We will preserve every original fusion result and report the configurations using two and three calls separately.

Thanks authors for the detailed response. They have largely address my concerns on my side. I will revise my score accordingly.

Reviewer_5zSvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

HDEA is a training-free frame selector for multiple-choice long video QA. Each two-frame clip gets a score per option pair, defined as the gap between the SigLIP similarities of the frame to “question + option A” and to “question + option B”. A 32-frame core is built by greedy max-coverage over these pair scores inside the AKS temporal partition. The answer model’s prediction on the core then weights the pairs, extra frames are added and the three predictions are averaged in log space. With Qwen3-VL-8B the pipeline reaches 67.4 / 65.2 / 74.3 on VideoMME, LongVideoBench and MLVU.

Critical strengths

The method is fully specified, the configuration is fixed across four backbones, and code is released. The 32-frame core beats uniform sampling and a reimplemented AKS on the same backbone for all four models (Table 2, Figure 3), and this part of the result looks solid. The appendix also reports the negative controls on stage 2, which makes the paper easy to assess.

Critical weaknesses
  • The lower block of Table 1 copies numbers from papers that run on older backbones, while HDEA runs on Qwen3-VL-8B, whose 32 uniformly sampled frames alone (Table 2) is already close to several listed methods. The one backbone shared with prior work is LLaVA-Video-7B, and there HDEA (Figure 3) is below both AKS and FOCUS in Table 1, which use one call. On the same backbone the method trails prior selectors, so the “highest in the table” claim comes from the backbone.

  • With four options the coverage objective is a sum of six maxima, so at most six clips can ever have positive marginal gain, and Figure 1 shows the objective is nearly saturated at K=4. The rest of the 16-clip core is filled by the AKS tie rule (relevance, then time). The proposed objective therefore decides a few clips on top of AKS, the paper never reports how many, and the 1/2 approximation guarantee concerns an objective that greedy almost maximizes in a few steps anyway.

  • PairwisePointwise is below Uniform on all three benchmarks, so the pair score on its own is a weak relevance signal. The large gap to Broad Coverage is then explained by Broad Coverage spending most of its budget under the AKS rule, not by set complementarity. Against OptionPool32, a plain option-aware pointwise baseline under the same constraints, the gaps on LongVideoBench and MLVU amount to a few questions (Table D.1). A control that fills the budget the same way (top-K pointwise plus AKS fill) is needed, and Figure 1 cannot serve as evidence because greedy optimizes the metric being plotted.

Questions for authors
  • Per benchmark, how many of the 16 core clips are selected with positive marginal gain, and how many by the tie rule?

  • Can you report one-call Uniform and AKS at a token budget matched to HDEA, and the backbone at its default sampling?

  • Given Table 3 and Appendix F, why keep fusion instead of reporting E48 as the method?

  • Were K1, K2 and the fusion rule chosen on a held-out split?

  • Can you add the temporal-ranking control on VideoMME and LongVideoBench?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 5zSv

We thank the reviewer for recognizing the controlled Core results and released implementation. We have completed the requested controls for allocation, matched filling, resources, default sampling, and temporal selection.

New QA controls use frozen Qwen3-VL-8B on all 2,700 VideoMME / 1,337 LongVideoBench / 2,174 MLVU-Dev questions. For MLVU, we report M-AVG across seven tasks. Mean is the descriptive, unweighted average of the three displayed benchmark scores.

AuthorsOpenReview ↗

Response to Reviewer 5zSv

Reply to W2, W3 & Q1: Saturation, allocation, and matched filling

Concern: Does Broad Coverage mainly rely on relevance filling after early saturation, and does this explain its advantage over pointwise selection?

Reply: The number of candidate pairs does not place an upper bound on selections with positive gain. In ΔH(ui∣S)=∑a<bmax⁡{0,di(a,b)−gab(S)}\Delta_H(u_i\mid S)=\sum_{a<b}\max\{0,d_i(a,b)-g_{ab}(S)\}, a pair's current maximum gab(S)g_{ab}(S) can be improved by successive clips. The same pair can therefore contribute positive gain more than once. Our actual selection trajectories contain 331 / 67 / 259 questions with more selections showing positive gain than candidate pairs on VideoMME / LongVideoBench / MLVU, respectively.

We replayed the trajectories with the fixed temporal constraints and the implementation's rounding of marginal gains to seven decimal places. Among questions that use the full budget, selections with positive gain account for approximately 31% to 48% of the 16 clips in the core, averaging 5.01 / 7.62 / 5.10 clips. Relevance filling remains substantial:

BenchmarkFull budget / fallback questionsClips with positive gain, mean (median)Zero gain relevance fill, mean (median)Questions with more positive steps than pairs
VideoMME2,649 / 515.01 (5)10.99 (11)331
LongVideoBench1,074 / 2637.62 (8)8.38 (8)67
MLVU2,174 / 05.10 (5)10.90 (11)259

VideoMME and MLVU have six candidate pairs; LongVideoBench, with five options, has ten. Means use questions that fill the budget, while all fallbacks remain in QA evaluation. Two required clips in VideoMME are counted separately; there are none elsewhere. Ties among positive gains average 0.0185 / 0.0885 / 0.0041 per question that fills the budget.

For the requested control with pointwise selection and matched relevance filling, selection trajectories computed without ground truth answers or QA outcomes determine kqk_q, the number of selections with positive gain outside the required set. The control selects kqk_q clips using pointwise scores, then fills the remaining budget with the same query relevance and temporal tie rules. Required clips, feasible domain, budget of 32 frames, and answering model are fixed.

SelectorVideoMMELongVideoBenchMLVU M-AVGMean
Relevance-only, same constraints62.7461.6370.9165.09
Pointwise + matched relevance fill65.4862.9872.4066.95
OptionPool3264.8164.1773.2367.40
Broad Coverage / HDEA-Core3266.5664.6273.6268.27

The Relevance-only baseline reuses the same AKS32 inputs, logits, and predictions on VideoMME and MLVU. On LongVideoBench, 762 selected frame sets and 182 predictions differ, despite identical overall accuracy.

Key finding: With the number of discriminative selections and the relevance filling rule matched, Broad Coverage improves over pointwise selection on all three benchmarks by +1.07 [0.00, 2.15] / +1.65 [0.07, 3.32] / +1.22 [0.12, 2.33] pp (95% paired bootstrap CIs clustered by video, 10,000 replicates). The baseline with matched filling itself substantially improves over PairwisePointwise32, showing the importance of relevance filling. The additional Core gains support a contribution from selecting clips based on the current set, beyond the amount of filling. Pointwise ranking uses each clip's standalone pair score; selection by marginal gain rewards only improvements over the pairwise coverage already present in the set.

Uncertainty: These bootstrap intervals are unadjusted, and the VideoMME lower endpoint rounds to 0.00. None of the three matched-fill contrasts reaches significance after Holm correction across the twelve primary McNemar tests at the question level. Those tests use correctness at the question level; the bootstrap resamples videos and, for MLVU, evaluates accuracy averaged across tasks. Core minus OptionPool is +1.74 / +0.45 / +0.38 pp, with the latter two bootstrap intervals clustered by video including zero.

AuthorsOpenReview ↗

Response to Reviewer 5zSv

Reply to W1 & Q2: Matched selectors and computational resources

Concern: Are the reported advantages attributable to the backbone or additional compute?

Reply: With the same answering model, candidate catalog, a budget of 32 frames, and one call, Core improves over AKS on all three benchmarks and has the highest observed mean, 68.27%, among the matched selectors below. We also report the requested larger budget controls with one call, separately from the comparison at a fixed budget.

All selectors using 32 frames share native visual processing, the same prompt, and scoring based on option tokens. MDP³ retains its kernel, dynamic programming procedure, and sequential constraints using shared cached SigLIP features, and is labeled as an adaptation. OptionMax uses si=max⁡azi(ca)s_i=\max_a z_i(c_a) to score each clip under Core's constraints.

SettingNominal callsVideoMMELongVideoBenchMLVU M-AVGMean
Selectors using 32 frames
Uniform32163.5958.2663.9561.93
AKS32, matched setting162.7461.6370.9165.09
OptionMax32166.3763.7273.8967.99
MDP³-32, matched adaptation167.4160.5872.5066.83
HDEA-Core32166.5664.6273.6268.27
Larger budgets and default sampling
Uniform, matched to Full tokens168.4161.2671.1166.93
AKS, matched to Full tokens (adaptation)166.5664.1074.5868.41
Default processor sampling168.7463.9572.5068.40
HDEA output configurations
Direct E48 prediction267.5265.1574.3569.01
Original fusion (3 calls)367.4465.2274.2568.97

Results at the fixed budget: Core minus AKS is +3.81 / +2.99 / +2.71 pp. Core exceeds the MDP³ adaptation by 4.04 pp on LongVideoBench (McNemar test at the question level, p=0.00605p=0.00605 after Holm correction), while MDP³ leads on VideoMME and OptionMax is slightly higher on MLVU.

Resource comparison: Direct E48E_{48} has a higher observed mean (69.01%) than Uniform matched to Full tokens (66.93%), AKS matched to Full tokens (68.41%), and default processor sampling (68.40%). Relative to default processor sampling, it gains 1.20 pp on LongVideoBench and 1.85 pp on MLVU, but is lower on VideoMME. AKS with a matching token budget is also higher on MLVU. E48E_{48} uses at most 48 source frames per call and 80 frame instances across two calls; repeated core frames count in both calls.

The token target matches the original Full configuration with three calls, not E48. It sums visual tokens across Core/E40/E48, merging only exactly identical fallback inputs; shared core frames in different contexts are counted again. Frame and pixel budgets are set from processor geometry, with all questions within ±5% of the target and mean actual/target ratios 0.9890 / 0.9928 / 1.0070. The enlarged AKS control includes a step to fill missing quota slots. These matches use measured token counts. They do not use a fixed 120 frame budget or match FLOPs or latency.

Default processor sampling uses 2 FPS, capped at 768 frames, and accesses the full video beyond HDEA's candidate catalog. The cap is not the number of frames used for every video. We will retain the higher published AKS/FOCUS results on LLaVA-Video-7B under their reported configurations and base selector comparisons on the matched experiments above rather than the heterogeneous table's highest entry.

Comments on the paper

1 comment
Area_Chair_GChtOpenReview ↗

Discussion Reminder

Dear Reviewers,

I hope this message finds you well. As we are now in the discussion period, I would like to kindly remind you to leave comments for the authors through OpenReview if you have not already done so.

Thank you very much for your time and effort. Please let me know if you have any questions.

Best

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.