- Rating
- 1(−2)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1(−1)
Read review
Paper summary
This paper studies how to select more effective video evidence under a limited visual budget for long-video multiple-choice question answering. The authors propose HDEA, which leverages the competition among candidate answers to guide evidence selection. In the first stage, the method selects a complementary set of video clips according to the discrimination needs of different candidate-answer pairs, forming an evidence core. In the second stage, it uses the candidate-answer probabilities obtained from the initial prediction to acquire more targeted supplementary evidence, and combines predictions obtained under different visual contexts. The method requires no additional training and is evaluated on three long-video QA benchmarks with multiple frozen video MLLMs.
Critical strengths
-
The paper organizes video evidence around the distinctions among candidate answers and further considers complementarity among different pieces of evidence. Compared with frame selection methods that rely mainly on question relevance, this design is better aligned with the decision-making structure of multiple-choice question answering.
-
Comparisons with variants such as OptionPool and PairwisePointwise help disentangle the contributions of candidate-answer information and set-level complementary selection. In particular, Broad Coverage substantially outperforms pointwise selection while using the same pairwise discrimination information, providing useful evidence for the importance of modeling complementarity at the set level.
Critical weaknesses
- Stage 2 introduces additional supplementary evidence and extra answer-model inference, but the resulting performance gains are relatively limited. The current results do not yet clearly demonstrate how much of the improvement comes specifically from the competition-aware focused selection mechanism, rather than simply from using more video evidence and additional answer-model inference.
- The full HDEA pipeline requires three answer-model inference calls, while the three-way prediction fusion provides almost no additional improvement over directly using E48. Therefore, the trade-off between the additional computational cost introduced by Stage 2 and its practical benefit is not yet sufficiently clear.
Questions for authors
- Could the authors further clarify how much of the performance improvement in Stage 2 can be attributed specifically to competition-aware focused selection, as opposed to simply using more video evidence and additional answer-model inference?
- The three-way prediction fusion in the full HDEA pipeline appears to provide almost no improvement over directly using E48. Could the authors further clarify the necessity and practical value of this fusion step?
13 comments
Response to Reviewer VhS4
Reply to W2 & Q2: Practical value of fusion
Concern: Does combining three predictions justify a third call to the answering model?
Reply: We will use the direct prediction as the default HDEA output in the revision. This retains both evidence acquisition stages with two calls to the answering model, while the original three-call fusion remains an optional configuration.
| Output | Nominal calls | VideoMME | LongVideoBench | MLVU M-AVG | Mean |
|---|---|---|---|---|---|
| Direct E48 prediction | 2 | 67.52 | 65.15 | 74.35 | 69.01 |
| Original three-call fusion | 3 | 67.44 | 65.22 | 74.25 | 68.97 |
Fusion minus is −0.07 / +0.07 / −0.10 pp for Qwen3-VL-8B. Across all twelve model and benchmark settings, is higher in seven, tied in one, and lower in four, with an unweighted mean difference of +0.07 pp. The omitted call evaluates ; construction depends on the core prediction, not the prediction.
For Qwen3-VL-8B, archived cumulative answering stage times for versus fusion are 9.95 vs. 14.63 / 10.16 vs. 14.64 / 14.29 vs. 21.42 seconds, corresponding to reductions of 32.0% / 30.6% / 33.3%. These sums retain actual repeated fallback evaluations and exclude SigLIP feature extraction and selector overhead.
The revised default therefore removes one answer model call and about 32% of the recorded answering stage time on the primary model. We will apply this output choice uniformly across the four models and three benchmarks, retain every original fusion result, and identify the default change as a revision during the review period.
-
Regarding Stage 2, the additional experiments show that competition-aware focused selection provides very limited gains over uniform pair weighting, none of which reach statistical significance. More importantly, under the same visual budget, simply selecting the first four supplementary evidence groups in chronological order achieves comparable or even slightly better average performance at a lower inference cost. Although the authors report this result, they do not sufficiently explain why additional model inference is still necessary to obtain candidate-answer probabilities and use them to guide supplementary evidence selection. Therefore, my concern regarding the actual contribution of Stage 2 remains unresolved.
-
In light of the experimental results for Stage 2, I have also reconsidered the design of Stage 1 and its supporting experimental evidence. In my initial review, the substantial performance improvements of Broad Coverage over PairwisePointwise were an important reason for my inclination toward acceptance. However, the newly added matched-fill control shows that these improvements become considerably smaller when the evidence-filling strategy is controlled. Furthermore, the simpler OptionMax32 achieves comparable performance without explicitly modeling candidate-pair discrimination or set-level complementarity. These results weaken the evidence supporting what I originally considered a critical strength of the paper and raise further concerns about the actual contribution of the core mechanism in Stage 1.
I appreciate the additional controlled experiments provided by the authors and hope they can further clarify the necessity and independent contributions of the core mechanisms in both stages in light of these new experimental results, thereby addressing the above concerns.
Response to Reviewer VhS4
We thank the reviewer for recognizing the decision aligned selection and the controlled support for complementarity. We address the two concerns with additional selection controls and a two-call default output for the revision.
All new controls use frozen Qwen3-VL-8B and the complete VideoMME / LongVideoBench / MLVU-Dev question sets: 2,700 / 1,337 / 2,174 questions, including fallback cases. MLVU is evaluated using M-AVG across seven task categories. Scores are percentages; gains are percentage points computed from unrounded scores. Mean is the descriptive, unweighted average of the three displayed benchmark scores.
Reply to W1 & Q1: Contribution of focused selection
Concern: Does Stage 2 help through competition aware selection, or mainly through additional evidence and inference?
Reply: With Qwen3-VL-8B, expanding Core32 to E48 improves accuracy by +0.96 / +0.52 / +0.74 pp across the three benchmarks. Across four answer models, it improves ten of twelve settings, ties one, and decreases one. Within Stage 2, competition weighting yields positive E48 differences over uniform pair weights on all three benchmarks (+0.19 / +0.37 / +0.51 pp). These results separate the overall expansion benefit from the smaller weighting contribution, whose confidence intervals include zero.
The controls fix the parent Core32, eight non-overlapping four frame groups outside the core, expansion budgets, prompt, visual processing, and option-token scoring. Uniform pair weights replace πaπb with a constant. With the groups numbered chronologically as A1 to A8, the new chronological-prefix control uses A1/A2 for E40 and A1/A2/A3/A4 for E48.
For competition minus uniform pair weights at E48, the 95% bootstrap confidence intervals clustered by video are [−0.52, 0.85] / [−0.53, 1.28] / [−0.40, 1.40]. These unadjusted intervals use 10,000 paired physical-video-cluster replicates. At E40, the weighting differences are +0.44 / 0.00 / −0.06 pp. The chronological prefix control is also competitive, with slightly higher E48 scores on LongVideoBench and MLVU.
The new chronological prefix control supplements the original temporal control reported in Table F.3. All original results are retained, including the MLVU fusion accuracy of 74.77.
The existing comparison with single pass Broad Coverage 48 also gives +0.89 / +0.60 / −0.52 pp for core preserving E48 on the three benchmarks (original Table 3). We will distinguish these overall expansion results from the smaller contribution of posterior pair weighting. For the E48 comparisons, posterior free controls retain the same Stage 1 Broad Coverage core and 48-frame budget, but require only one answer model call; they match the evidence budget, not the inference cost.