acceptodds
Under review as a conference paper at ICLR 2027

DES: Discriminative Evidence Selection for Long-Video Question Answering

Abstract

Answering questions about long videos requires identifying relevant visual evidence distributed over time. With a limited frame budget, multiple-choice video question answering also requires evidence that distinguishes among candidate answers. However, question-relevant frames may provide little discriminative information, and several high-scoring frames may depict the same content. We introduce Discriminative Evidence Selection (DES), a training-free framework that estimates frame utility by combining semantic relevance, differences in frame–option matching, object and action cues, and local temporal evidence. DES constructs a candidate pool from local score maxima, temporal segment representatives, and globally high-scoring frames. It then selects a diverse subset using maximal marginal relevance, progressively relaxing the minimum temporal gap between selected frames as needed to fill the frame budget. The selected frames are provided in chronological order to a frozen video-language model. Experiments on four benchmarks with frame budgets of 8, 16, and 32 show that DES outperforms uniform sampling in most settings. With LLaVA-OneVision-7B and eight input frames, DES improves accuracy over BOLT by 2.0 and 3.3 percentage points on Video-MME and MLVU, respectively. Under the same eight-frame budget, evaluations on Video-MME without subtitles using five additional video-language models show accuracy gains of 3.2–6.1 percentage points over uniform sampling.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video question answering, Training-free frame selection, Video-language models, Multi-evidence scoring, Diversity-aware selection.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WRXXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces DES, a training-free frame selector for multiple-choice long-video question answering. It combines question relevance, answer-option matching, visual cues, and temporal information, then selects frames using redundancy control and temporal spacing. Experiments show improvements over uniform sampling and BOLT in most tested settings.

Critical strengths
  1. DES works with frozen video-language models without additional training. The evaluation covers four benchmarks and six downstream models overall, providing useful evidence of its applicability.

  2. The paper includes meaningful ablations. Table 4 shows that multi-cue scoring helps when combined with diversity-aware selection, although it does not improve direct Top-K selection. The authors also report settings where DES underperforms the baselines.

Critical weaknesses
  1. The advantage over closely related methods remains unclear. Comparisons mainly use Uniform and BOLT, while methods such as MDP³ already combine relevance and diversity. The ablations support the usefulness of DES’s components, but do not establish whether its discrimination score improves over simpler option-aware scoring with the rest of the pipeline fixed.

  2. The computational cost and some reported results need clarification. DES improves accuracy at fixed downstream frame budgets, but also processes candidate frames at 1 FPS using CLIP. Without end-to-end timing, its overall accuracy–cost trade-off is unclear. The MLVU results also differ between Table 1 (67.2/68.8/68.3 at 8/16/32 frames) and Table 8 (67.8/68.6/68.6). This may be a reporting issue, but it needs to be reconciled to establish the exact gains.

Questions for authors
  1. Can DES be compared with MDP³ using the same candidate frames, downstream models, and frame budgets? With the other cues and diversity-aware selection fixed, how does the margin-and-concentration cue compare with maximum similarity over question–option pairs?

  2. What is the end-to-end runtime relative to Uniform and BOLT, including frame decoding, CLIP feature extraction, selection, and downstream inference? Are video features reused across questions, and how does this affect the cost per question?

  3. Which MLVU results are correct, those in Table 1 or Table 8? Do the differences reflect reporting errors or different experimental settings?

3 comments
AuthorsOpenReview ↗

Response to CW1–2 and Q1–3: MDP³ comparison, discrimination-cue ablation, end-to-end runtime, and MLVU corrections.

Thank you for your comments on comparisons with related methods, computational cost, and result consistency.

CW1 / Q1: Comparison with MDP³

MDP³ reports higher accuracy than DES. We have not yet completed the requested controlled comparison using identical candidate frames, downstream models, and frame budgets. Reported accuracy (%) with LLaVA-OneVision-7B and 8 input frames is shown below.

Method (source)LongVideoBenchVideo-MME (no subtitles)MLVU
MDP³ (original paper)59.059.669.8
DES (our experiments)55.758.167.8

MDP³ uses 128 uniformly sampled candidates and SigLIP; DES uses candidate frames sampled at 1 FPS and CLIP-L/14. These cross-paper results cannot replace a controlled comparison or establish why the performance gap occurs. We will add the reported results and protocol differences to the manuscript.

CW1 / Q1: Incremental value of the discrimination cue

Replacing our cue with maximum similarity over question–option pairs reduces accuracy by 1.3 percentage points (pp); removing it reduces accuracy by 1.1 pp. On Video-MME with LLaVA-OneVision-7B and 8 frames, all other scoring cues and the DAS selection procedure are held fixed.

Discrimination cueAccuracy (%)
Removed57.0
Maximum frame-to-text similarity over question–option pairs56.8
Margin-and-concentration (full DES)58.1

Maximum similarity measures the strongest option match, whereas our cue also captures relative matching differences across options. This ablation supports its incremental value within DES under this setting; the cue remains a proxy for discriminative utility, not a guarantee of answer correctness. We will add this controlled ablation to the manuscript.

CW2 / Q2: End-to-end runtime and feature reuse

Across all 2,700 Video-MME questions, DES improves accuracy by 4.2 pp over Uniform and 2.0 pp over BOLT, while reducing total runtime by 18.3% and 13.0%, respectively. All methods use LLaVA-OneVision-7B, 8 input frames, the same hardware, and the same timing protocol.

MethodAccuracy (%)Total runtime (s)Mean processing time (s/question)
Uniform53.96,1352.272
BOLT56.15,7632.134
DES58.15,0151.857

Timing starts with data loading and ends at completion of downstream inference, covering the stages actually executed by each method: data loading/frame decoding, CLIP image/text feature extraction, scoring/frame selection, and downstream inference. Candidate-frame CLIP image features are not reused across questions. Mean processing time is total evaluation time divided by 2,700. These runtime results apply to the measured configuration. We will add this table and timing protocol to the manuscript.

CW2 / Q3: MLVU result consistency

The correct MLVU results are those in Table 8, including the corresponding per-task scores. The inconsistency arose from mixing results from different runs with the same experimental configuration and evaluation protocol. We apologize for this reporting oversight. M-Avg is the unweighted mean accuracy across seven tasks.

FramesUniform (%)BOLT (%)DES (%)Gain vs. Uniform (pp)Gain vs. BOLT (pp)
858.563.967.8+9.3+3.9
1661.465.868.6+7.2+2.8
3263.066.668.6+5.6+2.0

Under these settings, DES achieves higher accuracy than Uniform and BOLT at all three frame budgets. We will make the following corrections:

  • Table 1: replace the 8/16/32-frame MLVU scores 67.2/68.8/68.3 with 67.8/68.6/68.6.
  • Abstract: correct the 8-frame MLVU gain over BOLT from 3.3 to 3.9 pp (67.8 − 63.9).
  • Main text: update the associated gains and cross-budget comparisons to match Table 8, ensuring consistency across the main table, per-task results, abstract, and text.

We appreciate your review and look forward to your response!

AuthorsOpenReview ↗

New MDP³ results on MLVU at 16 and 32 frames

CW1 / Q1: Comparison with MDP³

Thank you for suggesting this comparison. We evaluated MDP³ and DES as frame selectors on MLVU at two frame budgets, 16 and 32 frames per question. At each budget, each method selected the specified number of frames from the video associated with the question. LLaVA-OneVision-7B then answered the original question using the frames selected by that method. The table reports downstream question-answering accuracy.

Frames selected per questionMDP³ accuracy (%)Source of MDP³ resultDES accuracy (%, our experiments)DES − MDP³ (percentage points)
869.8Original paper67.8−2.0
1667.3Our run68.6+1.3
3267.9Our run68.6+0.7

Using frames selected by DES, LLaVA-OneVision-7B achieved higher accuracy than it did with frames selected by MDP³. The difference was 1.3 percentage points at 16 frames and 0.7 percentage points at 32 frames.

Each method retained its own frame-selection protocol. MDP³ used SigLIP to score 128 uniformly sampled candidate frames, while DES used CLIP-L/14 to score candidate frames sampled at one frame per second. We will report these results and experimental settings in the revised manuscript and identify the source of each result.

Thanks for the authors' response and the additional experiments/results , my concerns have been partially addressed, but the controlled comparison with MDP3 is yet not fully completed and also I would like to verify the promised changes will be reflected in the actual paper. I will maintain my score. Again, thanks for the authors effort!!

Reviewer_8kTbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces DES, a training-free frame selection method for multiple-choice VideoQA. It combines question relevance, answer-option discrimination, visual cues, and temporal evidence, followed by diversity-aware selection using MMR. Experiments across four benchmarks and multiple VLMs demonstrate improvements over uniform sampling and BOLT in most settings.

Critical strengths

The distinction between question-relevant and answer-discriminative evidence is well motivated and practically relevant to multiple-choice VideoQA.

Cross-model evaluations and component ablations provide useful empirical evidence, particularly under small frame budgets.

Critical weaknesses

Limited technical novelty. The approach combines established similarity metrics, manually designed scoring cues, MMR, and temporal spacing. Moreover, the proposed multi-evidence scores do not improve over query-only scores when paired with Top-K selection, raising questions about their independent contribution.

Insufficient efficiency and competitive evaluation. The paper does not provide end-to-end computational costs, and comparisons with stronger recent option-aware selection methods remain limited.

The authors don't provide enough comparison with previous frame selection/ training-free agentic methods, causing question about actual performance against the previous SoTA.

Questions for authors

Can the authors compare DES against other frame selection methods, under identical candidate pools, selection budgets, and computational costs?

How robust is the discriminative score when a frame strongly matches an incorrect answer option?

1 comment
AuthorsOpenReview ↗

We clarify DES’s contribution and scoring–selection complementarity, provide additional ablations and end-to-end timing results, compare against reproduced and published baselines, and discuss limitations when frames strongly match incorrect answer options.

Thank you for recognizing the motivation and empirical evidence. Accuracy is in %, and gains are in percentage points (pp).

W1: Contribution and scoring-selection complementarity

DES combines answer discrimination and frame-set complementarity for multiple-choice VideoQA. Relative option-matching differences, together with semantic, visual and local temporal cues, estimate frame utility; established MMR and temporal constraints then construct a fixed-budget evidence set. We acknowledge that MEUE did not improve Top-K in the reported experiment and added 8-frame ITS comparisons to examine its contribution.

Table R4. Component analysis on Video-MME without subtitles.

ModelFrames/sourceScoringTop-KITSDASDAS-ITS
OV16/corrected DASMEUE-59.159.5+0.4
OV8/new ITSQuery-only55.356.456.40.0
OV8/new ITSMEUE55.257.058.1+1.1
Qwen8/new ITSMEUE-58.759.9+1.2

OV = LLaVA-OneVision-7B; Qwen = Qwen2.5-VL-7B. "-" means not listed.

At 8 frames, MEUE improves OV accuracy over Query-only by 0.6 pp with ITS and about 1.7 pp with DAS: its benefit is not exclusive to DAS. DAS's gain over ITS rises from 0.0 pp at the reported precision with Query-only to 1.1 pp with MEUE. The Qwen result adds evidence for DAS over ITS with MEUE. These findings support complementarity in the tested settings, not a universally superior standalone scorer.

W2/W3/Q1: End-to-end runtime, recent baselines and comparison conditions

We report end-to-end runtime on all 2,700 Video-MME questions with 8 frames, using the same downstream model, hardware and timing scope.

MethodAccuracyTotal end-to-end time (s)Mean (s/question)
Uniform53.961352.272
BOLT56.157632.134
DES58.150151.857

DES achieves higher accuracy and lower end-to-end runtime under this setup.

We reproduced FOCUS and compared with published A.I.R., FrameOracle and DIG results. DIG and A.I.R.'s official Video-MME selection pipelines use answer-option information, adding option-conditioned comparisons.

LLaVA-OneVision-7B, Video-MME, no subtitles.

Method/source8 frames16 frames32 frames
FOCUS (our reproduction)58.157.958.3
A.I.R. (published)57.959.1 (<=16)61.4 (<=32)
FrameOracle (published)57.5--
DES (ours)58.159.560.1

A.I.R. values come from its appendix. FrameOracle's Table 2 uses a trained selector choosing 8 of 64 candidates. DES ties FOCUS at 8 frames and gains 1.6/1.8 pp at 16/32 frames; A.I.R. reports higher accuracy with up to 32 frames.

Qwen2.5-VL-7B: each cell is DIG / DES.

FramesVideo-MMELongVideoBenchMLVU
854.9 / 59.955.2 / 56.258.6 / 64.4
1658.8 / 61.757.9 / 58.364.0 / 67.2
3261.7 / 63.860.4 / 60.267.2 / 68.5

DIG values are from its Table 2 with a 7B reward model; its Video-MME overall score averages the short/medium/long scores. DES values are ours. DES is higher in 8/9 comparisons, except 32-frame LongVideoBench (60.2 vs. 60.4).

Q2: Strong matches to incorrect options

An incorrect option can indeed produce a high margin and concentration. The score is a proxy for discriminative value, not confidence in answer correctness. It contributes to frame utility alongside other cues; the frozen VLM answers using the selected frames, question and all options. These mechanisms do not guarantee protection against misleading matches.

In the original OV/Video-MME/8-frame ablation, removing answer discrimination reduces accuracy from 58.1% to 57.0%. This supports its aggregate contribution in that setting, not robustness specifically to incorrect-option matches. We retain this failure mode as a limitation.

We appreciate your review and look forward to your response!

Reviewer_2efoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies frame selection for Video QA within a limited budget, and proposes to combine the fusion of multiple utilities and diversity-aware frame selection to reduce redundancy. Specifically, first, Multi-Evidence Utility Estimator (MEUE) fuses semantic, option-specific, visual, and temporal cues for estimating frame utility scores. Second, Diversity-Aware Selection (DAS) select a frame set within a budget by considering redundancy. In the experiment, the proposed model mostly outperforms the uniform selection on several benchmarks and multimodal LLMs. The presented ablation study also shows the effectiveness of each proposed component.

Critical strengths
  • The proposed idea of utilizing relative matching differences across answer options as a proxy for discriminative value is interesting and well-motivated.
  • Writing is easy to follow.
Critical weaknesses
  • My main concern is that the experimental results do not show the significance of the proposed approach. This paper mainly uses the uniform frame selection as the main baseline, which is far less effective and may be a too weak baseline. Even though BOLD (Liu et al. (2025b) was additionally used as another baseline with one MLLM (Table 1), it was not exploited in other experiments. Furthermore, the frame selection for video QA is a very active area of research, and it needs to include more recent and relevant work as baselines, e.g., [1-4]. In addition, the performance gain over BOLT is sometimes marginal. Finally, the effectiveness of DAS is just slightly better than ITS (59.41 vs. 59.11 in Table 5).
  • The experiment is also not very comprehensive. At least, BOLT needs to be added in other experiments, e.g., in Table 2. Also ablation study and other analyses may need to be checked with other MLLMs as well.
  • Some claims in the paper are over-stated. Specifically, it is stated "DES can achieve higher accuracy with a smaller downstream frame budget on the first three benchmarks.", but it was validated by comparison with the uniform sampling, and it needs to be compared with more recent and effective baselines.

References

[1] Zhu et al., FOCUS: Efficient Keyframe Selection for Long Video Understanding, ICLR 2026

[2] Zou et al., A.I.R.: Adaptive, Iterative, and Reasoning-based Frame Selection for Video Question Answering, ICLR 2026

[3] Li et al., Divide, then Ground: Adapting Frame Selection to Query Types for Long-Form Video Understanding, CVPR 2026

[4] Li et al., FrameOracle: Learning What to See and How Much to See in Videos, ICML 2026

Questions for authors
  • Can you report the experimental results with more recent relevant baselines I mentioned above?
  • I found that, in Table 1, the baseline results, especially BOLT's, are different from what are reported in the prior work (BOLT), e.g., on LongVideoBench with 32 frames, this paper reports 58.0 but BOLT reports 59.6. Can you explain why?
1 comment
AuthorsOpenReview ↗

We report comparisons with recent baselines, BOLT evaluation on additional models, and new 8-frame ITS comparisons alongside the original 16-frame ablation. We also clarify the claim about using fewer frames and distinguish reproduced from published results.

Thank you for recognizing our option-discriminative design and clear presentation. Accuracies are in %, gains are in percentage points (pp), and “—” indicates entries not shown.

W1/Q1: Recent baselines

We reproduced FOCUS and compared DES with published A.I.R., FrameOracle, and DIG results.

Table R1. Video-MME, LLaVA-OneVision-7B.

Method8 frames16 frames32 frames
FOCUS (our reproduction)58.157.958.3
A.I.R. (published)57.959.1 (≤16)61.4 (≤32)
FrameOracle (published)57.5——
DES58.159.560.1

A.I.R. values come from its appendix Table 11; FrameOracle values come from its Table 2 and use a trained selector to choose exactly 8 of 64 candidate frames. Published values retain the original evaluation settings.

At the reported precision, DES matches FOCUS at 8 frames and exceeds it by 1.6 and 1.8 pp at 16 and 32 frames, respectively. A.I.R. reports higher accuracy at its ≤32-frame budget.

Table R2. Each cell shows DIG (published Table 2) / DES (ours). Both methods use Qwen2.5-VL-7B as the downstream model; DIG also uses it as the reward model.

FramesVideo-MMELongVideoBenchMLVU
854.9 / 59.955.2 / 56.258.6 / 64.4
1658.8 / 61.757.9 / 58.364.0 / 67.2
3261.7 / 63.860.4 / 60.267.2 / 68.5

DIG's Video-MME score averages its reported short-, medium-, and long-video accuracies. DES scores higher in eight of nine comparisons, except LongVideoBench at 32 frames.

W2: Cross-model and component analysis (also addressing W1)

The submitted appendix contains 18 BOLT comparisons across two Qwen models, three benchmarks, and three budgets; DES scores higher in 16. We now add BOLT comparisons on Video-LLaVA, LongVA, and LLaVA-Video alongside the existing Qwen results.

Table R3. Video-MME, all 2700 questions, no subtitles, 8 frames. BOLT values are our reproductions.

ModelBOLTDESGain
Video-LLaVA39.540.8+1.3
LongVA50.352.4+2.1
LLaVA-Video58.360.6+2.3
Qwen2.5-VL-7B-Instruct57.459.9+2.5
Qwen3-VL-8B-Instruct58.962.0+3.1

We acknowledge the modest 0.3 pp DAS–ITS gain in the original 16-frame experiment in Table 5. We add 8-frame ITS comparisons to examine scoring and selection jointly and extend the analysis to Qwen2.5-VL-7B.

Table R4. Video-MME without subtitles: original 16-frame experiment and new 8-frame ITS comparisons.

ModelFrames/statusScoringTop-KITSDASDAS−ITS
LLaVA-OneVision-7B16, original Table 5MEUE—59.159.4+0.3
LLaVA-OneVision-7B8, new ITSQuery-only55.356.456.40.0
LLaVA-OneVision-7B8, new ITSMEUE55.257.058.1+1.1
Qwen2.5-VL-7B8, new ITSMEUE—58.759.9+1.2

For OneVision at 8 frames, Top-K and DAS results were already in the submission; ITS is newly added. These combinations use the same candidate frames and downstream inference settings. The Qwen2.5 comparison covers all 2700 questions.

At 8 frames, ITS and DAS tie under Query-only scoring on OneVision at the reported precision. With MEUE, DAS exceeds ITS by 1.1 pp on OneVision and 1.2 pp on Qwen2.5. These results support combining MEUE with DAS in the tested settings.

We will add the baseline comparisons with source labels, extend main-text Table 2 with BOLT, and include the new ITS comparisons in the ablation section.

W3: Scope of the fewer-frame claim

We agree that the comparison should be explicit and will revise the statement to:

Under the evaluated LLaVA-OneVision-7B settings, DES with 16 frames achieves higher accuracy than uniform sampling with 32 frames on Video-MME, LongVideoBench, and MLVU.

Q2: BOLT result provenance

Table 1 reports our BOLT reproduction under our evaluation settings. On LongVideoBench at 32 frames, BOLT scores 58.0 (original paper: 59.6); DES scores 59.5, a 1.5 pp gain over our reproduction. We will explicitly label BOLT as “our reproduction” in the table note.

We appreciate your review and look forward to your response!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.