acceptodds
Under review as a conference paper at ICLR 2027

FULCRUM: Option-Contrastive Frame Selection for Long-Video Understanding

Abstract

Video-capable vision-language models (VLMs) answer questions about long videos under a limited frame budget, making frame selection critical to the evidence available to the answering model. However, question relevance alone does not indicate whether a frame supports one candidate answer over the others. We introduce FULCRUM, a training-free frame selector for multiple-choice video question answering built on option-contrastive relevance. Using a frozen image–text encoder, FULCRUM scores each frame against each question–option pair and combines the strongest match with its margin over the mean match across options. This captures both relevance to an answer hypothesis and its support relative to competing hypotheses. The resulting scores guide budget allocation across temporal segments, while a coverage floor preserves broad temporal coverage and diversity-aware selection reduces redundancy. FULCRUM requires a single encoding pass over the candidate frames and option hypotheses, without querying the answering model during selection. Under a shared evaluation protocol on LongVideoBench and Video-MME, with Qwen2.5-VL-7B and LLaVA-Video-7B, FULCRUM outperforms uniform sampling and reimplementations of ten published selectors at all evaluated budgets of 4, 8, 16, and 32 frames. With eight frames on LongVideoBench, it improves Qwen2.5-VL-7B accuracy from 44.2% to 62.5%, exceeding the strongest competing selector by 3.1 percentage points and surpassing Qwen2.5-VL-72B with Adaptive Keyframe Sampling at the same frame budget. Across both models and benchmarks, four FULCRUM-selected frames outperform 32 uniformly sampled frames.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
frame selection, keyframe selection, long video understanding, video question answering, multimodal large language models, training-free

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_EnTXOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FULCRUM, a training-free frame selection method for multiple-choice long-video question answering. The main idea is to select frames based not only on question relevance but also on their ability to distinguish between candidate answers. The method computes CLIP similarities between frames and question–option pairs and combines maximum similarity with a contrastive score defined as the difference between the maximum and mean similarities across options. Temporal budget allocation and diversity-aware selection are then used to select informative, non-redundant frames.

Experiments on LongVideoBench and Video-MME with two VLMs and different frame budgets demonstrate consistent improvements over uniform sampling and ten existing frame selection methods.

Critical strengths
  1. Strong empirical results. The method consistently outperforms the evaluated baselines across datasets, models, and frame budgets. Particularly notable is that four FULCRUM-selected frames outperform 32 uniformly sampled frames, demonstrating the potential of improved frame selection under constrained visual budgets.

  2. Extensive experimental validation. The paper provides comprehensive ablations, particularly comparing question-only relevance, fused options, and option-conditioned relevance. These experiments help isolate the contribution of the proposed contrastive score. Additional evaluations across model sizes and encoders further support the robustness of the results.

Critical weaknesses
  1. Limited methodological novelty and missing comparison with closely related work. The use of answer options to guide frame selection has already been investigated in prior work, particularly QAaF. The main novelty appears to be the maximum-minus-mean similarity score, combined with established temporal allocation and diversity mechanisms. I am not convinced that this constitutes a sufficiently substantial methodological contribution. Moreover, despite its close relationship to FULCRUM, QAaF is not included in the main experimental comparisons. The maximum-similarity ablation is informative but does not replace a comparison with the complete method. It is also unclear whether the proposed contrastive score offers meaningful advantages over other straightforward measures of discrimination, such as the margin between the two highest option similarities, the variance of similarities across options, or the entropy of the softmax-normalized similarities. These alternatives would respectively capture the separation between the strongest competing hypotheses, the overall spread of option scores, and the uncertainty across candidate answers.

  2. Insufficient computational efficiency analysis. Although the method achieves strong accuracy with fewer selected frames, the frame selection process introduces additional computational overhead, including encoding up to 128 candidate frames with CLIP. The paper does not quantify the resulting end-to-end computational savings. Given the emphasis on efficient long-video understanding, reporting actual inference latency, computational cost, and memory usage, together with accuracy–latency comparisons against strong baselines, would be important for evaluating the practical significance of the method.

Questions for authors
  1. How does FULCRUM compare against the complete QAaF method under the same evaluation protocol and computational budget? This comparison seems important for establishing the contribution relative to closely related work.

  2. What motivates the choice of maximum-minus-mean similarity over other discrimination measures, such as the top-two margin (difference between the highest and second-highest option similarities), similarity variance (spread of the scores across options), or softmax entropy (uncertainty of the normalized option similarity distribution)? Would these alternatives achieve comparable performance when the remaining components are unchanged?

  3. What is the end-to-end computational cost of FULCRUM, including candidate frame encoding and selection? How does its accuracy–latency trade-off compare against the strongest baselines, and how does the computational cost scale beyond 128 candidate frames?

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for recognising the strong and consistent results, in particular that four FULCRUM frames beat 32 uniform frames, and the ablations that isolate the contrast score. Tables R1–R6 are in the two replies below this comment.

W1 + Q1. Novelty and the full QAaF method. We agree that reading the answer options is not new, and the paper credits QAaF and VTR-VLM for it (§2, App. A). Both turn the options into one score per frame: the best-matching option (QAaF) or one fused text (VTR-VLM). FULCRUM instead scores how far a frame separates the options. Table 2 of the paper shows the difference: using the options without contrast adds 0.8 to 1.4 points over the question alone; the contrast adds 3.3 to 3.5.

Full QAaF (Table R1). We ran the complete QAaF method under our protocol: same CLIP encoder, same 128 candidate frames, same answerers and prompts. Its cost is the same as FULCRUM's (4.40 vs 4.41 s per question, Table R4). FULCRUM beats QAaF in all 16 settings, by 2.1 to 7.2 points.

Where the gap comes from (Table R2). QAaF differs from FULCRUM in two ways: no contrast term, and top-k selection instead of our allocation and diversity steps. Adding these one at a time, on LVB with Qwen: QAaF 56.1 → with our allocation and diversity 60.1 → with the contrast term 62.5. The contrast term adds 1.5 to 2.4 points in all four settings, with everything else unchanged. The gain needs the question's own options: options from another video's question, or a paraphrase of the question, fall below the question alone (Table 2 of the paper).

W1 + Q2. Why max − mean, and not margin, variance or entropy? We ran all three in place of max − mean, keeping everything else the same (Table R3). The reviewer's intuition is right that any measure of separation helps: every alternative beats having no contrast term. This supports our main claim. Max − mean is the best in all four settings, for clear reasons:

  • The top-two margin ignores every option below the second.
  • The variance scores a frame that rejects one option the same as a frame that supports one.
  • Entropy needs a temperature, and its results change with it (Table R3 shows three).
  • Max − mean needs none. By Eq. 3 it is the log-odds of the favored option against the geometric mean of all options, and the within-video standardization (§3.2) removes the 1/τ factor.

W2 + Q3. End-to-end cost and scaling beyond 128 frames.

  • Cost (Table R4): one CLIP pass over the 128 candidate frames takes 2.75 s per video and is shared by all questions on that video (1.55 s per question on LVB). Selection adds 0.007 s. Only uniform sampling is cheaper than FULCRUM.
  • End-to-end savings (Table R5): FULCRUM with 4 frames takes 3.02 s and reaches 60.8 on LVB. Uniform with 32 frames takes 11.21 s and reaches 52.9. FULCRUM is 3.7× faster and 7.9 points more accurate.
  • Against the strongest baselines (Table R5): FULCRUM is the most accurate method within every latency budget from 3.1 s to 15 s.
  • Beyond 128 frames (Table R6): encoding time grows in proportion to the pool (2.75, 5.5 and 11.0 s per video for 128, 256 and 512 frames); the rest of selection stays under 0.01 s. A pool of about four times the budget is enough.

Revisions. We will add QAaF to Table 1, the breakdown of the gap to QAaF, the comparison of contrast terms, the cost tables, and the pool-size results.

We hope these additions address the reviewer's concerns, and we respectfully ask the reviewer to consider updating their assessment.

AuthorsOpenReview ↗

Additional Findings (Part 1)

Supporting Tables (1/2): QAaF and the Contrast Term (W1, Q1, Q2)

Table R1. Full QAaF under our protocol. Accuracy (%). The other methods are in Table 1 of the paper.

AnswererMethodLVB 4LVB 8LVB 16LVB 32VMME 4VMME 8VMME 16VMME 32
Qwen2.5-VL-7BUniform42.744.248.852.939.643.346.551.2
AdaRD-Key55.259.460.860.149.354.557.958.2
QAaF53.756.158.357.452.555.957.858.2
FULCRUM60.862.563.263.257.559.861.260.8
LLaVA-Video-7BUniform44.447.351.253.841.645.148.352.6
AdaRD-Key57.859.561.162.255.258.359.660.1
QAaF55.358.359.259.354.457.658.859.3
FULCRUM62.562.564.264.260.461.561.561.4

Table R2. Where the gap to QAaF comes from (k=8).

VariantLVB (Qwen)VMME (Qwen)LVB (LLaVA)VMME (LLaVA)
QAaF: best-match score, top-k frames56.155.958.357.6
Best-match score + our allocation and diversity60.157.960.560.0
+ contrast term (FULCRUM)62.559.862.561.5

Table R3. Only the contrast term changes; everything else is FULCRUM (k=8).

Contrast termLVB (Qwen)VMME (Qwen)LVB (LLaVA)VMME (LLaVA)
Max − mean (FULCRUM)62.559.862.561.5
Neg. entropy, τ=0.01 (CLIP's temperature)62.059.462.161.2
Neg. entropy, τ=0.0561.859.361.961.1
Neg. entropy, τ=161.859.261.961.0
Variance across options61.859.261.961.0
Top-two margin61.759.161.861.0
No contrast term60.157.960.560.0

At τ=1 entropy ranks frames the same way as the variance, which is why those two rows match.

AuthorsOpenReview ↗

Additional Findings (Part 2)

Supporting Tables (2/2): Cost, Latency and Pool Size (W2, Q3)

Table R4. Cost per question at k=8 (T4 GPU, Qwen2.5-VL-7B). Answering takes 2.85 s for every method. Methods that run once per video share the encoder pass across the 1.78 questions per video of LongVideoBench.

MethodScorerRunsTotal (s/q)Answerer calls in selectionLVBVMME
Uniform——2.85044.243.3
BOLTCLIP ViT-L/14per video4.40052.151.7
Q-FrameCLIP ViT-L/14per video4.40054.653.9
QAaFCLIP ViT-L/14per video4.40056.155.9
MaxInfoCLIP ViT-L/14per video4.41056.855.0
WFS-SBCLIP ViT-L/14per video4.41054.854.7
F²CCLIP ViT-L/14per video4.41054.154.3
FULCRUMCLIP ViT-L/14per video4.41062.559.8
FOCUSBLIP ITMper question5.67052.953.1
VTR-VLMCLIP + answererper question6.40150.450.0
MDP3SigLIPper video6.95055.954.6
AKSBLIP ITMper question9.36055.654.6
DIGanswererper question11.38≈2054.953.4
AdaRD-KeyBLIP-2 ITMper question14.46059.454.5

Table R5. Accuracy against latency (Qwen2.5-VL-7B). Top: fewer frames, less time. Bottom: best accuracy within each latency budget, each method at its best k that fits.

ConfigurationTotal (s/q)LVB
Uniform, 32 frames11.2152.9
AdaRD-Key, 32 frames22.8260.1
FULCRUM, 4 frames3.0260.8
Budget (s/q)BenchmarkUniformBest competitorFULCRUM
3.1LVB44.2 (k=8)53.7 (QAaF, k=4)60.8 (k=4)
3.1VMME43.3 (k=8)52.5 (QAaF, k=4)57.5 (k=4)
5LVB44.2 (k=8)56.8 (MaxInfo, k=8)62.5 (k=8)
5VMME43.3 (k=8)55.9 (QAaF, k=8)59.8 (k=8)
10LVB48.8 (k=16)58.3 (QAaF, k=16)63.2 (k=16)
10VMME46.5 (k=16)58.0 (F²C, k=16)61.2 (k=16)
15LVB52.9 (k=32)59.8 (F²C, k=32)63.2 (k=32)
15VMME51.2 (k=32)59.0 (F²C, k=32)61.2 (k=16)

Table R6. Larger candidate pools (Qwen2.5-VL-7B). Cells are uniform / FULCRUM. Uniform does not depend on the pool.

PoolBenchmarkk=32k=64k=128Encode (s/video)
128LVB52.9 / 63.255.4 / 60.956.6 / 56.62.75
256LVB52.9 / 63.655.4 / 63.556.6 / 62.15.5
512LVB52.9 / 64.255.4 / 63.756.6 / 63.711.0
128VMME51.2 / 60.854.7 / 59.856.5 / 56.52.75
256VMME51.2 / 61.054.7 / 61.556.5 / 61.55.5
512VMME51.2 / 61.254.7 / 61.756.5 / 61.711.0

Thank you for the detailed response and additional experiments. I appreciate that the authors directly addressed my concerns, particularly through the full QAaF comparison, the ablations on alternative contrastive scores, and the end-to-end latency analysis.

I still think the methodological novelty is somewhat limited, especially since the alternative contrastive scores achieve quite similar results. However, the additional experiments provide convincing evidence of the method's effectiveness and practical value. Overall, I believe my main concerns have been sufficiently addressed, and I will raise my score.

Reviewer_mzWLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes FULCRUM, a training free video frame selection method for multiple choice QA. Instead of ranking based on similarity of the video to the question (or QA) like prior work, it uses two scores (the best matching option and the best match - mean match over all options) to make this frame selection decision in three stages: option-contrastive scoring, proportional budget allocation, and a diversity-aware greedy selection within segments.

Critical strengths
  1. The idea has been well supported with closed form mathematical justifications (ties to relevant theoretical concepts from the classic literature at places). The ablation study is helpful and good amount of supporting experimental validation is done.
  2. The scoping to only MCQs and limitation that it is on per-frame basis is honest and helpful.
Critical weaknesses
  1. Computational complexity for frame-heavy videos: App. B.3 gives Big-O complexity, but it does not mention the wall-clock latency, throughput, or selection-time overhead or compare them to the other baselines.
  2. The authors cap all videos to 128 frames regardless of actual video length for every evaluation methods; a heavy downsampling even before these video frame selection strategies are applied. This paper doesn't discuss the effect of changing this capping number for long videos.

Minor weaknesses:

  1. Eq. 3 is a non-trivial identity and I would like to see its derivation.
  2. Figure 1 is too cluttered and font size is too small.
Questions for authors
  1. Related to point 1 of Weakness section: which baselines are more efficient than FULCRUM? Any quantitative guideline to suggest a trade-off between the complexity and accuracy gain would be helpful.
  2. Since this method is limited to only MCQs, is there any way to transfer this to open-ended video question answering tasks?
3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for noting the closed-form justifications, the helpful ablations and validation, and the honest scoping. Tables R1–R5 are in the reply below this comment.

W1 + Q1. Wall-clock cost, and which baselines are more efficient. Table R1 gives measured cost per question at k=8 for every method, on T4 GPU. The answer to the question: only uniform sampling is cheaper than FULCRUM. Six other methods run one CLIP pass per video and cost the same as FULCRUM, within 0.01 s (4.40–4.41 s per question). The other six score each question with a heavier model or call the answerer, and cost 5.67 to 14.46 s, 1.3 to 3.3 times as much.

Where FULCRUM's time goes: one CLIP pass over the 128 candidate frames takes 2.75 s per video and is shared by all questions on that video (1.55 s per question on LVB). The rest of selection, the steps whose complexity App. B.3 gives, takes 0.007 s.

Trade-off and guideline:

  • FULCRUM is the most accurate method within every latency budget from 3.1 s to 15 s (Table R3).
  • For a practitioner: FULCRUM with 4 frames (3.02 s, 60.8 on LVB) beats uniform sampling with 32 frames (11.21 s, 52.9) and is 3.7× faster (Table R2).
  • The more questions per video, the cheaper FULCRUM gets, since the encoder pass is shared. Methods that score per question, such as AKS and AdaRD-Key, do not get this saving.

W2. The 128-frame cap. As App. C.2 states, every method, uniform included, picks from the same 128 candidate frames, so the cap is the same for all. To measure its effect we reran FULCRUM with 256 and 512 candidates (Table R4). At the paper's budgets (up to 32 frames), the 128-frame pool is within 1.0 point of a pool four times larger. The cap matters only when the budget nears the pool size. Rule of thumb: use a pool of about four times the budget. Encoding time grows in proportion to the pool (2.75, 5.5 and 11.0 s per video). We will add this to App. C.2.

Minor 1. Derivation of Eq. 3. Let πj(t)=exp⁡(at,j/τ)/Zt\pi_j(t)=\exp(a_{t,j}/\tau)/Z_t, with Zt=∑lexp⁡(at,l/τ)Z_t=\sum_l \exp(a_{t,l}/\tau). Then log⁡πj(t)=at,j/τ−log⁡Zt\log \pi_j(t) = a_{t,j}/\tau - \log Z_t for every jj. With j⋆=arg⁡max⁡jat,jj^\star=\arg\max_j a_{t,j}:

log⁡πj⋆(t)−1J∑jlog⁡πj(t)=(at,j⋆τ−log⁡Zt)−(1J∑jat,jτ−log⁡Zt)=disc⁡(t)τ.\log \pi_{j^\star}(t) - \frac{1}{J}\sum_{j}\log \pi_j(t) = \Big(\frac{a_{t,j^\star}}{\tau} - \log Z_t\Big) - \Big(\frac{1}{J}\sum_{j}\frac{a_{t,j}}{\tau} - \log Z_t\Big) = \frac{\operatorname{disc}(t)}{\tau}.

The log⁡Zt\log Z_t terms cancel. The average of the log-probabilities is the log of their geometric mean, so disc⁡(t)/τ\operatorname{disc}(t)/\tau is the log-odds of the favored option against the geometric mean of all options. The within-video standardization of §3.2 then removes the factor 1/τ1/\tau, so the selected frames do not depend on τ\tau. We will add this derivation to App. B.

Minor 2. Figure 1. We agree. The revised Figure 1 will show only the six numbered steps with large labels. The detailed panels will move to the appendix.

Q2. Open-ended questions. App. F.4 tests the closest step: the selector reads options written by the answerer from the question alone (Table R5). These gain 1.1 to 1.7 points over the question alone in all four settings. So one route to open-ended questions is to generate candidate answers first, then select frames that separate them. Even with no options at all, FULCRUM beats uniform sampling by 12.3 to 15.1 points. We will add this to §6.

Revisions. We will add the cost tables and guideline, the pool-size results, the derivation of Eq. 3, a redrawn Figure 1, and the open-ended discussion.

We hope these additions address the reviewer's concerns, and we respectfully ask the reviewer to consider updating their assessment.

AuthorsOpenReview ↗

Additional Findings

Supporting Tables: Cost, Pool Size, Generated Options (W1, Q1, W2, Q2)

Table R1. Cost per question at k=8 (T4 GPU, Qwen2.5-VL-7B). Answering takes 2.85 s for every method. Methods that run once per video share the encoder pass across the 1.78 questions per video of LongVideoBench.

MethodScorerRunsTotal (s/q)Answerer calls in selectionLVBVMME
Uniform——2.85044.243.3
BOLTCLIP ViT-L/14per video4.40052.151.7
Q-FrameCLIP ViT-L/14per video4.40054.653.9
QAaFCLIP ViT-L/14per video4.40056.155.9
MaxInfoCLIP ViT-L/14per video4.41056.855.0
WFS-SBCLIP ViT-L/14per video4.41054.854.7
F²CCLIP ViT-L/14per video4.41054.154.3
FULCRUMCLIP ViT-L/14per video4.41062.559.8
FOCUSBLIP ITMper question5.67052.953.1
VTR-VLMCLIP + answererper question6.40150.450.0
MDP3SigLIPper video6.95055.954.6
AKSBLIP ITMper question9.36055.654.6
DIGanswererper question11.38≈2054.953.4
AdaRD-KeyBLIP-2 ITMper question14.46059.454.5

Table R2. Fewer frames, less time (Qwen2.5-VL-7B, LVB).

ConfigurationTotal (s/q)LVB
Uniform, 32 frames11.2152.9
AdaRD-Key, 32 frames22.8260.1
FULCRUM, 4 frames3.0260.8

Table R3. Best accuracy within each latency budget (Qwen2.5-VL-7B; each method uses its best k that fits).

Budget (s/q)BenchmarkUniformBest competitorFULCRUM
3.1LVB44.2 (k=8)53.7 (QAaF, k=4)60.8 (k=4)
3.1VMME43.3 (k=8)52.5 (QAaF, k=4)57.5 (k=4)
5LVB44.2 (k=8)56.8 (MaxInfo, k=8)62.5 (k=8)
5VMME43.3 (k=8)55.9 (QAaF, k=8)59.8 (k=8)
10LVB48.8 (k=16)58.3 (QAaF, k=16)63.2 (k=16)
10VMME46.5 (k=16)58.0 (F²C, k=16)61.2 (k=16)
15LVB52.9 (k=32)59.8 (F²C, k=32)63.2 (k=32)
15VMME51.2 (k=32)59.0 (F²C, k=32)61.2 (k=16)

Table R4. Larger candidate pools (Qwen2.5-VL-7B). Cells are uniform / FULCRUM. Uniform spreads its frames over the whole video, so it does not depend on the pool.

PoolBenchmarkk=32k=64k=128Encode (s/video)
128LVB52.9 / 63.255.4 / 60.956.6 / 56.62.75
256LVB52.9 / 63.655.4 / 63.556.6 / 62.15.5
512LVB52.9 / 64.255.4 / 63.756.6 / 63.711.0
128VMME51.2 / 60.854.7 / 59.856.5 / 56.52.75
256VMME51.2 / 61.054.7 / 61.556.5 / 61.55.5
512VMME51.2 / 61.254.7 / 61.756.5 / 61.711.0

At k=128 from a 128-frame pool there is nothing to select, so both methods tie. A pool of about four times the budget is enough.

Table R5. What the selector reads (App. F.4, Table 23; k=8). The answerer always answers with the benchmark's options.

Selector readsLVB (Qwen)VMME (Qwen)LVB (LLaVA)VMME (LLaVA)
Nothing (uniform sampling)44.243.347.345.1
Question only59.357.859.658.9
Options written by the answerer61.058.961.160.2
Benchmark options (FULCRUM)62.861.162.561.5

Thank you for clarifying the cost and latency. It is good that FULCRUM is similarly efficient as six other methods which also use CLIP. But, the idea to translate to open-ended question answering doesn't help much when the answer is longer and multiple possible ways to answer can be correct, like in video summarization task. So, considering both positive and negative sides of the paper, overall I think my given score is justified.

Reviewer_9pEAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed a training-free frame selector for multiple-choice video QA. Instead of scoring frames against the question alone, it scores how well each frame separates the answer options (option-contrastive log-odds), then allocates the frame budget proportionally with a coverage floor and picks diverse frames. Using only a frozen CLIP encoder, it beats uniform sampling and 10 published selectors at every budget (4–32 frames) on LongVideoBench and Video-MME with two answerers (Qwen2.5VL/LLaVA-Video)— and 4 FULCRUM frames even outperform 32 uniform frames.

Critical strengths
  1. FULCRUM outperforms uniform sampling and 10 published selectors at every budget from 4 to 32 frames, on two benchmarks(LongVideoBench/VideoMME) and with two answerers (Qwen2.5-VL/LLava-Video), and giving strong evidences that the gains come from the selection method itself.

  2. The method exploits option information through two complementary statistics: rel(t) for support of the best-matching option hypothesis and disc(t) for how far the favored option stands above the average, the latter having a clean log-odds interpretation whose partition function cancels by construction, making the score independent of the encoder's temperature.

  3. Diversity is handled explicitly and jointly rather than frame-by-frame: a relevance-weighted log-determinant objective measures redundancy against all frames already chosen across segments, comes with a greedy approximation guarantee, and a relevance gate falls back to diversity alone on flat score profiles to avoid chasing noise.

  4. The method is training-free and model-agnostic, needing only a contrastive image–text encoder, leaving the answerer frozen, making a single pass before answering, and producing output that can be passed to any token compressor or downstream Video-LLM.

  5. The experiment is notably thorough: it adds extensive ablations, parameter sweeps, encoder robustness checks, answerer-scale experiments from 3B to 72B, and stratified analyses (language prior, question type, video duration, failure modes), all supporting its claimed gains.

Critical weaknesses
  1. The method requires multi-choice questions, and open-ended questions fall outside its scope.

  2. Gains are demonstrated up to 32 frames, whether the method remains SOTA at larger frame budgets is untested, and longer video benchmarks beyond LongVideoBench/Video-MME (e.g., LVBench, MLVU) are not evaluated.

  3. The method adds an extra image–text encoder pass (CLIP) whose cost is not quantified against the savings achieved by selecting fewer frames.

  4. A wrong option that names an object visible throughout the video may draws frames toward incorrect answers.

  5. Scoring frames independently cannot capture multi-frame temporal structure, and it depends on the quality of the underlying contrastive encoder's fine-grained discriminability.

  6. Performance is sensitive to hyperparameters, such as the segment count m, coverage floor f, concavity c, and gate threshold θ, and the locked configuration may not optimal in every sweep.

Questions for authors
  1. What are the results on LVBench and MLVU, and with larger frame budgets?
  2. What is the computation cost–accuracy trade-off of this method?
  3. How does the method perform when combined with token-compression methods?
6 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for recognising the consistent gains over uniform sampling and ten selectors, the log-odds reading of disc, the joint diversity step with its guarantee, the training-free design, and the thorough experiments. Tables R1–R7 are in the two replies below this comment.

W1. Multiple-choice only. We agree, and the paper states this scope. Multiple choice is the format of the major long-video benchmarks (LongVideoBench, Video-MME, LVBench, MLVU). App. F.4 tests the closest open-ended step: the selector reads options written by the answerer from the question alone (Table R7). These gain 1.1 to 1.7 points over the question alone in all four settings. So generating candidate answers first, then selecting frames that separate them, is a workable route. Even with no options, FULCRUM beats uniform sampling by 12.3 to 15.1 points. We will add this to §6.

W2 + Q1. Larger budgets, LVBench, MLVU.

  • Budgets up to 128 (Table R1): FULCRUM is the most accurate method at every budget up to 64, with both answerers and benchmarks. At 128 every method uses the whole pool, so all tie.
  • Larger pools (Table R2): with 512 candidate frames, FULCRUM at k=128 reaches 63.7 on LVB against 56.6 for uniform. A pool of about four times the budget is enough.
  • LVBench (Table R3; mean 68 min, nothing tuned): FULCRUM is the most accurate at every budget with both answerers, 5.6 to 10.8 points above uniform and 1.1 to 4.6 above the best competitor.
  • MLVU did not finish in the rebuttal window. We will add it to the revision.

W3 + Q2. Cost of the encoder pass. One CLIP pass over the 128 candidate frames takes 2.75 s per video and is shared by all questions on that video: 1.55 s per question on LVB. Selection adds 0.007 s. Only uniform sampling is cheaper (Table R4). The pass pays for itself: FULCRUM with 4 frames takes 3.02 s and reaches 60.8 on LVB, while uniform with 32 frames takes 11.21 s and reaches 52.9 (Table R5).

Q3. Token compression. Selection and compression work together (Table R6). At the same 50% token budget, 16 FULCRUM frames reach 63.2 on LVB, against 52.2 for 32 uniform frames compressed by VisionZip. Compressing FULCRUM's own frames costs only 0.2 to 1.8 points.

W4. A wrong option naming an object seen throughout. We agree. This is one of the four failure modes in App. J (Table 32). If the object is in every frame, disc is flat and the gate falls back to coverage and diversity. If it is in most frames, the coverage floor still places frames across the whole video. FULCRUM does not fall below uniform sampling on any of the 17 LongVideoBench categories (Table 13). We will state this limitation in the main text.

W5. Per-frame scoring and the encoder. We agree that a single frame cannot show the direction of a motion or the order of two events (App. J); two of the three categories where FULCRUM trails AdaRD-Key ask for the order of two moments (Table 13). Selection itself is joint: the allocation spreads the budget, and the log-determinant step checks each frame against all frames already chosen. On the encoder: with six encoders (App. G.5, Table 28), the contrast term adds +2.7 to +3.2 points on LVB every time, so the gain does not depend on one encoder.

W6. Hyperparameter sensitivity. App. G (Tables 25–26) sweeps every parameter. Accuracy stays within 1.5 points of the fixed configuration everywhere except three extremes (f=0, f=1, θ=0.95). The reviewer is right that the fixed configuration is not the best in every sweep; it is within 0.3 points of the best in all of them. The same configuration was used unchanged for LVBench and the larger budgets.

Revisions. We will add LVBench, MLVU, budgets to 128 and pools to 512, the cost tables, the compression results, the open-ended discussion, and the two limitations above.

We hope these additions address the reviewer's concerns, and we respectfully ask the reviewer to consider updating their assessment.

AuthorsOpenReview ↗

Additional Findings.

Supporting Tables (1/2): Larger Budgets, Larger Pools, LVBench (W2, Q1)

Table R1. Budgets up to 128 frames, from the 128-frame pool. At k=128 every method uses the whole pool, so all methods tie.

Answerer, benchmarkMethodk=48163264128
Qwen, LVBUniform42.744.248.852.955.456.6
AdaRD-Key55.259.460.860.159.256.6
QAaF53.756.158.357.457.856.6
FULCRUM60.862.563.263.260.956.6
Qwen, VMMEUniform39.643.346.551.254.756.5
AdaRD-Key49.354.557.958.258.456.5
QAaF52.555.957.858.258.456.5
FULCRUM57.559.861.260.859.856.5
LLaVA, LVBUniform44.447.351.253.855.356.0
AdaRD-Key57.859.561.162.259.856.0
QAaF55.358.359.259.358.256.0
FULCRUM62.562.564.264.260.856.0
LLaVA, VMMEUniform41.645.148.352.654.655.6
AdaRD-Key55.258.359.660.158.655.6
QAaF54.457.658.859.358.255.6
FULCRUM60.461.561.561.459.355.6

FULCRUM is the most accurate method at every budget up to 64. Its lead shrinks at 64 because it must pick half of the pool. Table R2 uses a larger pool.

Table R2. Larger candidate pools (Qwen2.5-VL-7B). Cells are uniform / FULCRUM. Uniform spreads its frames over the whole video, so it does not depend on the pool.

PoolBenchmarkk=32k=64k=128Encode (s/video)
128LVB52.9 / 63.255.4 / 60.956.6 / 56.62.75
256LVB52.9 / 63.655.4 / 63.556.6 / 62.15.5
512LVB52.9 / 64.255.4 / 63.756.6 / 63.711.0
128VMME51.2 / 60.854.7 / 59.856.5 / 56.52.75
256VMME51.2 / 61.054.7 / 61.556.5 / 61.55.5
512VMME51.2 / 61.254.7 / 61.756.5 / 61.711.0

A pool of about four times the budget is enough.

Table R3. LVBench (103 videos, 1,549 questions, mean 68 min). Same configuration as the paper, nothing tuned.

AnswererkUniformAdaRD-KeyAKSFULCRUM
Qwen2.5-VL-7B429.134.338.039.8
831.737.940.942.0
1635.641.242.344.2
3239.242.543.545.0
LLaVA-Video-7B431.737.937.742.5
834.240.140.643.5
1638.243.243.345.9
3241.745.444.047.3
AuthorsOpenReview ↗

Additional Findings (Part 2)

Supporting Tables (2/2): Cost, Token Compression, Generated Options (W1, W3, Q2, Q3)

Table R4. Cost per question at k=8 (T4 GPU, Qwen2.5-VL-7B). Answering takes 2.85 s for every method. Methods that run once per video share the encoder pass across the 1.78 questions per video of LongVideoBench.

MethodScorerRunsTotal (s/q)LVBVMME
Uniform——2.8544.243.3
QAaFCLIP ViT-L/14per video4.4056.155.9
FULCRUMCLIP ViT-L/14per video4.4162.559.8
MDP3SigLIPper video6.9555.954.6
AKSBLIP ITMper question9.3655.654.6
DIGanswererper question11.3854.953.4
AdaRD-KeyBLIP-2 ITMper question14.4659.454.5

Table R5. Fewer frames, less time (Qwen2.5-VL-7B, LVB).

ConfigurationTotal (s/q)LVB
Uniform, 32 frames11.2152.9
AdaRD-Key, 32 frames22.8260.1
FULCRUM, 4 frames3.0260.8

FULCRUM with 4 frames is 3.7× faster than uniform sampling with 32 frames and 7.9 points more accurate.

Table R6. Selection and token compression (Qwen2.5-VL-7B). Visual tokens are relative to 32 uncompressed frames.

FramesCompressorVisual tokensLVBVMME
32 uniformnone100%52.951.2
32 uniformVisionZip, keep 50%50%52.250.9
16 FULCRUMnone50%63.261.2
32 FULCRUMVisionZip, keep 50%50%63.060.2
32 FULCRUMFastV, keep 50%55%62.759.4

Table R7. What the selector reads (App. F.4, Table 23; k=8). The answerer always answers with the benchmark's options.

Selector readsLVB (Qwen)VMME (Qwen)LVB (LLaVA)VMME (LLaVA)
Nothing (uniform sampling)44.243.347.345.1
Question only59.357.859.658.9
Options written by the answerer61.058.961.160.2
Benchmark options (FULCRUM)62.861.162.561.5

Comments on the paper

3 comments
AuthorsOpenReview ↗

Additional Experiments (Part 1): Cost and Latency

We thank all reviewers. This comment and the two below hold the tables cited in our responses. Part 1: cost. Part 2: QAaF and the contrast statistic. Part 3: larger budgets and pools, LVBench, and token compression.

A. Cost and latency (9pEA W3, Q2; mzWL W1, Q1; EnTX W2, Q3)

Setup: T4 GPU x2, Qwen2.5-VL-7B as the answerer. Answering with 8 frames takes 2.85 s per question for every method. One CLIP ViT-L/14 pass over the 128-frame pool takes 2.75 s per video, including frame loading. A method that encodes once per video shares this pass across all questions about that video. LongVideoBench has 1.78 questions per video, so the pass costs 1.55 s per question.

Table A1. Cost per question at k=8.

MethodScorerRunsScorer (s/q)Total (s/q)Answerer calls in selectionLVBVMME
Uniform——0.002.85044.243.3
BOLTCLIP ViT-L/14per video1.554.40052.151.7
Q-FrameCLIP ViT-L/14per video1.554.40054.653.9
QAaFCLIP ViT-L/14per video1.554.40056.155.9
MaxInfoCLIP ViT-L/14per video1.554.41056.855.0
WFS-SBCLIP ViT-L/14per video1.554.41054.854.7
F²CCLIP ViT-L/14per video1.554.41054.154.3
FULCRUMCLIP ViT-L/14per video1.554.41062.559.8
FOCUSBLIP ITMper question2.815.67052.953.1
VTR-VLMCLIP + answererper question3.556.40150.450.0
MDP3SigLIPper video4.056.95055.954.6
AKSBLIP ITMper question6.509.36055.654.6
DIGanswererper question8.5311.38≈2054.953.4
AdaRD-KeyBLIP-2 ITMper question11.6014.46059.454.5

Only uniform sampling is cheaper than FULCRUM. The six other methods that run one CLIP pass per video cost the same, within 0.01 s. The remaining six cost 1.3 to 3.3 times as much. FULCRUM's selection step itself takes 0.007 s. With one question per video, so nothing is shared, FULCRUM takes 5.61 s, still below AKS (9.36 s) and AdaRD-Key (14.46 s).

Table A2. Fewer frames, less time (Qwen2.5-VL-7B, LVB).

ConfigurationTotal (s/q)LVB
Uniform, 32 frames11.2152.9
AdaRD-Key, 32 frames22.8260.1
FULCRUM, 4 frames3.0260.8

FULCRUM with 4 frames is 3.7× faster than uniform sampling with 32 frames and 7.9 points more accurate.

Table A3. Best accuracy within each latency budget (Qwen2.5-VL-7B; each method uses its best k that fits).

Budget (s/q)BenchmarkUniformBest competitorFULCRUM
3.1LVB44.2 (k=8)53.7 (QAaF, k=4)60.8 (k=4)
3.1VMME43.3 (k=8)52.5 (QAaF, k=4)57.5 (k=4)
5LVB44.2 (k=8)56.8 (MaxInfo, k=8)62.5 (k=8)
5VMME43.3 (k=8)55.9 (QAaF, k=8)59.8 (k=8)
10LVB48.8 (k=16)58.3 (QAaF, k=16)63.2 (k=16)
10VMME46.5 (k=16)58.0 (F²C, k=16)61.2 (k=16)
15LVB52.9 (k=32)59.8 (F²C, k=32)63.2 (k=32)
15VMME51.2 (k=32)59.0 (F²C, k=32)61.2 (k=16)

FULCRUM is the most accurate at every budget.

AuthorsOpenReview ↗

Additional Experiments (Part 2): QAaF and the Contrast Statistic

B. QAaF and F²C under our protocol (EnTX W1, Q1)

Both use the same CLIP ViT-L/14 encoder, the same 128-frame pool, and the same answerers and prompts as every other method. The other methods are in Table 1 of the paper.

Table B1. Accuracy (%).

AnswererMethodLVB 4LVB 8LVB 16LVB 32VMME 4VMME 8VMME 16VMME 32
Qwen2.5-VL-7BQAaF53.756.158.357.452.555.957.858.2
F²C51.854.158.159.851.554.358.059.0
FULCRUM60.862.563.263.257.559.861.260.8
LLaVA-Video-7BQAaF55.358.359.259.354.457.658.859.3
F²C50.353.857.156.750.253.857.156.0
FULCRUM62.562.564.264.260.461.561.561.4

FULCRUM beats QAaF in all 16 settings, by 2.1 to 7.2 points, and F²C in all 16, by 1.8 to 12.2 points.

Table B2. Where the gap to QAaF comes from (k=8).

VariantLVB (Qwen)VMME (Qwen)LVB (LLaVA)VMME (LLaVA)
QAaF: best-match score, top-k frames56.155.958.357.6
Best-match score + our allocation and diversity60.157.960.560.0
+ contrast term (FULCRUM)62.559.862.561.5

On LVB with Qwen, the 6.4-point gap splits into 4.0 points from our allocation and diversity and 2.4 points from the contrast term. With everything else the same, the contrast term adds 1.5 to 2.4 points in all four settings.

C. Choice of contrast statistic (EnTX W1, Q2)

Table C1. Only the contrast term changes; everything else is FULCRUM (k=8).

Contrast termLVB (Qwen)VMME (Qwen)LVB (LLaVA)VMME (LLaVA)
Max − mean (FULCRUM)62.559.862.561.5
Neg. entropy, τ=0.01 (CLIP's temperature)62.059.462.161.2
Neg. entropy, τ=0.0561.859.361.961.1
Neg. entropy, τ=161.859.261.961.0
Variance across options61.859.261.961.0
Top-two margin61.759.161.861.0
No contrast term60.157.960.560.0

Every contrast term beats having none, so measuring how a frame separates the options is what matters. Max − mean is the best in all four settings:

  • The top-two margin ignores every option below the second.
  • The variance scores a frame that rejects one option the same as a frame that supports one.
  • Entropy needs a temperature, and its results change with it. At τ=1 it ranks frames the same way as the variance, which is why those two rows match.
  • Max − mean needs no temperature. By Eq. 3 it is the log-odds of the favored option against the geometric mean of all options, and the within-video standardization of §3.2 removes the remaining 1/τ factor.
AuthorsOpenReview ↗

Additional Experiments (Part 3): Larger Budgets, Larger Pools, LVBench, Token Compression

D. Larger budgets and pools (9pEA W2, Q1; mzWL W2; EnTX Q3)

Table D1. Budgets up to 128 frames, from the 128-frame pool. At k=128 every method uses the whole pool, so all methods tie.

Answerer, benchmarkMethodk=48163264128
Qwen, LVBUniform42.744.248.852.955.456.6
AdaRD-Key55.259.460.860.159.256.6
QAaF53.756.158.357.457.856.6
FULCRUM60.862.563.263.260.956.6
Qwen, VMMEUniform39.643.346.551.254.756.5
AdaRD-Key49.354.557.958.258.456.5
QAaF52.555.957.858.258.456.5
FULCRUM57.559.861.260.859.856.5
LLaVA, LVBUniform44.447.351.253.855.356.0
AdaRD-Key57.859.561.162.259.856.0
QAaF55.358.359.259.358.256.0
FULCRUM62.562.564.264.260.856.0
LLaVA, VMMEUniform41.645.148.352.654.655.6
AdaRD-Key55.258.359.660.158.655.6
QAaF54.457.658.859.358.255.6
FULCRUM60.461.561.561.459.355.6

FULCRUM is the most accurate method at every budget up to 64. Its lead shrinks at 64 because it must pick half of the pool. Table D2 uses a larger pool.

Table D2. Larger candidate pools (Qwen2.5-VL-7B). Cells are uniform / FULCRUM. Uniform spreads its frames over the whole video, so it does not depend on the pool.

PoolBenchmarkk=32k=64k=128Encode (s/video)
128LVB52.9 / 63.255.4 / 60.956.6 / 56.62.75
256LVB52.9 / 63.655.4 / 63.556.6 / 62.15.5
512LVB52.9 / 64.255.4 / 63.756.6 / 63.711.0
128VMME51.2 / 60.854.7 / 59.856.5 / 56.52.75
256VMME51.2 / 61.054.7 / 61.556.5 / 61.55.5
512VMME51.2 / 61.254.7 / 61.756.5 / 61.711.0

A pool of about four times the budget is enough. At the paper's budgets (up to 32), the 128-frame pool is within 1.0 point of a pool four times larger.

Table D3. LVBench (103 videos, 1,549 questions, mean 68 min). Same configuration as the paper, nothing tuned.

AnswererkUniformAdaRD-KeyAKSFULCRUM
Qwen2.5-VL-7B429.134.338.039.8
831.737.940.942.0
1635.641.242.344.2
3239.242.543.545.0
LLaVA-Video-7B431.737.937.742.5
834.240.140.643.5
1638.243.243.345.9
3241.745.444.047.3

FULCRUM is the most accurate at every budget with both answerers.

E. Token compression (9pEA Q3)

Table E1. Qwen2.5-VL-7B. Visual tokens are relative to 32 uncompressed frames.

FramesCompressorVisual tokensLVBVMME
32 uniformnone100%52.951.2
32 uniformVisionZip, keep 50%50%52.250.9
16 FULCRUMnone50%63.261.2
32 FULCRUMVisionZip, keep 50%50%63.060.2
32 FULCRUMFastV, keep 50%55%62.759.4

At the same token budget, choosing frames with FULCRUM beats compressing uniform frames by 11.0 points on LVB. Compressing FULCRUM's frames costs 0.2 to 1.8 points, so the two work together.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.