- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces FULCRUM, a training-free frame selection method for multiple-choice long-video question answering. The main idea is to select frames based not only on question relevance but also on their ability to distinguish between candidate answers. The method computes CLIP similarities between frames and question–option pairs and combines maximum similarity with a contrastive score defined as the difference between the maximum and mean similarities across options. Temporal budget allocation and diversity-aware selection are then used to select informative, non-redundant frames.
Experiments on LongVideoBench and Video-MME with two VLMs and different frame budgets demonstrate consistent improvements over uniform sampling and ten existing frame selection methods.
Critical strengths
-
Strong empirical results. The method consistently outperforms the evaluated baselines across datasets, models, and frame budgets. Particularly notable is that four FULCRUM-selected frames outperform 32 uniformly sampled frames, demonstrating the potential of improved frame selection under constrained visual budgets.
-
Extensive experimental validation. The paper provides comprehensive ablations, particularly comparing question-only relevance, fused options, and option-conditioned relevance. These experiments help isolate the contribution of the proposed contrastive score. Additional evaluations across model sizes and encoders further support the robustness of the results.
Critical weaknesses
-
Limited methodological novelty and missing comparison with closely related work. The use of answer options to guide frame selection has already been investigated in prior work, particularly QAaF. The main novelty appears to be the maximum-minus-mean similarity score, combined with established temporal allocation and diversity mechanisms. I am not convinced that this constitutes a sufficiently substantial methodological contribution. Moreover, despite its close relationship to FULCRUM, QAaF is not included in the main experimental comparisons. The maximum-similarity ablation is informative but does not replace a comparison with the complete method. It is also unclear whether the proposed contrastive score offers meaningful advantages over other straightforward measures of discrimination, such as the margin between the two highest option similarities, the variance of similarities across options, or the entropy of the softmax-normalized similarities. These alternatives would respectively capture the separation between the strongest competing hypotheses, the overall spread of option scores, and the uncertainty across candidate answers.
-
Insufficient computational efficiency analysis. Although the method achieves strong accuracy with fewer selected frames, the frame selection process introduces additional computational overhead, including encoding up to 128 candidate frames with CLIP. The paper does not quantify the resulting end-to-end computational savings. Given the emphasis on efficient long-video understanding, reporting actual inference latency, computational cost, and memory usage, together with accuracy–latency comparisons against strong baselines, would be important for evaluating the practical significance of the method.
Questions for authors
-
How does FULCRUM compare against the complete QAaF method under the same evaluation protocol and computational budget? This comparison seems important for establishing the contribution relative to closely related work.
-
What motivates the choice of maximum-minus-mean similarity over other discrimination measures, such as the top-two margin (difference between the highest and second-highest option similarities), similarity variance (spread of the scores across options), or softmax entropy (uncertainty of the normalized option similarity distribution)? Would these alternatives achieve comparable performance when the remaining components are unchanged?
-
What is the end-to-end computational cost of FULCRUM, including candidate frame encoding and selection? How does its accuracy–latency trade-off compare against the strongest baselines, and how does the computational cost scale beyond 128 candidate frames?
4 comments
Additional Findings (Part 1)
Supporting Tables (1/2): QAaF and the Contrast Term (W1, Q1, Q2)
Table R1. Full QAaF under our protocol. Accuracy (%). The other methods are in Table 1 of the paper.
| Answerer | Method | LVB 4 | LVB 8 | LVB 16 | LVB 32 | VMME 4 | VMME 8 | VMME 16 | VMME 32 |
|---|---|---|---|---|---|---|---|---|---|
| Qwen2.5-VL-7B | Uniform | 42.7 | 44.2 | 48.8 | 52.9 | 39.6 | 43.3 | 46.5 | 51.2 |
| AdaRD-Key | 55.2 | 59.4 | 60.8 | 60.1 | 49.3 | 54.5 | 57.9 | 58.2 | |
| QAaF | 53.7 | 56.1 | 58.3 | 57.4 | 52.5 | 55.9 | 57.8 | 58.2 | |
| FULCRUM | 60.8 | 62.5 | 63.2 | 63.2 | 57.5 | 59.8 | 61.2 | 60.8 | |
| LLaVA-Video-7B | Uniform | 44.4 | 47.3 | 51.2 | 53.8 | 41.6 | 45.1 | 48.3 | 52.6 |
| AdaRD-Key | 57.8 | 59.5 | 61.1 | 62.2 | 55.2 | 58.3 | 59.6 | 60.1 | |
| QAaF | 55.3 | 58.3 | 59.2 | 59.3 | 54.4 | 57.6 | 58.8 | 59.3 | |
| FULCRUM | 62.5 | 62.5 | 64.2 | 64.2 | 60.4 | 61.5 | 61.5 | 61.4 |
Table R2. Where the gap to QAaF comes from (k=8).
| Variant | LVB (Qwen) | VMME (Qwen) | LVB (LLaVA) | VMME (LLaVA) |
|---|---|---|---|---|
| QAaF: best-match score, top-k frames | 56.1 | 55.9 | 58.3 | 57.6 |
| Best-match score + our allocation and diversity | 60.1 | 57.9 | 60.5 | 60.0 |
| + contrast term (FULCRUM) | 62.5 | 59.8 | 62.5 | 61.5 |
Table R3. Only the contrast term changes; everything else is FULCRUM (k=8).
| Contrast term | LVB (Qwen) | VMME (Qwen) | LVB (LLaVA) | VMME (LLaVA) |
|---|---|---|---|---|
| Max − mean (FULCRUM) | 62.5 | 59.8 | 62.5 | 61.5 |
| Neg. entropy, τ=0.01 (CLIP's temperature) | 62.0 | 59.4 | 62.1 | 61.2 |
| Neg. entropy, τ=0.05 | 61.8 | 59.3 | 61.9 | 61.1 |
| Neg. entropy, τ=1 | 61.8 | 59.2 | 61.9 | 61.0 |
| Variance across options | 61.8 | 59.2 | 61.9 | 61.0 |
| Top-two margin | 61.7 | 59.1 | 61.8 | 61.0 |
| No contrast term | 60.1 | 57.9 | 60.5 | 60.0 |
At τ=1 entropy ranks frames the same way as the variance, which is why those two rows match.
Additional Findings (Part 2)
Supporting Tables (2/2): Cost, Latency and Pool Size (W2, Q3)
Table R4. Cost per question at k=8 (T4 GPU, Qwen2.5-VL-7B). Answering takes 2.85 s for every method. Methods that run once per video share the encoder pass across the 1.78 questions per video of LongVideoBench.
| Method | Scorer | Runs | Total (s/q) | Answerer calls in selection | LVB | VMME |
|---|---|---|---|---|---|---|
| Uniform | — | — | 2.85 | 0 | 44.2 | 43.3 |
| BOLT | CLIP ViT-L/14 | per video | 4.40 | 0 | 52.1 | 51.7 |
| Q-Frame | CLIP ViT-L/14 | per video | 4.40 | 0 | 54.6 | 53.9 |
| QAaF | CLIP ViT-L/14 | per video | 4.40 | 0 | 56.1 | 55.9 |
| MaxInfo | CLIP ViT-L/14 | per video | 4.41 | 0 | 56.8 | 55.0 |
| WFS-SB | CLIP ViT-L/14 | per video | 4.41 | 0 | 54.8 | 54.7 |
| F²C | CLIP ViT-L/14 | per video | 4.41 | 0 | 54.1 | 54.3 |
| FULCRUM | CLIP ViT-L/14 | per video | 4.41 | 0 | 62.5 | 59.8 |
| FOCUS | BLIP ITM | per question | 5.67 | 0 | 52.9 | 53.1 |
| VTR-VLM | CLIP + answerer | per question | 6.40 | 1 | 50.4 | 50.0 |
| MDP3 | SigLIP | per video | 6.95 | 0 | 55.9 | 54.6 |
| AKS | BLIP ITM | per question | 9.36 | 0 | 55.6 | 54.6 |
| DIG | answerer | per question | 11.38 | ≈20 | 54.9 | 53.4 |
| AdaRD-Key | BLIP-2 ITM | per question | 14.46 | 0 | 59.4 | 54.5 |
Table R5. Accuracy against latency (Qwen2.5-VL-7B). Top: fewer frames, less time. Bottom: best accuracy within each latency budget, each method at its best k that fits.
| Configuration | Total (s/q) | LVB |
|---|---|---|
| Uniform, 32 frames | 11.21 | 52.9 |
| AdaRD-Key, 32 frames | 22.82 | 60.1 |
| FULCRUM, 4 frames | 3.02 | 60.8 |
| Budget (s/q) | Benchmark | Uniform | Best competitor | FULCRUM |
|---|---|---|---|---|
| 3.1 | LVB | 44.2 (k=8) | 53.7 (QAaF, k=4) | 60.8 (k=4) |
| 3.1 | VMME | 43.3 (k=8) | 52.5 (QAaF, k=4) | 57.5 (k=4) |
| 5 | LVB | 44.2 (k=8) | 56.8 (MaxInfo, k=8) | 62.5 (k=8) |
| 5 | VMME | 43.3 (k=8) | 55.9 (QAaF, k=8) | 59.8 (k=8) |
| 10 | LVB | 48.8 (k=16) | 58.3 (QAaF, k=16) | 63.2 (k=16) |
| 10 | VMME | 46.5 (k=16) | 58.0 (F²C, k=16) | 61.2 (k=16) |
| 15 | LVB | 52.9 (k=32) | 59.8 (F²C, k=32) | 63.2 (k=32) |
| 15 | VMME | 51.2 (k=32) | 59.0 (F²C, k=32) | 61.2 (k=16) |
Table R6. Larger candidate pools (Qwen2.5-VL-7B). Cells are uniform / FULCRUM. Uniform does not depend on the pool.
| Pool | Benchmark | k=32 | k=64 | k=128 | Encode (s/video) |
|---|---|---|---|---|---|
| 128 | LVB | 52.9 / 63.2 | 55.4 / 60.9 | 56.6 / 56.6 | 2.75 |
| 256 | LVB | 52.9 / 63.6 | 55.4 / 63.5 | 56.6 / 62.1 | 5.5 |
| 512 | LVB | 52.9 / 64.2 | 55.4 / 63.7 | 56.6 / 63.7 | 11.0 |
| 128 | VMME | 51.2 / 60.8 | 54.7 / 59.8 | 56.5 / 56.5 | 2.75 |
| 256 | VMME | 51.2 / 61.0 | 54.7 / 61.5 | 56.5 / 61.5 | 5.5 |
| 512 | VMME | 51.2 / 61.2 | 54.7 / 61.7 | 56.5 / 61.7 | 11.0 |
Thank you for the detailed response and additional experiments. I appreciate that the authors directly addressed my concerns, particularly through the full QAaF comparison, the ablations on alternative contrastive scores, and the end-to-end latency analysis.
I still think the methodological novelty is somewhat limited, especially since the alternative contrastive scores achieve quite similar results. However, the additional experiments provide convincing evidence of the method's effectiveness and practical value. Overall, I believe my main concerns have been sufficiently addressed, and I will raise my score.
Rebuttal by Authors
We thank the reviewer for recognising the strong and consistent results, in particular that four FULCRUM frames beat 32 uniform frames, and the ablations that isolate the contrast score. Tables R1–R6 are in the two replies below this comment.
W1 + Q1. Novelty and the full QAaF method. We agree that reading the answer options is not new, and the paper credits QAaF and VTR-VLM for it (§2, App. A). Both turn the options into one score per frame: the best-matching option (QAaF) or one fused text (VTR-VLM). FULCRUM instead scores how far a frame separates the options. Table 2 of the paper shows the difference: using the options without contrast adds 0.8 to 1.4 points over the question alone; the contrast adds 3.3 to 3.5.
Full QAaF (Table R1). We ran the complete QAaF method under our protocol: same CLIP encoder, same 128 candidate frames, same answerers and prompts. Its cost is the same as FULCRUM's (4.40 vs 4.41 s per question, Table R4). FULCRUM beats QAaF in all 16 settings, by 2.1 to 7.2 points.
Where the gap comes from (Table R2). QAaF differs from FULCRUM in two ways: no contrast term, and top-k selection instead of our allocation and diversity steps. Adding these one at a time, on LVB with Qwen: QAaF 56.1 → with our allocation and diversity 60.1 → with the contrast term 62.5. The contrast term adds 1.5 to 2.4 points in all four settings, with everything else unchanged. The gain needs the question's own options: options from another video's question, or a paraphrase of the question, fall below the question alone (Table 2 of the paper).
W1 + Q2. Why max − mean, and not margin, variance or entropy? We ran all three in place of max − mean, keeping everything else the same (Table R3). The reviewer's intuition is right that any measure of separation helps: every alternative beats having no contrast term. This supports our main claim. Max − mean is the best in all four settings, for clear reasons:
W2 + Q3. End-to-end cost and scaling beyond 128 frames.
Revisions. We will add QAaF to Table 1, the breakdown of the gap to QAaF, the comparison of contrast terms, the cost tables, and the pool-size results.
We hope these additions address the reviewer's concerns, and we respectfully ask the reviewer to consider updating their assessment.