- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CueKFS is a training-free keyframe selection method for long-video QA. A reasoning VLM decomposes the question into visual cues using a compact video overview. Each cue retrieves relevant frames via SigLIP2 similarity, after which the VLM refines the cues (keep/rewrite/split/drop) based on the retrieved evidence. The refined cues guide a second retrieval stage, with the frame budget allocated across cues. CueKFS consistently outperforms WFS-SB on Video-MME, MLVU, and LongVideoBench across three downstream VLMs and three frame budgets, requiring a median of only two VLM calls.
Critical strengths
- Well-motivated approach. Refining cues based on retrieved evidence addresses the limitations of static question decomposition.
- Training-free and efficient. Batched cue refinement keeps VLM calls low, with consistent gains across 27 settings, including with a strong downstream VLM (GPT-5.5).
- Thorough analysis. The paper provides extensive ablations, behavioral analyses (coverage shift and action frequency), and a candid discussion of failure cases.
Critical weaknesses
- Potential test-set tuning. Hyperparameters appear to be tuned on Video-MME, which is also used for evaluation in Table 1. This contributes approximately 1.7 points to the reported gain, raising concerns about generalization.
- Unfair comparisons. CueKFS uses GPT-5.5 with access to the question, answer options, and multiple frames, while most baselines rely only on embedding models. Cost-matched comparisons with related methods (T*, MSJoE, VideoAgent) and a direct GPT-5.5 frame-selection baseline are missing. Per-question latency and API cost are also unreported.
- Limited novelty and uncertain gains. The method primarily combines existing techniques and heuristics, with approximately 15 hyperparameters. Gains at k=32 are small, with no variance or significance analysis to establish their reliability.
Questions for authors
- Were the hyperparameters in Table 3 tuned on the same Video-MME test set used in Table 1? If so, how does performance change with held-out tuning or default settings?
- How much of the gain depends on GPT-5.5 as the reasoning VLM? Could you reproduce Table 1 using Qwen3.6-27B for cue reasoning across all downstream models, and include a baseline where GPT-5.5 directly selects k frames from the overview?
- How does CueKFS compare with T*, MSJoE, and VideoAgent under matched VLM costs? Could you report per-question latency, input image counts, and API cost relative to WFS-SB?
- How does performance change when answer options are excluded from cue decomposition and refinement?
- Could you report variance across runs or statistical significance, particularly at k=32, where improvements are small?
5 comments
Weakness 2 / Question 3. VLM cost and selector latency
On Video-MME at k=16, we fix GPT-5.5 as the reader and use Qwen3.6-27B on one H200 for VLM-based selection. We report accuracy and VLM cost for each method’s selection workload, using VLM runtime for Qwen and API charges for GPT-5.5. CueKFS uses the Table 1 settings, and A.I.R. retains its default analysis image cap and batches its per-frame requests. Because the selectors all use an embedding or relevance-extraction stage, we time selection from decoded frames and precomputed features or relevance scores to the selected frame IDs. VLM time covers prefill and decoding, including VideoAgent’s caption generation. Selector latency also covers image encoding, retrieval, MSJoE’s learned sampler, and T*’s detector search.
| Method | Qwen calls/q | Qwen input / output tokens/q | Qwen input images/q | Qwen VLM time (s/q) | Selector latency after embeddings (s/q) | GPT-5.5 selector API ($/q) | Acc. with Qwen (%) | Acc. with GPT-5.5 (%) |
|---|---|---|---|---|---|---|---|---|
| WFS-SB | 0 | 0 / 0 | 0 | 0 | 0.2 | 0 | 78.74 | 78.74 |
| T* | 1 | 3,811 / 160 | 32 | 2.32 | 4.3 | 0.0378 | 79.30 | 79.81 |
| MSJoE | 1 | 685 / 192 | 8 | 2.20 | 2.3 | 0.0191 | 78.41 | 78.81 |
| A.I.R. | 19 (batched) | ≈16,700 / ≈550 | 19 | 3.1 | 4.1 | 0.1107 | 78.67 | 79.19 |
| VideoAgent | 8 | ≈4,700 / ≈774 | ≈8 | 9.1 | 9.7 | 0.0475 | 79.15 | 79.67 |
| CueKFS | 2 | ≈10,100 / ≈320 | ≤80 with one refinement batch | 5.02 | 5.9 | 0.0743 | 82.52 | 83.30 |
CueKFS scores +3.37 points over VideoAgent at 5.9 vs. 9.7 s/question of selector latency after embeddings. Relative to WFS-SB, CueKFS adds 5.7 s/question of selector time and gains 3.78 points with Qwen. Its margins over the strongest alternative are +3.22 points with Qwen and +3.49 points with GPT-5.5 as the selector.
Question 4. Performance without options
On Video-MME, we remove the answer options from the selector's prompts.
| Selector | k=8 (%) | k=16 (%) | k=32 (%) |
|---|---|---|---|
| WFS-SB | 76.22 | 78.74 | 80.96 |
| CueKFS, question only | 79.78 | 82.78 | 83.85 |
| CueKFS, question and options | 80.67 | 83.30 | 84.15 |
At k=16, removing the options reduces CueKFS accuracy by 0.52 points, leaving it 4.04 points ahead of WFS-SB. Options can be essential to cue decomposition and refinement when they specify what evidence to retrieve. For example, in “Which of the following did not happen in the video?”, the question alone leaves the candidate events unspecified.
Weakness 3 / Question 5. Novelty and significance of the gains
We respectivefully disagree with the comment of "primarily combines existing techniques and heuristics". Our novelty is that CueKFS dynamically adjusts the visual evidence for retrieval under our proposed cue-wave calculation and budget allocation. Specifically, CueKFS inspects the frames found for each cue, revises or splits that cue, and searches the full candidate pool again (lines 211–224). This differs from T*, which changes where its detector searches for the initial targets, and MSJoE, which combines the initial queries for its sampler (lines 088–098). It also differs from VideoAgent’s use of captions to formulate subsequent retrieval queries. Our Peaks and Plateaus give the reasoner evidence from brief and sustained matches (lines 191–209). After the second search, Cue Budget Allocation preserves distinct evidence by distributing the final frame budget across cues.
Table 2 measures these contributions under controlled settings. Adding refinement improves average accuracy by 1.78 points. Within refinement, replacing overview frames with cue-wave evidence adds 2.94 points. Allocating frames across the resulting cues adds 2.38 points over global top-k selection. On long videos, the final cues find stronger matches in the full candidate pool than in the overview, with a 53% higher median SigLIP2 score. Refinement also changes which frames the cues cover, with a median Jaccard overlap of 0.65 between the initial and final frame sets (Tables 5–6). Moreover, in our response to the other reviewers, replacing our adjacent-frame segmentation with WFS-SB’s one changes accuracy from 80.63% to 80.18%, and removing CD, IS, and BO changes it to 79.48%. These further support the effectiveness of our design choices.
With all these evidences, we believe our innovation is clear, motivated, and effective.
Regarding the gains on k=32, we want to hightlight that our gains on k=32 exceed three points on two benchmarks, which is not small. We also want to mention that a larger frame budget gives the reader more chances to see relevant evidence despite imperfect selections, which makes the difference between methods relatively less significant on k=32 by design. Even though, CueKFS still improves over WFS-SB by 3.19 points on Video-MME and 3.22 points on LongVideoBench with GPT-5.5. On MLVU, the gain with InternVL3 is 2.84 points.
To assess variance across runs and statistical significance at k=32, we compare three independent Video-MME runs with the GPT-5.5 reader and pair the accuracy differences between WFS-SB and CueKFS by video.
| WFS-SB accuracy (mean ± SD, %) | CueKFS accuracy (mean ± SD, %) | Paired gain (points) | 95% CI (points) | p-value |
|---|---|---|---|---|
| 80.93 ± 0.15 | 84.14 ± 0.24 | +3.21 | [+1.84, +4.58] | 3.76 × 10⁻⁶ |
The paired difference across runs is +3.21 points. The original video-level paired analysis also establishes statistical significance at k=32. Its 95% confidence interval of [+1.85, +4.59] points excludes zero.
Weakness 1 / Question 1. Generalization of tuning
The original Video-MME release provides only a public test split, so the sensitivity sweep in Table 3 uses the same split as the evaluation in Table 1. Even before hyperparameter tuning, CueKFS outperforms WFS-SB, the strongest Video-MME baseline in Table 1 with the GPT-5.5 reader. At the same frame budgets, CueKFS averages 81.04% with Qwen3.6-27B and 81.52% with GPT-5.5, compared with WFS-SB’s 78.64%.
The results on MLVU and LongVideoBench already demonstrate cross-benchmark generalization. With the Video-MME settings frozen and no tuning on either benchmark, CueKFS achieves 81.95% on MLVU and 77.51% on LongVideoBench, outperforming WFS-SB by 2.42 and 4.06 points, respectively.
To further test whether the selected hyperparameters generalize with held-out tuning, we repeat the full one-factor-at-a-time sweep from Table 3 on CG-Bench mini, a benchmark unused in our original experiments. We use the same selection criterion, averaging accuracy across frame budgets. The sweep selects the same values as Video-MME except for the deduplication thresholds. The cosine similarity threshold stays at 0.98, while the temporal window increases from 1 s to 3 s. We compare both CueKFS configurations and WFS-SB on the same CG-Bench mini split at k=16 with the GPT-5.5 reader.
The frozen configuration already outperforms WFS-SB by 3.3 points at the same frame budget. Repeating the sweep changes accuracy by only 0.2 points, supporting the transfer of the Video-MME configuration to CG-Bench mini.
Weakness 2 / Questions 1 and 2. Default settings, Qwen3.6-27B setting, and direct GPT-5.5 selection
To separate the effect of the VLMs, we evaluate Qwen3.6-27B with both the pre-sweep settings and the final Table 1 settings. In the direct-selection control, GPT-5.5 receives the same question, answer options, and overview as CueKFS and selects 16 frames from that overview, which is capped at 48 frames.
At the default settings, Qwen3.6-27B retains most of the gain over WFS-SB obtained with GPT-5.5 as the reasoner. On Video-MME at k=16, CueKFS with Qwen gains 1.89 points over WFS-SB with the same GPT-5.5 reader. Before the sweep, switching only the reasoner to GPT-5.5 adds 0.48 points on average across frame budgets (Table 3). With the final settings, the GPT-5.5–Qwen gap is +0.78 points On Video-MME at k=16.
Direct GPT-5.5 selection provides only a small gain over uniform sampling. On Video-MME with the GPT-5.5 reader, direct selection scores 77.41%, compared with 77.11% for uniform sampling and 78.74% for WFS-SB. A 48-frame overview can miss the specific objects, actions, or moments a question asks about. Embedding model can search the full candidate pool for that evidence, whereas direct selection can only rank the frames it already sees. With the same reasoner, reader, starting overview, and output budget, CueKFS gains 5.89 points over direct selection by refining its cues and retrieving evidence by embedding model.