- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
SetGain selects a compact frame set for long-video QA. Candidate proposal works in two steps: a VLM decomposes the question (with options) into event captions, and the top-128 frames by best-event SigLIP similarity form the pool. A frozen cross-encoder (Qwen3-VL-Reranker-2B) then scores each candidate (). A 2.17M-parameter bidirectional Transformer over all candidate tokens, conditioned on the selected set (flag and max-similarity ), predicts a residual . Selection greedily adds with until a stop head fires, with the budget bounded to 4–8 frames.
Training proceeds in two stages:
- Marginal coverage regression. The selector regresses annotation-derived marginal temporal-coverage targets and stop targets (Seeker-173K).
- CGPO. A GRPO stage over Gumbel-perturbed selection trajectories, with a reward that combines answer correctness, a likelihood-margin term relative to uniform sampling of the pool, and a soft- evidence-coverage term, plus a stop-BCE loss to prevent budget collapse.
On Haystack-LVBench, SetGain improves temporal F1 over cross-encoder top- (16.0 → 18.1) with 6.7 vs. 8 frames, reaches 2.23× the F1 of TimeSearch-R, improves QA accuracy over TimeSearch-R on LongVideoBench, MLVU, and VideoMME with the same frozen answerer, and runs 2.15× faster end to end than T*.
Critical strengths
- Controlled comparisons. Top-, MMR, and DPP share the pool, the scorer, and the answerer, and Fig. 4b compares them at matched budgets. The authors explicitly acknowledge that much of the gap to prior work comes from the scorer.
- Well-motivated training signals. Marginal-coverage targets supervise the quantity the selector should estimate. The stop-BCE analysis (Fig. 5b) convincingly shows budget collapse without it.
- Efficiency. The method trains 2.17M parameters in about 26 GPU-hours on RTX 3090s, with transparent latency/TFLOPs accounting (Table 2) and costs (App. D).
- Ablations and analysis. The cumulative set-feature ablation is informative (Table 3), the analysis of what the stop head reads is useful (Table 6), and the failure modes are discussed honestly (App. C.4).
Critical weaknesses
W1. The effect of the proposed component is small relative to the headline claims, and no variance is reported. The headline numbers ("2.23× temporal F1", "+8.2 QA" vs. TimeSearch-R) mainly reflect the event-decomposed SigLIP pool and the Qwen3-VL-Reranker-2B scorer. Cross-encoder top- alone already reaches F1 16.0 vs. 8.1 and QA 55.3 vs. 52.3. The contribution of set conditioning is +2.1 F1 over top- and +1.3 F1 over an identically trained pointwise head (Table 3). All results are single runs without seeds or confidence intervals, and the number of Haystack-LVBench questions is not stated. Please report multiple seeds and a paired bootstrap for Fig. 4a, Table 3, and Table 1, and foreground the controlled numbers in the abstract.
W2. Precision gains are partly mechanical, and the "adaptive" budget is narrow.
- With accurate top-ranked candidates, selecting fewer frames raises precision almost by construction. The budget is clamped to .
- Mean rises monotonically with the number of selected frames (Table 5: 0.018 → 0.387, against a 0.5 threshold), suggesting the stop head largely learns a count prior. Table 6 shows some dependence on frame identity.
Please report the distribution of selected (how many questions hit 4 or 8), QA accuracy at matched against top-/MMR/DPP, and results with a wider budget (e.g., up to 16 or 32), where redundancy should matter more.
W3. The training reward is aligned with the evaluation metric. is a soft over temporal compatibility with the annotated segments. The evaluation temporal P/R/F1 also scores matches to annotated segments (5-s matching). SetGain is thus trained on a smoothed version of the evaluation metric, while TimeSearch-R and T* are not. The temporal-F1 gains are therefore partly expected, and QA accuracy is the more meaningful outcome. Fig. 5a shows that removing drops F1 below the initialization, which is consistent with this.
W4. Downstream QA (Table 1) lacks the controlled baselines. Table 1 has no Qwen2.5-VL-7B uniform-8/32 baseline and no cross-encoder top-, MMR, or DPP on LongVideoBench, MLVU, or VideoMME. Without them, the +2.9 average over TimeSearch-R cannot be attributed to set conditioning rather than to the scorer. The VideoMME (+1.3) and MLVU (+0.7) margins are also small.
W5. Positioning against learned subset selection. Greedy selection by marginal gain under facility-location-style coverage is classical submodular maximization. Learned, set-conditioned frame selection has been studied in video summarization: sequential DPPs (Gong et al., 2014), learning mixtures of submodular objectives (Gygli et al., 2015), and DPP-LSTM (Zhang et al., 2016). The feature is essentially MMR's redundancy term. Please discuss these and, ideally, compare with a learned-kernel DPP trained on the same coverage targets.
W6. Dependence on multiple-choice options. The event decomposition emits "one caption PER option" (App. E.1), so the candidate pool explicitly retrieves each option's scene, and the cross-encoder query includes the options. This is fair when baselines also see options, but it limits the method to MCQ. How does SetGain behave on open-ended questions or without options in the proposal step?
W7. Generality. All results use one answerer (Qwen2.5-VL-7B) in an 8-frame regime. Does set conditioning still help with a stronger answerer or larger budgets? The failure mode in App. C.4 (stopping inside the anchor scene) suggests the selection is dominated by similarity to the described scene. Per-event coverage tracking, which the authors propose as future work, may be necessary.
Questions for authors
- How many questions does Haystack-LVBench contain, and what are the seeds and CIs for Fig. 4a, Table 1, and Table 3?
- What do uniform-8/32, cross-encoder top-, MMR, and DPP achieve with the same Qwen2.5-VL-7B answerer on LongVideoBench, MLVU, and VideoMME?
- What is the distribution of selected at inference? How often does selection hit ?
- How does SetGain perform without options in the event decomposition and evidence-scoring prompts?
- Were the TimeSearch-R temporal P/R/F1 values taken from the original paper or re-computed under your protocol?
2 comments
I have no further questions and will maintain the score.
We thank the reviewer for the careful questions. All SetGain results below use the updated checkpoint (see summary).
Q1. Size of Haystack-LVBench, and seeds and confidence intervals for Fig. 4a, Table 1, and Table 3.
Haystack-LVBench has 342 questions. Standard errors for Table 1 are in H5Xc Q1 and Zv7p Q1, and for Fig. 4a in rx4f Q2.
All SetGain results come from a single training run with seed 0, so the standard errors capture variation across questions but not across training runs (App. C.1).
Submitted Table 3 is now Table 4. On Haystack-LVBench, each row adds one component, with both stages retrained per variant:
A pointwise head trained with the same pipeline improves only slightly over top-k (16.4 vs. 16.0 F1) and selects all eight frames. Set conditioning adds 1.4 F1 over this identically trained head (17.8 vs. 16.4; 8.5% relative), improving both precision (+2.1) and recall (+1.9) with 22% fewer frames. This gain is close to one standard error and the steps between rows are smaller, so we read the table as the combined effect of set conditioning, not a ranking of its components. The training-component ablation (revised Table 3) also reports standard errors.
Q2. Uniform-8/32, cross-encoder top-k, MMR, and DPP with the same Qwen2.5-VL-7B answerer on LongVideoBench, MLVU, and VideoMME.
Please see H5Xc Q1 (uniform-8/32, top-k, MMR, DPP) and Zv7p Q1 (top-k, MMR, DPP). All use SetGain's frozen answerer; top-k, MMR, and DPP also share its pool and reranker.
Q3. Distribution of the selected k at inference, and how often selection hits the 4- or 8-frame bound.
The updated checkpoint lowers the minimum from four to three frames. On Haystack-LVBench, selection hits this lower bound for 27.8% of questions and the upper bound for 52.0%, and the remaining 20.2% stop at 4–7 frames. LongVideoBench reaches the cap more often (78.1%). As App. D.2 shows, the stopping head acts mainly as a question-level gate: about 28% of Haystack questions exceed τ after the first frame and stop at the minimum, while most others fill to eight. The gate tracks how localized the question is: on MLVU, needle questions select 5.41 frames on average, against 7.92 for topic reasoning and 7.90 for action counting (App. D.1). The budget is therefore set per question by how localized its evidence is, rather than by a fixed count. The 52.0% at the cap suggests that many questions benefit from more evidence, consistent with the QA gain from 62.0 to 62.6 when SetGain uses all eight frames (Reviewer H5Xc, Q2).
Q4. SetGain without options in the event-decomposition and evidence-scoring prompts.
Please see Reviewer H5Xc, Q3. The question-level pool removes the option-based event decomposition and retrieves candidates from the question alone. With this pool, SetGain is 0.8–4.7 QA points below the full pipeline across the four benchmarks (2.1 on average), yet it still exceeds top-k with the decomposed pool on every benchmark (+1.9 to +2.6). The options therefore help the proposal step but are not what drives SetGain's gains. The evidence scorer still receives the options, as do all baselines, since every benchmark we evaluate is multiple-choice.
Q5. Were the TimeSearch-R temporal P/R/F1 values taken from the original paper or recomputed under our protocol?
In the submission, they were taken from the original paper [1], which follows the same LV-Haystack protocol [2]. We have since rerun the released checkpoint under our protocol; the revised Fig. 4a reports this rerun, which also produced the frames behind its QA accuracy. Haystack-LVBench, mean ± standard error:
The rerun is slightly lower than the published values, so the submitted numbers did not favor SetGain: SetGain reaches 2.31× TimeSearch-R's temporal F1 against the rerun and 2.20× against the published value. App. C.7 (Table 14) reports both.
[1] Pan et al. TimeSearch-R: Adaptive Temporal Search for Long-Form Video Understanding via Self-Verification Reinforcement Learning. ICLR 2026.
[2] Ye et al. Re-thinking Temporal Search for Long-Form Video Understanding. CVPR 2025.