acceptodds
Under review as a conference paper at ICLR 2027

SetGain: Set-Conditioned Adaptive Evidence Selection for Long Video Understanding

Abstract

Long-form video question answering requires selecting a compact yet sufficient evidence set from thousands of frames. Existing methods follow predefined search procedures or optimize selection through trajectory-level outcomes, without modeling how much new information a candidate contributes beyond the evidence already collected. We propose SetGain, a set-conditioned evidence-selection framework that jointly learns which frame to select next and when to stop. A gain model conditioned on the candidate pool and the evolving selected set estimates each remaining frame's marginal contribution, trained by supervised initialization with marginal temporal-coverage targets followed by reinforcement learning with answer-quality and evidence-coverage rewards. On Haystack-LVBench, SetGain improves temporal F1 by 13.1% relative over a pointwise top- selector using the same candidate pool and evidence scores, with 16% fewer frames; end to end it reaches the temporal F1 and 8.2 points higher QA accuracy (15.7% relative) than the strongest comparable prior method, TimeSearch-R. It further improves average QA accuracy by 2.9 points (4.7% relative) over TimeSearch-R on VideoMME, MLVU, and LongVideoBench, and reduces end-to-end latency by 53.5% ( speedup) relative to T*, the fastest prior search method. These results demonstrate that explicitly modeling marginal evidence gain improves temporal localization, downstream answering, and inference efficiency. Code is available at https://anonymous.4open.science/r/SetGain.

Primary area:
reinforcement learning
Keywords:
Temporal search, Reinforcement learning, Adaptive evidence selection, Long-form video understanding, Vision-language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_DoJxOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

SetGain selects a compact frame set for long-video QA. Candidate proposal works in two steps: a VLM decomposes the question (with options) into event captions, and the top-128 frames by best-event SigLIP similarity form the pool. A frozen cross-encoder (Qwen3-VL-Reranker-2B) then scores each candidate (sis_i). A 2.17M-parameter bidirectional Transformer over all candidate tokens, conditioned on the selected set (flag mim_i and max-similarity ρi\rho_i), predicts a residual Δi\Delta_i. Selection greedily adds arg⁡max⁡iGi\arg\max_i G_i with Gi=si+ΔiG_i = s_i + \Delta_i until a stop head fires, with the budget bounded to 4–8 frames.

Training proceeds in two stages:

  1. Marginal coverage regression. The selector regresses annotation-derived marginal temporal-coverage targets and stop targets (Seeker-173K).
  2. CGPO. A GRPO stage over Gumbel-perturbed selection trajectories, with a reward that combines answer correctness, a likelihood-margin term relative to uniform sampling of the pool, and a soft-FβF_\beta evidence-coverage term, plus a stop-BCE loss to prevent budget collapse.

On Haystack-LVBench, SetGain improves temporal F1 over cross-encoder top-kk (16.0 → 18.1) with 6.7 vs. 8 frames, reaches 2.23× the F1 of TimeSearch-R, improves QA accuracy over TimeSearch-R on LongVideoBench, MLVU, and VideoMME with the same frozen answerer, and runs 2.15× faster end to end than T*.

Critical strengths
  1. Controlled comparisons. Top-kk, MMR, and DPP share the pool, the scorer, and the answerer, and Fig. 4b compares them at matched budgets. The authors explicitly acknowledge that much of the gap to prior work comes from the scorer.
  2. Well-motivated training signals. Marginal-coverage targets supervise the quantity the selector should estimate. The stop-BCE analysis (Fig. 5b) convincingly shows budget collapse without it.
  3. Efficiency. The method trains 2.17M parameters in about 26 GPU-hours on RTX 3090s, with transparent latency/TFLOPs accounting (Table 2) and costs (App. D).
  4. Ablations and analysis. The cumulative set-feature ablation is informative (Table 3), the analysis of what the stop head reads is useful (Table 6), and the failure modes are discussed honestly (App. C.4).
Critical weaknesses

W1. The effect of the proposed component is small relative to the headline claims, and no variance is reported. The headline numbers ("2.23× temporal F1", "+8.2 QA" vs. TimeSearch-R) mainly reflect the event-decomposed SigLIP pool and the Qwen3-VL-Reranker-2B scorer. Cross-encoder top-kk alone already reaches F1 16.0 vs. 8.1 and QA 55.3 vs. 52.3. The contribution of set conditioning is +2.1 F1 over top-kk and +1.3 F1 over an identically trained pointwise head (Table 3). All results are single runs without seeds or confidence intervals, and the number of Haystack-LVBench questions is not stated. Please report multiple seeds and a paired bootstrap for Fig. 4a, Table 3, and Table 1, and foreground the controlled numbers in the abstract.

W2. Precision gains are partly mechanical, and the "adaptive" budget is narrow.

  • With accurate top-ranked candidates, selecting fewer frames raises precision almost by construction. The budget is clamped to [4,8][4, 8].
  • Mean pstopp_{\text{stop}} rises monotonically with the number of selected frames (Table 5: 0.018 → 0.387, against a 0.5 threshold), suggesting the stop head largely learns a count prior. Table 6 shows some dependence on frame identity.

Please report the distribution of selected kk (how many questions hit 4 or 8), QA accuracy at matched kk against top-kk/MMR/DPP, and results with a wider budget (e.g., up to 16 or 32), where redundancy should matter more.

W3. The training reward is aligned with the evaluation metric. RcovR_{\text{cov}} is a soft FβF_\beta over temporal compatibility with the annotated segments. The evaluation temporal P/R/F1 also scores matches to annotated segments (5-s matching). SetGain is thus trained on a smoothed version of the evaluation metric, while TimeSearch-R and T* are not. The temporal-F1 gains are therefore partly expected, and QA accuracy is the more meaningful outcome. Fig. 5a shows that removing RcovR_{\text{cov}} drops F1 below the initialization, which is consistent with this.

W4. Downstream QA (Table 1) lacks the controlled baselines. Table 1 has no Qwen2.5-VL-7B uniform-8/32 baseline and no cross-encoder top-kk, MMR, or DPP on LongVideoBench, MLVU, or VideoMME. Without them, the +2.9 average over TimeSearch-R cannot be attributed to set conditioning rather than to the scorer. The VideoMME (+1.3) and MLVU (+0.7) margins are also small.

W5. Positioning against learned subset selection. Greedy selection by marginal gain under facility-location-style coverage is classical submodular maximization. Learned, set-conditioned frame selection has been studied in video summarization: sequential DPPs (Gong et al., 2014), learning mixtures of submodular objectives (Gygli et al., 2015), and DPP-LSTM (Zhang et al., 2016). The ρi\rho_i feature is essentially MMR's redundancy term. Please discuss these and, ideally, compare with a learned-kernel DPP trained on the same coverage targets.

W6. Dependence on multiple-choice options. The event decomposition emits "one caption PER option" (App. E.1), so the candidate pool explicitly retrieves each option's scene, and the cross-encoder query includes the options. This is fair when baselines also see options, but it limits the method to MCQ. How does SetGain behave on open-ended questions or without options in the proposal step?

W7. Generality. All results use one answerer (Qwen2.5-VL-7B) in an 8-frame regime. Does set conditioning still help with a stronger answerer or larger budgets? The failure mode in App. C.4 (stopping inside the anchor scene) suggests the selection is dominated by similarity to the described scene. Per-event coverage tracking, which the authors propose as future work, may be necessary.

Questions for authors
  1. How many questions does Haystack-LVBench contain, and what are the seeds and CIs for Fig. 4a, Table 1, and Table 3?
  2. What do uniform-8/32, cross-encoder top-kk, MMR, and DPP achieve with the same Qwen2.5-VL-7B answerer on LongVideoBench, MLVU, and VideoMME?
  3. What is the distribution of selected kk at inference? How often does selection hit kmax⁡=8k_{\max} = 8?
  4. How does SetGain perform without options in the event decomposition and evidence-scoring prompts?
  5. Were the TimeSearch-R temporal P/R/F1 values taken from the original paper or re-computed under your protocol?
2 comments

We thank the reviewer for the careful questions. All SetGain results below use the updated checkpoint (see summary).

Q1. Size of Haystack-LVBench, and seeds and confidence intervals for Fig. 4a, Table 1, and Table 3.

Haystack-LVBench has 342 questions. Standard errors for Table 1 are in H5Xc Q1 and Zv7p Q1, and for Fig. 4a in rx4f Q2.

All SetGain results come from a single training run with seed 0, so the standard errors capture variation across questions but not across training runs (App. C.1).

Submitted Table 3 is now Table 4. On Haystack-LVBench, each row adds one component, with both stages retrained per variant:

VariantT-PT-RT-F1Frames
Cross-encoder + top-k14.8 ± 1.622.2 ± 2.016.0 ± 1.68.00
Trained head, no set state15.3 ± 1.621.9 ± 2.016.4 ± 1.68.00
+ selected flag m18.2 ± 1.820.6 ± 2.017.4 ± 1.75.65
+ ρ18.4 ± 1.821.3 ± 2.017.7 ± 1.75.99
+ pool context (SetGain, full)17.4 ± 1.723.8 ± 2.117.8 ± 1.66.24

A pointwise head trained with the same pipeline improves only slightly over top-k (16.4 vs. 16.0 F1) and selects all eight frames. Set conditioning adds 1.4 F1 over this identically trained head (17.8 vs. 16.4; 8.5% relative), improving both precision (+2.1) and recall (+1.9) with 22% fewer frames. This gain is close to one standard error and the steps between rows are smaller, so we read the table as the combined effect of set conditioning, not a ranking of its components. The training-component ablation (revised Table 3) also reports standard errors.

Q2. Uniform-8/32, cross-encoder top-k, MMR, and DPP with the same Qwen2.5-VL-7B answerer on LongVideoBench, MLVU, and VideoMME.

Please see H5Xc Q1 (uniform-8/32, top-k, MMR, DPP) and Zv7p Q1 (top-k, MMR, DPP). All use SetGain's frozen answerer; top-k, MMR, and DPP also share its pool and reranker.

Q3. Distribution of the selected k at inference, and how often selection hits the 4- or 8-frame bound.

BenchmarkMean framesAt 34–7At 8
Haystack-LVBench6.2427.8%20.2%52.0%
LongVideoBench7.379.5%12.4%78.1%

The updated checkpoint lowers the minimum from four to three frames. On Haystack-LVBench, selection hits this lower bound for 27.8% of questions and the upper bound for 52.0%, and the remaining 20.2% stop at 4–7 frames. LongVideoBench reaches the cap more often (78.1%). As App. D.2 shows, the stopping head acts mainly as a question-level gate: about 28% of Haystack questions exceed τ after the first frame and stop at the minimum, while most others fill to eight. The gate tracks how localized the question is: on MLVU, needle questions select 5.41 frames on average, against 7.92 for topic reasoning and 7.90 for action counting (App. D.1). The budget is therefore set per question by how localized its evidence is, rather than by a fixed count. The 52.0% at the cap suggests that many questions benefit from more evidence, consistent with the QA gain from 62.0 to 62.6 when SetGain uses all eight frames (Reviewer H5Xc, Q2).

Q4. SetGain without options in the event-decomposition and evidence-scoring prompts.

Please see Reviewer H5Xc, Q3. The question-level pool removes the option-based event decomposition and retrieves candidates from the question alone. With this pool, SetGain is 0.8–4.7 QA points below the full pipeline across the four benchmarks (2.1 on average), yet it still exceeds top-k with the decomposed pool on every benchmark (+1.9 to +2.6). The options therefore help the proposal step but are not what drives SetGain's gains. The evidence scorer still receives the options, as do all baselines, since every benchmark we evaluate is multiple-choice.

Q5. Were the TimeSearch-R temporal P/R/F1 values taken from the original paper or recomputed under our protocol?

In the submission, they were taken from the original paper [1], which follows the same LV-Haystack protocol [2]. We have since rerun the released checkpoint under our protocol; the revised Fig. 4a reports this rerun, which also produced the frames behind its QA accuracy. Haystack-LVBench, mean ± standard error:

TimeSearch-RFramesPRTemporal F1
Published [1]8.85.422.38.1
Rerun under our protocol9.45.3 ± 0.620.2 ± 1.97.7 ± 0.8

The rerun is slightly lower than the published values, so the submitted numbers did not favor SetGain: SetGain reaches 2.31× TimeSearch-R's temporal F1 against the rerun and 2.20× against the published value. App. C.7 (Table 14) reports both.

[1] Pan et al. TimeSearch-R: Adaptive Temporal Search for Long-Form Video Understanding via Self-Verification Reinforcement Learning. ICLR 2026.

[2] Ye et al. Re-thinking Temporal Search for Long-Form Video Understanding. CVPR 2025.

I have no further questions and will maintain the score.

Reviewer_Zv7pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SetGain selects frames for long-video question answering from a 128-frame candidate pool. A frozen VLM first breaks the question into events; a frozen encoder proposes frames for those events, and a frozen reranker scores them. A small Transformer estimates what each frame adds to the selected set and when to stop. Training combines annotation-derived marginal-coverage targets with answer and evidence rewards. On Haystack-LVBench, it improves temporal F1 from 16.0 to 18.1 and QA accuracy from 55.3% to 60.5% against top-k using the same pool, scorer, and answerer, while selecting 6.7 rather than 8 frames.

Critical strengths
  1. The controlled Haystack comparison makes it possible to see what the selector adds. Against top-k with the same candidates, scores, and answerer, SetGain improves both localization and QA while using fewer frames (Figure 4a). Table 3 then retrains a pointwise head under the same pipeline, which is a useful check on the contribution of set conditioning.

  2. The complete system is fast in the authors' setup. The reported 2.17-million-parameter selector adds little computation after candidate scoring, and Table 2 reports 11.40 seconds per question for the full search-and-answer pipeline versus 28.79 seconds for TimeSearch-R. This is a useful system-level result, even though the two search pipelines differ.

Critical weaknesses
  1. The evidence that the selection rule improves QA beyond Haystack is thin. Top-k, MMR, and DPP are compared with SetGain at matched budgets on Haystack, but not on LongVideoBench, MLVU, or VideoMME. An identically trained pointwise head already gets 16.8 temporal F1 in Table 3, leaving 1.3 points for the added set features. On MLVU and VideoMME, the reported QA gains over TimeSearch-R are 0.7 and 1.3 points, without uncertainty estimates and with a different candidate/scoring pipeline.

  2. The paper does not establish the ceiling imposed by its first-stage candidate pool. The global top-128 proposal can underrepresent an event even when later selection and stopping work as intended. Without an oracle or evidence-recall ceiling for that pool, the "premature stopping" examples in Appendix C.4 cannot be clearly assigned to the stopping policy rather than missing candidates.

Questions for authors
  1. Can you report QA accuracy for top-k, MMR, and DPP on LongVideoBench, MLVU, and VideoMME using the same candidate pool, reranker, and answerer as SetGain? Paired uncertainty estimates would help with the smaller gains.
  2. What fraction of annotated evidence segments is represented in the 128-frame pool, and what temporal recall is achievable from that pool? In the Appendix C.4 failures, were the missed frames available to the selector?
2 comments

We thank the reviewer for the constructive suggestions; the recommended comparison led to the updated checkpoint used for all SetGain results below (see summary).

Q1. QA accuracy of top-k, MMR, and DPP on LongVideoBench, MLVU, and VideoMME with the same pool, reranker, and answerer, with uncertainty estimates.

Together with Reviewer H5Xc's Q1, this comparison showed that the submitted checkpoint trailed MMR and DPP on these benchmarks (64.5 vs. 65.7–65.8 on average), mainly on holistic questions. All selectors share SetGain's candidate pool, reranker scores, and frozen Qwen2.5-VL-7B answerer. We report QA accuracy as mean ± standard error.

SelectorFramesLongVideoBenchMLVUVideoMMEAvg.
Top-k862.2 ± 1.369.5 ± 1.060.7 ± 0.964.1
MMR862.2 ± 1.372.3 ± 1.062.5 ± 0.965.7
DPP862.8 ± 1.372.2 ± 1.062.4 ± 0.965.8
SetGain7.4665.3 ± 1.373.6 ± 0.963.7 ± 0.967.5

SetGain is best on all three benchmarks and on average (67.5 vs. 65.7–65.8 for MMR and DPP) while selecting fewer frames (7.46 vs. 8). Its lead over the stronger of MMR and DPP is +2.5 on LongVideoBench, +1.3 on MLVU, and +1.2 on VideoMME. The revision reports these standard errors in App. C.1 (Table 8).

Q2. Evidence coverage of the 128-frame pool, achievable recall, and the App. C.4 failures.

We measure coverage on the Haystack questions with ground-truth evidence (±5 s temporal window).

Pool depth kQuestion-level poolDecomposed pool (ours)
3246.847.9
6453.255.4
128 (full pool)60.164.6

The full pool contains 64.6% of the annotated evidence, which bounds the achievable temporal recall. SetGain's recall (Reviewer rx4f, Q2) is far below this bound, so the main gap lies in selecting within the pool, the problem SetGain addresses, rather than in pool construction. Event decomposition also raises the bound at every depth (+4.5 points for the full pool).

With the updated checkpoint, one of the original App. C.4 cases (lvbench_00139) is now answered correctly, and we replaced it with lvbench_00201, which fails in the same way (now App. D.5). The three cases contain eight ground-truth moments; SetGain covers two, both in the scene the question describes, and misses six. Four of the six are in the candidate pool but ranked low by the reranker (30th to 110th of 128), likely because they do not match the described scene, and the gain predictor does not lift them (27th to 124th), so even an eight-frame budget would not reach them. The other two have no candidate within 5 s and would require a better proposer. The failures therefore lie mainly in ranking rather than in candidate proposal.

Thanks for the reply, I dont have other concerns right now. I will keep my score.

Reviewer_rx4fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces an evidence selection framework that progressively selects visual evidence by estimating the marginal gain of adding each candidate to the current evidence set. To efficiently learn this adaptive selection strategy, the authors propose Marginal Gain Learning, which uses set-dependent coverage and stopping targets to supervise gain estimation. The strategy is then refined through Coverage-Guided Policy Optimization (CGPO) to better align evidence selection with the answerer’s needs. Extensive experiments demonstrate that explicitly modeling marginal evidence gain improves temporal localization, downstream question answering, and inference efficiency.

Critical strengths

The experimental results show that the proposed method outperforms TimeSearch-R on VideoMME, MLVU, and LongVideoBench while achieving lower end-to-end latency than both T* and TimeSearch-R. This paper proposes a novel Marginal Gain Learning approach for visual evidence selection, which appears promising.

Critical weaknesses

The loss function introduces several hyperparameters, including β_KL and λ_stop, which may make tuning difficult. The experiments lack comparisons with baselines that pair Qwen2.5-VL-7B with uniform sampling or a simple selection strategy in Table 1. The experiments lack comparisons using larger models to assess whether the proposed method generalizes across model scales. Such comparisons are important because the stronger capabilities of larger models may diminish the benefits of the proposed evidence selection strategy.

Questions for authors

Please provide detailed experiments showing how these hyperparameters β_KL and λ_stop are tuned and how sensitive the model’s performance is to their values. Please provide detailed comparisons with baselines that pair Qwen2.5-VL-7B with uniform sampling or a simple selection strategy in Table 1. Please provide the comparisons with larger models such as Qwen3-30B or Qwen3.5 9B.

3 comments

We thank the reviewer for the helpful questions. All SetGain results below use the updated checkpoint (see summary).

Q1. How are β_KL and λ_stop tuned, and how sensitive is performance to them?

We tune β_KL, λ_stop, and the stopping threshold τ on 601 validation questions (301 LongVideoBench, 300 MLVU) held out from both benchmarks and split by video, so no validation video appears in their test questions. We select by temporal F1 on the LongVideoBench part and QA accuracy on all 601, giving β_KL = 0.01, λ_stop = 0.5, and τ = 0.3. For sensitivity, we vary β_KL from 0.01 to 0.2 and λ_stop from 0.3 to 1.0 with the other fixed, again selecting τ on validation (0.3 for the β_KL variants, 0.5 for λ_stop = 1.0, and a fixed eight-frame budget for λ_stop = 0.3). Haystack-LVBench; reported setting in bold.

β_KLλ_stopFramesT-PT-RT-F1
0.010.56.2417.423.817.8
0.050.57.2315.924.317.1
0.200.55.0416.522.417.2
0.010.38.0015.624.417.1
0.011.06.6017.323.517.8
Cross-encoder + top-k–8.0014.822.216.0

Temporal F1 varies by at most 0.7 points (17.1–17.8), and every setting beats top-k on the same pool and scorer (16.0). The stopping loss needs sufficient weight: with λ_stop = 0.3, adaptive stopping no longer beats a fixed budget, consistent with the collapse to eight frames without the stopping loss (Table 3). λ_stop = 0.5 and 1.0 reach the same F1. This study is in App. C.6 (Table 13).

Q2. Comparison with Qwen2.5-VL-7B paired with uniform sampling or simple selection strategies.

We compare SetGain with uniform sampling and three training-free selectors (top-k, MMR, DPP), all with the same frozen Qwen2.5-VL-7B answerer; top-k, MMR, and DPP also share SetGain's pool and evidence scores, so only the selection rule differs. As in Fig. 4a, we report temporal (T-) and visual (V-) P/R/F1 on Haystack-LVBench and QA on both Haystack splits (mean ± standard error; bold: best per column).

SelectorFramesT-PT-RT-F1V-PV-RV-F1QA (LVBench)QA (Ego4D)
Uniform-881.4 ± 0.36.3 ± 1.12.2 ± 0.456.0 ± 0.872.0 ± 0.862.7 ± 0.845.0 ± 2.732.0 ± 3.3
Top-k814.8 ± 1.622.2 ± 2.016.0 ± 1.664.2 ± 1.076.8 ± 1.069.9 ± 1.055.3 ± 2.749.0 ± 3.5
MMR813.0 ± 1.225.7 ± 2.115.6 ± 1.467.5 ± 1.074.9 ± 0.971.0 ± 0.959.4 ± 2.747.0 ± 3.5
DPP811.7 ± 1.326.6 ± 2.115.3 ± 1.467.3 ± 0.975.7 ± 0.971.2 ± 0.959.9 ± 2.749.0 ± 3.5
SetGain6.2417.4 ± 1.723.8 ± 2.117.8 ± 1.669.3 ± 1.072.0 ± 1.070.6 ± 1.062.0 ± 2.654.0 ± 3.5

MMR and DPP improve QA over top-k, confirming that redundancy limits pointwise selection, but they gain recall by spreading frames across visually different content, which lowers temporal precision and F1 below top-k. SetGain improves over top-k by +1.8 temporal F1 and +6.7 Haystack-LVBench QA with 22% fewer frames, and has the highest visual precision and QA on both Haystack splits; its visual F1 (70.6) is slightly below MMR and DPP (71.0–71.2), within one standard error.

Q3. Comparison with larger models such as Qwen3-30B or Qwen3.5-9B.

We replace the answerer with stronger models, keeping each method's selected frames fixed; since Qwen3-30B is text-only, we use Qwen3-VL-30B-A3B. Results are on Haystack-LVBench; the last four columns are QA accuracy. SetGain uses Qwen3-VL-8B to score answers during training, so that column is not held out for SetGain.

SelectorFramesT-PT-RT-F1Qwen2.5-VL-7BQwen3-VL-8BQwen3.5-9BQwen3-VL-30B-A3B
TimeSearch-R9.45.3 ± 0.620.2 ± 1.97.7 ± 0.855.8 ± 2.759.6 ± 2.761.7 ± 2.662.0 ± 2.6
Top-k814.8 ± 1.622.2 ± 2.016.0 ± 1.655.3 ± 2.762.0 ± 2.663.7 ± 2.661.4 ± 2.6
MMR813.0 ± 1.225.7 ± 2.115.6 ± 1.459.4 ± 2.762.6 ± 2.664.9 ± 2.663.5 ± 2.6
DPP811.7 ± 1.326.6 ± 2.115.3 ± 1.459.9 ± 2.762.6 ± 2.666.1 ± 2.662.9 ± 2.6
SetGain6.2417.4 ± 1.723.8 ± 2.117.8 ± 1.662.0 ± 2.664.6 ± 2.668.7 ± 2.567.0 ± 2.5

Mean ± standard error.

SetGain gives the highest QA with every answerer: +2.6 to +6.7 over top-k, +2.0 to +3.8 over MMR, +2.0 to +4.1 over DPP, and +5.0 to +7.0 over TimeSearch-R. The lead holds with both held-out answerers: +2.6 over the strongest baseline with Qwen3.5-9B (DPP) and +3.5 with Qwen3-VL-30B-A3B (MMR). A stronger answerer can reason better over the frames it receives, but it cannot recover evidence that was never selected.

Please provide detailed implementation details and parameter settings for DPP to make the comparison more convincing.

We thank the reviewer for this suggestion.

Implementation. MMR [1] and DPP [2] use exactly the same inputs as top-k and SetGain: the same 128-frame candidate pool, the same relevance score per frame (the zero-shot Qwen3-VL-Reranker-2B score against the question), and the same frozen Qwen2.5-VL-7B answerer. Redundancy between frames is the cosine similarity of their SigLIP embeddings. Both select exactly 8 frames greedily, without adaptive stopping. MMR adds at each step the frame with the best trade-off between relevance and its maximum similarity to the frames already selected, weighted by λ. DPP uses the relevance-weighted similarity kernel and fast greedy MAP inference of [2], where θ sets the same trade-off. For both, a larger λ or θ weights relevance more, and the limit of 1 recovers top-k.

Parameter sweep (Haystack-LVBench, 8 frames, same answerer):

MethodSettingQATemporal F1
MMRλ = 0.357.613.7
MMRλ = 0.5 (reported)59.415.6
MMRλ = 0.858.516.0
DPPθ = 0.358.28.0
DPPθ = 0.559.69.1
DPPθ = 0.858.213.3
DPPθ = 0.9 (reported)59.915.3
SetGainadaptive (6.24 frames)62.017.8

A stronger diversity weight lowers temporal F1 for both baselines (MMR: 16.0 → 13.7; DPP: 15.3 → 8.0), while QA changes little (57.6–59.9). SetGain exceeds every setting of both on both metrics (+2.1 to +4.4 QA, +1.8 to +9.8 F1), so the comparison does not depend on λ or θ. The paper reports λ = 0.5 and θ = 0.9; we added these details and the sweep to the appendix.

[1] Carbonell & Goldstein. The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries. SIGIR 1998.

[2] Chen et al. Fast Greedy MAP Inference for Determinantal Point Process to Improve Recommendation Diversity. NeurIPS 2018.

Comments on the paper

4 comments

We thank the reviewer for the suggested controls, which led to the updated checkpoint used for all SetGain results below.

Q1. Cross-encoder top-k and uniform sampling on LongVideoBench, MLVU, and VideoMME with the same pool, reranker, and Qwen2.5-VL-7B answerer.

It showed that the submitted checkpoint trailed MMR and DPP here (64.5 vs. 65.7–65.8 average). All rows use the same frozen Qwen2.5-VL-7B answerer; top-k, MMR, and DPP also share SetGain's pool and reranker, while the other methods use their own frame scorers. QA, mean ± standard error:

SelectorFramesLongVideoBenchMLVUVideoMMEAvg.
Uniform-8854.3 ± 1.457.4 ± 1.156.9 ± 1.056.2
Uniform-323260.1 ± 1.364.2 ± 1.063.0 ± 0.962.4
TSPO857.4 ± 1.463.8 ± 1.055.1 ± 1.058.8
Q-Frame858.2 ± 1.367.6 ± 1.058.2 ± 0.961.3
AKS859.5 ± 1.366.7 ± 1.060.8 ± 0.962.3
TimeSearch-R7.857.1 ± 1.468.6 ± 1.059.1 ± 0.961.6
Top-k862.2 ± 1.369.5 ± 1.060.7 ± 0.964.1
MMR862.2 ± 1.372.3 ± 1.062.5 ± 0.965.7
DPP862.8 ± 1.372.2 ± 1.062.4 ± 0.965.8
SetGain7.4665.3 ± 1.373.6 ± 0.963.7 ± 0.967.5

As the reviewer suspected, the pool and reranker explain part of the gain over TimeSearch-R: top-k on our pool already adds 2.5 points on average. SetGain adds a further 3.4 over top-k, improving every benchmark (+3.1, +4.1, +3.0) with fewer frames; it is best among all selectors on all three (+1.7 to +1.9 over MMR and DPP on average) and beats uniform-32 everywhere with 4.3× fewer frames. Table 1 now includes these rows and states this attribution.

Q2. SetGain vs. top-k and uniform sampling at larger frame budgets (16, 32, …, 256).

Stopping is removed so every selector uses the full budget; we stop at 64 as the pool holds only 128 frames. QA on Haystack-LVBench:

BudgetUniformTop-kSetGain
845.055.362.6
1650.059.161.1
3252.659.960.8
6453.258.858.2

SetGain's advantage is largest at small budgets (+7.3 at 8, +2.0 at 16); at 32 and 64 frames the two are on par (+0.9 and −0.6). With adaptive stopping it reaches 62.0 with 6.24 frames, at least on par with top-k at every budget.

SetGain is not cheaper than uniform sampling, since search is over 90% of its compute (Table 2). But top-k on the same pool shares that search cost and the selection rule adds only 0.025 s, so every saved frame directly cuts answering cost; among search methods it is fastest (2.16× vs. T*, 2.54× vs. TimeSearch-R). More uniform frames do not replace search: uniform-64 stays 8.8 below SetGain. Compact evidence matters most with larger answerers (rx4f Q3).

Q3. Contribution of event decomposition: question-level SigLIP pool vs. per-event max.

We replace the per-event max with a question-level SigLIP pool (top-128 frames), keeping the reranker, answerer, and trained selector fixed; 8-frame budget (SetGain stops adaptively). QA; Q: question-level, D: decomposed (ours).

BenchmarkUniformTop-k (Q)Top-k (D)SetGain (Q)SetGain (D)
Haystack-LVBench45.050.955.357.362.0
LongVideoBench54.360.262.264.165.3
MLVU57.468.869.572.173.6
VideoMME56.959.860.762.963.7

Averaged over the four benchmarks, the selection rule adds more than decomposition (4.2 points over top-k with either pool, vs. 2.0–2.1 from decomposition). Even on the question-level pool, SetGain beats top-k on our decomposed pool on every benchmark (+1.9 to +2.6), and decomposition helps both selectors everywhere (+0.7 to +4.7), so the two are complementary.

Q4. Comparison with TSPO, AKS, and Q-Frame under the same answerer and budget.

Same Qwen2.5-VL-7B answerer and 8-frame budget, on Haystack-LVBench:

SelectorFramesPRTemporal F1QA
TSPO88.213.99.155.0
AKS83.316.45.255.3
Q-Frame83.417.15.556.7
SetGain6.2417.423.817.862.0

SetGain beats all three on every metric with fewer frames: 2.0–3.4× their temporal F1, higher precision and recall, and +5.3 to +7.0 QA; on the three QA benchmarks (Q1) it leads by 5.2–8.8 points on average. Their QA stays close to top-k despite much weaker localization, since answers can come from broad context or language priors [1], which is why SetGain also uses coverage rewards (Sec. 2.4). These are full pipelines with their own scorers (controlled comparison: rx4f Q2). See App. C.2 (Table 9).

[1] Niu et al. Counterfactual VQA: A Cause-Effect Look at Language Bias. CVPR 2021.

I. Acknowledgments

We thank all four reviewers for their constructive reviews. Reviewer DoJx recommended acceptance (Rating: 4); Reviewers rx4f, H5Xc, and Zv7p rated weak rejection (Rating: 2), mainly asking whether the selector itself, rather than the candidate pool and reranker, drives the QA gains.

Updated checkpoint. The controlled comparison requested by H5Xc and Zv7p showed that the submitted checkpoint trailed MMR and DPP on LongVideoBench, MLVU, and VideoMME (64.5 vs. 65.7–65.8 average QA), mainly on holistic questions. We retrained with 4,676 whole-video questions from Seeker-173K and a Qwen3-VL-8B training answerer; evaluation still uses the frozen Qwen2.5-VL-7B. Average QA rises to 67.5, and Haystack temporal F1 is nearly unchanged (17.8 vs. 18.1). All numbers below use this checkpoint.

II. Key Strengths

  • Controlled evaluation that isolates the selector from the pool and scorer (H5Xc, Zv7p, DoJx).
  • Lightweight and fast: 2.17M trainable parameters, about 26 GPU-hours, and lower latency than T* and TimeSearch-R (rx4f, H5Xc, Zv7p, DoJx).
  • Novel, well-motivated training signal from marginal-coverage targets and stopping supervision (rx4f, DoJx).
  • Clear presentation, rated clear and easy to follow by all four reviewers.

III. Key Concerns and Our Responses

Key ConcernReviewersOur Response
Attribution of QA gains to the selector rather than the pool and rerankerrx4f, H5Xc, Zv7p, DoJxAdded uniform, top-k, MMR, and DPP to Table 1 with the same pool, reranker, and answerer. Top-k on our pool adds +2.5 over TimeSearch-R; SetGain adds +3.4 more over top-k and +1.7–1.9 over MMR/DPP, and is best on all three benchmarks with 7.46 frames.
Role of the candidate pool: event decomposition, answer options, pool ceilingH5Xc, Zv7p, DoJxWith a question-level pool (no decomposition or options), SetGain still beats top-k by 4.2 points. The pool covers 64.6% of annotated evidence vs. 23.8% recall, so the bottleneck is selection.
Frame budget and stopping: small budgets, larger budgets, count priorH5Xc, DoJxSetGain's QA lead over top-k is +7.3 at 8 frames and narrows to parity at 64; adaptive SetGain (6.24 frames) beats top-k at every budget. The budget varies by question: 27.8% / 20.2% / 52.0% stop at 3 / 4–7 / 8 frames.
Stronger answerersrx4f, DoJxSetGain gives the highest QA with all four answerers, up to Qwen3-VL-30B-A3B, and leads the strongest baseline by +2.6 and +3.5 with the two held-out answerers.
Novelty vs. MMR/DPP, a pointwise head, and learned selectorsH5Xc, Zv7p, DoJxOn the same pool, MMR and DPP fall below top-k in temporal F1, while SetGain improves both F1 and QA. Set conditioning adds +1.4 F1 over an identically trained head without set state, which never stops early. SetGain beats TSPO, AKS, and Q-Frame by +5.3 to +7.0 QA on Haystack-LVBench.
Uncertainty and fairnessH5Xc, Zv7p, DoJxStandard errors over questions for all main comparisons, an explicit single-seed statement, and a TimeSearch-R rerun under our protocol (F1 7.7 vs. 8.1 published).
Hyperparameter sensitivityrx4fTuned on a held-out 601-question validation set; temporal F1 stays within 17.1–17.8 across β_KL and λ_stop, always above top-k (16.0).

IV. Commitment to Revision

All results above are in the revised paper: the updated checkpoint (Sec. 3), controlled baselines (Table 1), stronger answerers (Table 5), and new analyses (App. C.1–C.7, App. D). Fig. 4a now separates controlled rows from published ones, and Sec. 6 states the main failure mode, with per-event coverage tracking as future work, as H5Xc suggested.

We sincerely thank the AC and all reviewers for their time and expertise.

Reviewer_H5XcOpenReview ↗

Official Comment by Reviewer H5Xc

Thanks for the detailed response. While the rebuttal addresses my attribution questions, it also reveal that the submitted original model underperformed simple baselines (MMR and DPP) on certain categories of all three QA benchmarks. I maintain my original score for the following reasons.

  • The revised results come from a new checkpoint retrained with 4,676 additional questions targeting that question type, a stronger training answerer, and re-tuned hyperparameters. Better results from more data and a stronger training signal are not evidence for set-conditioned selection, and the rebuttal does not separate these recipe changes from the selector. The reviewed method is also no longer the one reported, so the revised recipe effectively needs a fresh review.
  • The larger-budget results confirm my earlier concern: at 32 and 64 frames SetGain is at parity with top-k, so its advantage is limited to very small budgets. Long-video understanding typically relies on larger frame budgets, so this considerably weakens the paper's relevance to the long-video setting it targets.
  • Lastly, although not a severe issue, but since the revised text is also in black, it is hard to tell what changed without re-reading the whole paper. Highlighting changes in a different color, a common practice for revisions, would have helped to increase readability.

We thank the reviewer for the careful follow-up and respond to each point below.

  • Recipe vs. set conditioning. We agree that the recipe changes contribute to the QA gains; the method itself is unchanged. Two results separate it from the recipe: under the identical new recipe, set conditioning adds 1.4 temporal F1 with 22% fewer frames over a pointwise head (Table 4), and the submitted checkpoint, trained with the original recipe, already exceeded top-k, MMR, and DPP in temporal F1 (18.1 vs. 15.3–16.0). We will add the no-set-state variant on the three QA benchmarks.

  • Larger budgets. We agree that SetGain and top-k are on par at 32–64 frames, and will state that SetGain targets the compact-evidence regime. There, SetGain with 6.24 frames is at least on par with top-k at any budget up to 64, and with 7.46 frames it exceeds uniform-32 by 5.1 points on the three QA benchmarks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.