acceptodds
Under review as a conference paper at ICLR 2027

Temporal Quantization for Adaptive Candidate Frame Construction in Video QA

Abstract

Video language models (VLMs) operate under limited visual-token and computational budgets, requiring videos to be represented by a compact set of frames. A common pipeline first constructs a temporally subsampled candidate frame set and then selects the final input frames from it. While conventional pipelines rely on query-agnostic temporal sampling, recent work has increasingly explored adaptive frame acquisition guided by the query and partial observations. Yet, how a fixed observation budget should represent the video timeline before downstream selection remains less explicitly characterized. To address this, we introduce QUEST, a training-free method that formulates candidate construction as relevance-weighted temporal quantization, explicitly measuring how well the acquired observations resolve the timeline. By jointly considering estimated query relevance and temporal gaps between observations, QUEST allocates finer temporal resolution to relevant yet under-resolved regions while maintaining coverage elsewhere. The resulting candidate set can be used with existing frame selectors without modifying their selection rules. Extensive experiments across multiple VLMs, frame selectors, and video understanding benchmarks demonstrate improved downstream accuracy under matched observation and selection budgets.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Large Language Model, Video understanding, Retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_ZeB6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies candidate-frame construction before downstream video-QA frame selection. QUEST scores uniformly initialized observations, weights each temporal interval by normalized endpoint relevance, and repeatedly samples the midpoint of the largest relevance-weighted gap. A one-step minimax argument motivates this rule when interval weights are held fixed. Experiments pair QUEST with four selectors and five VLMs on five benchmarks, primarily at N=128 candidates and K=32 selected frames. The reported average improvement over the corresponding uniform candidate pools is 0.87 percentage points; additional analyses measure evidence coverage, acquisition ablations, and latency.

Critical strengths
  1. The paper isolates a useful design decision: which frames become available to a subsequent selector. The relevance-weighted gap rule is simple, training-free, and supported by an explicit fixed-weight one-step argument (Sections 3–4). Reusing the selector’s cached relevance scores makes the construction modular, rather than requiring changes to the downstream VLM or selector.

  2. The matched uniform-versus-QUEST comparisons cover five VLMs, four selectors, and five benchmarks (Table 1), and improve every model–selector average, although individual benchmark scores sometimes decrease. The direct candidate-pool evaluations in Appendix D also improve annotated evidence coverage across all three scorers. This combination provides more informative support for the acquisition mechanism than downstream QA accuracy alone.

Critical weaknesses
  1. The comparison with existing adaptive acquisition does not isolate the proposed acquisition rule. Table 3 compares QUEST averaged over four downstream selectors with other complete acquisition pipelines. Appendix A.4 (p.15) states that QUEST uses CLIP/SigLIP/LongCLIP scorers with the question and answer choices, whereas CSES and FOCUS use BLIP-based relevance with the question only; CSES additionally retains adaptive stopping and a variable selected-frame count. Consequently, the reported superiority over adaptive alternatives could reflect these other differences. The clean Table 1 comparison against uniform pools establishes useful gains, but does not resolve whether the minimax acquisition rule improves over an equally configured adaptive competitor.
Questions for authors
  1. With the same relevance encoder, scoring text, downstream selector, and actual observed/selected-frame budgets, how does QUEST compare with CSES or FOCUS? A controlled comparison on representative settings, in addition to the existing native-pipeline results, would directly clarify the contribution of the acquisition rule.
2 comments
AuthorsOpenReview ↗

[Response to Reviewer ZeB6 (1/1)]

We appreciate the reviewer's emphasis on fair comparisons with existing adaptive acquisition methods, particularly the need to control relevance scoring, observation budgets, and downstream selection protocols.

Q1, W1. Controlled comparison with adaptive acquisition. To isolate candidate acquisition from the scorer and downstream selector, we compared QUEST and CSES with a common BLIP scorer, scoring text with question and options, Qwen3-VL, and the same CSES selector. We additionally evaluated QUEST against FOCUS at matched per-sample acquisition counts. The target number of selected frames is K=32K=32. All averages and differences are computed before rounding.

(i) CSES. Both arms acquire exactly N=128N=128 candidate frames and use the CSES selector without stopping to select exactly K=32K=32 frames. We also disable CSES's acquisition stopping rules to enforce the fixed candidate budget. The only difference between the two arms is whether the candidate pool is constructed by CSES or QUEST.

BenchmarkCSESQUESTΔ\Delta (pp)
LongVideoBench65.5965.82+0.22
MLVU73.1074.91+1.81
VideoMME69.6769.74+0.07
LVBench46.4846.55+0.06
CG-Bench44.8545.76+0.90
Average59.9460.55+0.61

Replacing CSES acquisition with QUEST improves accuracy on all five benchmarks, with a +0.61 pp average gain. Unlike the original complete-pipeline comparison, this experiment keeps the relevance encoder, scoring text, downstream selector, and actual candidate and selected-frame counts fixed. It therefore directly tests the contribution of the acquisition rule under the requested controls.

(ii) FOCUS. As a complementary pipeline comparison, we preserve FOCUS's adaptive acquisition and match QUEST to its realized acquisition count. For each sample, QUEST acquires as many candidates as the number of unique frames scored during FOCUS acquisition, then uses the CSES selector with stopping disabled. The BLIP scorer, scoring text with question and options, Qwen3-VL, and evaluation protocol remain shared.

BenchmarkFOCUSQUESTΔ\Delta (pp)
LongVideoBench64.6266.19+1.57
MLVU72.9274.15+1.23
VideoMME66.0770.33+4.26
LVBench49.5852.68+3.10
CG-Bench47.4846.51-0.98
Average60.1461.97+1.84

QUEST obtains higher accuracy on four of five benchmarks, with a +1.84 pp benchmark-average gain at matched acquisition counts. The CSES experiment isolates the acquisition rule under a common selector, while this FOCUS experiment provides additional pipeline-level evidence when the acquisition budget varies by sample.

We have added the shared scorer CSES and FOCUS comparisons to Section 5.2 (Tables 4–5) and documented their acquisition, scoring, and selection settings in Appendix A.6. The added protocol specifies the stopping-rule controls used for CSES and the per-sample acquisition-budget matching used for FOCUS.

I have read the rebuttal and decided to maintain my score.

Reviewer_pY9yOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes QUEST - a query aware frame selection method that doesn't require training. QUEST basically builds the candidate frame set by sampling the gap between frames with the largest relevance, so long gaps that look relevant to the question get filled in first.

Critical strengths
  • Demonstrated compatibility and complementary nature with other frame selection methods.
  • Simple and relatively cheap method.
  • Large evaluation suite demonstrating small but consistent gains
Critical weaknesses
  • Performance gains are consistent but not significant enough (Table 1)
  • Algorithm is essentially greedy bisection, and a suboptimal starting point essentially results in uniform selection. Novelty is quite limited and method seems conceptually quite close to AKS, FOCUS, CSES etc.
Questions for authors
  • It is unclear to me why choosing the midpoint is the optimal strategy. If one of the frames is extremely relevant and the other less so - wouldn't it be better to weigh the choice of the frame accordingly, ie closer to the more relevant frame some how?
  • Since AKS scores the full 1fps candidate pool and therefore has strictly more relevance information than QUEST’s 128 sequentially acquired frames, could the authors clarify what explains QUEST’s higher accuracy in Table 2?
3 comments
AuthorsOpenReview ↗

[Response to Reviewer pY9y (1/2)]

We thank the reviewer for their careful assessment of the statistical significance of our results and for raising important questions about initialization, midpoint sampling, and observation budget.

W1. Magnitude and significance of improvements. The average gain is modest, but it results from changing candidate construction without additional training or larger observation and input-frame budgets. We quantified its uncertainty using 50,000 paired video-cluster bootstrap replicates over Table 1. The mean improvement is +0.874 pp (95% CI [+0.580, +1.175], two-sided video-cluster sign-flip p < 0.001), providing statistical support for this fixed-budget improvement.

We additionally examined whether the advantage can be larger with fewer candidate observations. Using Qwen3-VL and Top-K with K=32K=32, we compared gains over uniform at N=64N=64 with the corresponding Table 1 results at N=128N=128. Gains below are in percentage points.

BenchmarkN=128N=128N=64N=64
MLVU+1.13+1.59
LVBench+0.46+1.29

On MLVU and LVBench, the gains increase from +1.13 to +1.59 pp and from +0.46 to +1.29 pp. These results indicate that QUEST's advantage can be more pronounced under tighter observation budgets, in addition to the statistically supported average improvement in Table 1.

W2. Initialization and methodological contribution. (i) Initialization. A suboptimal initialization does not necessarily lead to uniform sampling, since QUEST updates interval relevance after each new observation. Subsequent observations can change the acquisition priorities established by the initial frames. When the resulting normalized interval weights remain similar, temporal gaps become more influential, and QUEST may approach uniform coverage. Thus, uniform-like behavior is possible when relevance provides little discrimination, but is not an inevitable consequence of suboptimal initialization.

(ii) Methodological contribution. Greedy bisection describes how a new observation is placed, but not the candidate-representation objective that determines where the observation budget should be spent. QUEST's contribution is to formulate that objective independently of final keyframe selection. Existing keyframe-selection methods, including AKS, CSES, and FOCUS, aim to obtain a useful final keyframe set. QUEST instead asks how to construct the candidate representation supplied to a selector under a fixed observation budget.

This distinction is important because a selector cannot recover evidence that candidate acquisition never observed. Moreover, low average temporal distortion may still leave large local observation gaps. QUEST formulates candidate construction as relevance-weighted temporal quantization and directly targets the worst-case query-aware covering error defined from current observations. Under fixed interval relevance weights, selecting the largest weighted gap and sampling its midpoint follows from the one-step minimax objective, rather than from bisection alone.

This separation also explains the compatibility with existing selectors noted in the review: QUEST changes the evidence available to them without modifying their selection rules. We evaluate the resulting candidate representation through annotated evidence retention and sampling quality, and assess downstream QA with the selector held fixed.

AuthorsOpenReview ↗

[Response to Reviewer pY9y (2/2)]

Q1. Midpoint versus relevance-weighted point We directly tested whether sampling closer to the more relevant endpoint improves performance with Qwen2.5-VL under the same setting as in Appendix E. We replace only midpoint placement with the relevance-weighted mean of the endpoint timestamps.

BenchmarkMidpointRelevance-weighted pointΔ\Delta
MLVU70.3769.27-1.10
LVBench43.1942.54-0.65
Average56.7855.91-0.87

Moving the sampling point closer to the relevant endpoint reduces the performance by 1.10 and 0.65 percentage points on MLVU and LVBench, respectively. While this approach is intuitive, the appropriate displacement depends on the relevance distribution within the interval. The relevance distribution may vary across videos and cannot be inferred from endpoint scores alone without additional assumptions. QUEST therefore uses the midpoint, which minimizes the maximum remaining temporal gap under the current fixed interval weight (Sec. 3).

Q2. Observation budget and downstream accuracy. In Table 2, we compare QUEST's observation-limited pipeline with AKS's standard 1 FPS pipeline. AKS scores a dense candidate pool and selects the final K=32K=32 keyframes by balancing relevance and temporal coverage. QUEST instead constructs its N=128N=128 candidate pool by sequentially refining the largest relevance-weighted temporal gap before downstream selection. Since only the final selected frames are provided to the VLM, their usefulness depends on how evidence is acquired and retained, rather than the number of scored candidates alone. This provides a possible explanation for QUEST's higher average accuracy despite fewer observations.

AKS's own ablation (Tang et al., 2025, Table 3) also shows that lower candidate sampling frequencies can achieve higher accuracy with the same LLaVA-Video-7B and K=32K=32:

BenchmarkAKS (1 FPS)AKS (lower FPS)
LongVideoBench61.962.1 (0.5 FPS)
VideoMME64.665.1 (0.25 FPS)

These results further illustrate that increasing the number of scored candidates does not guarantee higher downstream accuracy.

We have clarified the candidate-construction contribution in Sections 1, 2.2, and 6, and initialization behavior in Section 5.3. The one-step justification for midpoint placement is clarified in Section 3.3, with the relevance-weighted placement experiment added in Appendix E.2. Section 5.2 now summarizes the statistical evidence and clarifies the interpretation of observation-budget comparisons, while Appendix F provides the full statistical analysis and Appendix E.3 reports the smaller-budget results.

Reviewer_pY9yOpenReview ↗

Thank you.

Thanks for the author's response. I'm still not convinced that this work has enough novelty to warrant a higher score. QUEST's relevance-weighted gap prioritization and midpoint sampling closely resemble existing adaptive acquisition strategies and the minimax formulation does not establish a substantially new algorithmic principle. I would like to keep my score.

Reviewer_5XfoOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies adaptive candidate-frame construction for video question answering (Video QA) under fixed observation and downstream frame-selection budgets. The authors propose QUEST, a training-free method that formulates candidate construction as relevance-weighted temporal quantization. Starting from a small set of uniformly spaced frames, QUEST estimates temporal interval relevance using the query-relevance scores of observed frames and iteratively samples the midpoint of the interval with the largest relevance-weighted temporal gap. The method reuses the frame-level relevance scorer of an existing frame selector while leaving its downstream selection rule unchanged, allowing it to be integrated into existing video QA pipelines. The authors evaluate QUEST across five vision-language models, four frame selectors, and five video understanding benchmarks. Under matched observation and selection budgets, they report an average improvement of 0.87 percentage points over uniform candidate construction, and further compare QUEST with dense 1-FPS sampling and existing adaptive frame-acquisition methods.

Critical strengths

1.The paper identifies an important limitation of existing video QA pipelines: downstream frame selectors cannot recover relevant visual evidence that was omitted during candidate-frame construction. Explicitly studying candidate-frame construction under a fixed observation budget is a practically motivated and worthwhile research direction.

2.QUEST does not require additional training or modifications to existing frame selectors. It reuses the frame-level relevance scorer of an existing selector and adaptively allocates candidate frames based on temporal gaps and estimated query relevance. This makes the method relatively easy to integrate into existing video QA systems

Critical weaknesses
  1. The core algorithm is relatively simple: it estimates interval relevance using the relevance scores of observed endpoints, prioritizes intervals based on the product of relevance and temporal gap, and samples near the midpoint of the highest-priority interval. Although the temporal quantization perspective provides an intuitive problem formulation, it remains unclear whether it offers a sufficiently substantial conceptual or algorithmic advance over existing adaptive frame-acquisition methods that also consider relevance and temporal coverage. The paper needs to clarify what fundamentally new contributions it offers beyond this heuristic strategy and its theoretical interpretation.

  2. The minimax result establishes the optimality of a single sampling step only when interval relevance weights are held fixed. However, QUEST recomputes relevance weights after acquiring new frames, and the normalization depends on the minimum and maximum relevance scores among the currently observed frames. Consequently, the theorem does not establish global optimality or convergence guarantees for the complete sequential sampling procedure.

  3. The reported average improvement over uniform candidate-frame construction is only 0.87 percentage points. Although QUEST improves average performance across the evaluated configurations, some individual combinations of models, selectors, and benchmarks exhibit performance degradation.

  4. QUEST estimates interval relevance using the average relevance scores of the two observed endpoints, followed by min-max normalization and an exponential transformation. While computationally inexpensive, this approach implicitly assumes that endpoint scores provide a reliable proxy for the relevance of unobserved content within an interval. Consequently, a brief but highly relevant event located between two low-relevance frames may receive a low sampling priority and remain undiscovered.

Questions for authors
  1. Could the authors clarify the fundamental methodological differences between QUEST and existing adaptive frame-acquisition methods, particularly CSES and FOCUS? Specifically, beyond prioritizing intervals based on the product of estimated relevance and temporal gap and sampling at the interval midpoint, what is the core technical contribution of QUEST? What substantive differences distinguish it from existing relevance-aware and coverage-aware sampling strategies?

  2. The minimax proposition assumes that interval relevance weights remain fixed during a sampling step. However, the actual QUEST algorithm recomputes relevance weights based on the updated set of observed frames. Beyond the one-step optimality result, could the authors provide further justification for the effectiveness of this sequential sampling procedure? Additionally, how does the algorithm handle the special case in which all observed frames have identical relevance scores, causing the denominator of the min-max normalization to become zero?

  3. In the comparisons with CSES and FOCUS, the relevance-scoring configurations differ from those used by QUEST, and CSES retains its adaptive stopping mechanism. How do the authors ensure that these methods are fairly compared in terms of observation budgets, computational overhead, relevance-scoring resources, and evaluation protocols?

3 comments
AuthorsOpenReview ↗

[Response to Reviewer 5Xfo (1/3)]

We appreciate the reviewer's recognition of the importance of candidate-frame construction and their critical questions regarding QUEST's methodological contribution, theoretical justification, and empirical validation.

W1, Q1. Fundamental methodological differences. The core technical contribution concerns the representation constructed before final frame selection, rather than relevance weighting or midpoint sampling in isolation. CSES guides acquisition using estimated marginal coverage gain and performs coverage-based keyframe selection, while FOCUS uses confidence-guided exploration to identify informative regions for selection. These methods employ additional observations to obtain a useful final keyframe set. QUEST instead independently formulates the construction of a candidate representation under a fixed observation budget, using only currently available observations, rather than directly optimizing the final selected keyframes.

This distinction matters because evidence omitted during candidate acquisition cannot be recovered by a downstream selector. At the same time, current observations do not reveal the importance of unobserved regions precisely, and low average temporal distortion can still leave large observation gaps in individual regions. Candidate construction therefore requires directly controlling these local gaps, separately from optimizing the utility of the final selected frames.

QUEST expresses this requirement as relevance-weighted temporal quantization, with the worst-case query-aware covering error defined using relevance estimated from current observations. Under fixed current interval relevance weights, the one-step minimax objective yields both the choice of the largest relevance-weighted temporal gap and midpoint placement. Thus, the formulation specifies an optimization target for candidate representation and derives the acquisition rule from it, rather than offering only a theoretical interpretation of bisection. We evaluate this representation before downstream selection through annotated evidence retention and sampling quality, and separately measure its effect on QA while keeping the downstream selector fixed.

W2, Q2. Sequential optimality and min-max normalization.

(i) Global optimality and convergence guarantees. Under partial observation, the relevance of unobserved frames is unknown, so global optimality of the complete acquisition sequence cannot generally be guaranteed from current observations alone without additional assumptions. QUEST therefore adopts a best-effort strategy that optimizes each acquisition decision under the current fixed interval relevance weights, then updates these weights as new frames are observed. Our theorem formalizes this instantaneous optimality by showing that selecting an interval with the largest relevance-weighted temporal gap and sampling its midpoint minimizes the one-step worst-case covering error. This instantaneous optimality is the intended theoretical justification for the sequential procedure, rather than a claim of global optimality over the complete acquisition sequence.

(ii) Min-max normalization. The normalization step includes a safeguard before the updated relevance weights are computed. If the observed raw score range (max-min) is at most 10−410^{-4}, all interval relevance weights are set to 1 instead of dividing by that range. The next acquisition then refines the largest temporal gap. Consequently, identical scores produce a temporal-gap-only decision rather than an undefined normalization, and almost identical scores within this tolerance are handled in the same way. This branch is now explicit in Section 4.1 and Algorithm in Appendix C.

W3. Magnitude and robustness of improvements. We examined whether the positive average extends across models and selectors despite the regressions in individual configurations. Among the 100 matched Table 1 comparisons, QUEST improves 87, ties in one, and decreases in 12. Averaging each model–selector combination over the five benchmarks gives positive mean gains for all 20 combinations. Paired video-cluster bootstrap analysis with 50,000 replicates yields an overall gain of +0.874 pp (95% CI [+0.580, +1.175], two-sided video-cluster sign-flip p < 0.001).

We further tested dependence on particular models or selectors through leave-one-out analyses. Excluding any single model gives average gains of +0.842 to +0.929 pp, and excluding any selector gives +0.814 to +0.981 pp, with all corresponding 95% CIs above zero. These results indicate that the average benefit is not concentrated in one model or selector. The gain remains modest in absolute terms, but is obtained under matched observation and input frame budgets without changing the downstream selection rule. This supports the average benefit of candidate construction without implying improvement in every configuration.

AuthorsOpenReview ↗

[Response to Reviewer 5Xfo (2/3)]

W4. Evidence missed between observed endpoints. Endpoint scores provide an estimate of interval relevance, not a guarantee that the unseen interior has similar relevance. A brief event can therefore be underestimated by this proxy. Observing more frames reduces the risk of omission at additional computational cost, while large video language model pipelines limit the number of frames observed or selected. Under this constraint, QUEST does not use endpoint relevance alone to allocate observations. It combines that estimate with the temporal gap, so that a sufficiently large unresolved interval can warrant refinement despite low endpoint scores.

To evaluate whether this additional temporal term mitigates omissions, we removed only the temporal-gap factor from QUEST's acquisition priority. Initialization, scorer, midpoint placement, and observation budget N=128N=128 were unchanged. Among questions whose initial observations missed every annotated evidence region, we measured the percentage for which subsequent acquisition captures at least one such region. Ranges cover CLIP-L/14, SigLIP, and LongCLIP, and gains are computed from paired comparisons within each scorer.

BenchmarkInitially missed questionsRelevance-only (%)QUEST (%)Δ\Delta (pp)
LongVideoBench65236.20–43.2565.49–65.80+22.24–+29.29
VideoMME29136.77–41.9279.04–79.73+37.11–+42.61
CG-Bench-mini1,08934.07–36.5564.10–65.01+27.55–+30.30

Including the temporal-gap term increases recovery by 22.24–42.61 pp across all nine settings. All paired video-cluster bootstrap pointwise 95% CIs for these differences exclude zero.

We also examined the specific case of brief evidence between low-scoring endpoints. We considered annotated intervals missed by the initial observations and bracketed by consecutive initial frames outside all annotated evidence regions for the question. Both endpoint scores were required to fall in the lower half of scores from the initial WW observations. Brief intervals were shorter than the video's maximum inter-frame temporal gap under uniform 128 frame sampling. We used annotations only for evaluation and stratified these intervals by the minimum number of midpoint refinements along the event-containing branch needed to reach them. For targets requiring one or two refinements, QUEST achieved complete recovery by N=128N=128 in all nine settings, with 19–39 targets per setting, whereas relevance-only recovered 0–2.56%. Appendix D.2 reports the full breakdown, including targets requiring deeper refinement.

These results support the temporal-gap term as a way to complement an imperfect endpoint estimate when allocating observations. They establish its contribution to mitigating evidence omissions, rather than requiring the endpoint proxy to accurately represent all intervening content.

AuthorsOpenReview ↗

[Response to Reviewer 5Xfo (3/3)]

Q3. Fair comparison with CSES and FOCUS. The original results compare the independent pipelines using the original CSES and FOCUS implementations under their native settings. They therefore evaluate complete pipelines rather than isolating acquisition alone. To examine the configuration differences raised in the question, we added comparisons using the same BLIP scorer, scoring text with question and options, Qwen3-VL, and evaluation protocol. The target number of selected frames is K=32K=32. QUEST uses the CSES selector, which operates on BLIP relevance scores. All averages and differences are computed before rounding.

(i) CSES. CSES's stopping rules can reduce both the acquired candidate count and the final selected-frame count. We disable stopping in both stages to fix the candidate pool at N=128N=128 and the selected frames at K=32K=32. We then replace CSES's acquisition rule with QUEST while applying the same CSES selector to both candidate pools.

BenchmarkCSESQUESTΔ\Delta (pp)
LongVideoBench65.5965.82+0.22
MLVU73.1074.91+1.81
VideoMME69.6769.74+0.07
LVBench46.4846.55+0.06
CG-Bench44.8545.76+0.90
Average59.9460.55+0.61

QUEST improves accuracy on all five benchmarks, with a +0.61 pp average gain. Because the scorer, scoring text, VLM, budgets, and downstream selector are fixed, this comparison attributes the observed difference to candidate acquisition rather than those configuration differences.

(ii) FOCUS. FOCUS determines its acquisition count adaptively. Instead of imposing a common fixed count that would alter its acquisition behavior, we retain that behavior and set QUEST's candidate budget for each sample to the number of unique frames scored during FOCUS's acquisition. QUEST then applies the CSES selector with stopping disabled.

BenchmarkFOCUSQUESTΔ\Delta (pp)
LongVideoBench64.6266.19+1.57
MLVU72.9274.15+1.23
VideoMME66.0770.33+4.26
LVBench49.5852.68+3.10
CG-Bench47.4846.51-0.98
Average60.1461.97+1.84

At matched per-sample acquisition counts, QUEST achieves higher accuracy on four of five benchmarks and a +1.84 pp benchmark-average gain. This complements the acquisition-only CSES comparison by evaluating the pipelines at the observation budgets chosen adaptively by FOCUS.

We have clarified the candidate-construction formulation and its distinction from prior work in Sections 1, 2.2, and 6, and the scope of the one-step result in Section 3.3. The normalization safeguard is now specified in Section 4.1 and Appendix C. We have added statistical analyses in Appendix F, evidence-recovery and refinement-depth analyses in Appendices D.1–D.2, and the shared-scorer CSES and FOCUS comparisons in Section 5.2 (Tables 4–5), with detailed experimental settings in Appendix A.6.

Reviewer_Kx6oOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes QUEST, a training-free method for adaptive candidate frame construction in video question answering. The authors formulate candidate acquisition as relevance-weighted temporal quantization. Starting from a small set of uniformly sampled frames, QUEST estimates interval relevance from the scores of observed endpoints, prioritizes the interval with the largest relevance-weighted temporal gap, and samples an additional frame at its midpoint. The resulting candidate set is passed to an existing frame selector without modifying its selection rule. Experiments across five video-language models, four frame selectors, and five benchmarks show an average accuracy improvement of 0.87 percentage points over uniform candidate construction under matched observation and final input-frame budgets.

Critical strengths

1)The sampling rule is interpretable. The relevance-weighted temporal covering radius combines query relevance and observation sparsity within a single objective. 2)The method is simple and easy to integrate into existing pipelines. QUEST reuses the downstream selector’s relevance scorer and caches the computed scores, requiring neither additional training nor repeated large-model assessments. 3)The evaluation covers a broad range of configurations. The paper evaluates five video-language models, four frame selectors, and five benchmarks, reporting an overall average improvement under matched observation and final input-frame budgets.

Critical weaknesses

1)QUEST estimates the relevance of an entire interval using the mean relevance score of its two observed endpoints. If both endpoints are irrelevant but critical evidence appears briefly within the interval, that interval may receive an unduly low acquisition priority. The method may therefore be better suited to refining already identified relevant segments than to discovering new critical events. 2)Query relevance does not necessarily reflect the value of evidence for answering the question. Although the scoring text includes both the question and answer choices, semantic matching scores do not directly indicate whether a frame provides new information that distinguishes between candidate answers. For example, multiple frames repeatedly depicting the question’s background may receive high scores, while decisive details may not achieve the highest matching scores. The acquisition rule primarily considers relevance and temporal gaps, without explicitly modeling the additional evidence a new frame contributes beyond existing observations. 3)The overall accuracy gains are modest, and improvements are not consistent across all configurations. Table 1 reports an overall average improvement of 0.87 percentage points, with decreases in some model–selector–benchmark combinations. This suggests that the benefits depend to some extent on the evaluation setting.

Questions for authors

1)When all observed frames have identical relevance scores, how does the algorithm handle the zero denominator in the normalization? 2)When the score range is very small, how does the method prevent minor noise or numerical errors from producing consequential differences in acquisition priority?

2 comments
AuthorsOpenReview ↗

[Response to Reviewer Kx6o (1/2)]

We thank the reviewer for recognizing QUEST's interpretability, practical integration, and broad evaluation, and for carefully examining its limitations under imperfect relevance estimates and the consistency of its improvements.

W1. Evidence missed between observed endpoints. Low relevance at observed endpoints does not rule out brief, critical evidence within the intervening interval. Observing more frames can reduce such omissions, but increases computational cost. Large video language model pipelines therefore limit the number of frames observed or selected. Within this budget, QUEST prioritizes the largest relevance-weighted temporal gap, allowing sufficiently large intervals to receive further observations even when their endpoints score low.

We first tested whether this temporal-gap term helps acquire evidence absent from the initial observations. We compared QUEST with a relevance-only variant that removes only the temporal-gap term, keeping initialization, scorer, midpoint placement, and observation budget N=128N=128 unchanged. The metric is the percentage of initially missed questions for which subsequent acquisition captures at least one annotated evidence region. Ranges cover CLIP-L/14, SigLIP, and LongCLIP, with gains computed from paired comparisons within each scorer.

BenchmarkInitially missed questionsRelevance-only (%)QUEST (%)Δ\Delta (pp)
LongVideoBench65236.20–43.2565.49–65.80+22.24–+29.29
VideoMME29136.77–41.9279.04–79.73+37.11–+42.61
CG-Bench-mini1,08934.07–36.5564.10–65.01+27.55–+30.30

QUEST improves recovery by 22.24–42.61 pp across all nine settings, with all paired video-cluster bootstrap pointwise 95% CIs excluding zero. Thus, the benefit is not limited to allocating more frames to evidence already captured by the initial observations.

We then examined whether this behavior also occurs when the initial local relevance signal is low. We considered brief annotated intervals missed by the initial observations and located between two consecutive initial frames, both outside all annotated evidence regions for the question. Brief intervals were defined as shorter than the video's maximum inter-frame temporal gap under uniform 128 frame sampling. Both endpoint scores had to lie in the lower half of scores from the initial WW observations. Using annotations only for evaluation, we grouped intervals by the minimum number of midpoint refinements along the event-containing branch needed to reach them. By N=128N=128, QUEST recovered all targets requiring one or two such refinements in each of the nine settings, with 19–39 targets per setting, compared with 0–2.56% for relevance-only. The complete refinement-depth breakdown, including intervals requiring deeper refinement, is reported in Appendix D.2.

Together, these analyses show that QUEST acquires previously unseen evidence even when initial endpoint relevance is low, rather than confining acquisition to initially high-scoring regions. Brief events requiring deeper refinement can still be missed under the fixed budget.

W2. Query relevance and marginal evidence. (i) Alignment with annotated evidence. We tested whether relevance scores distinguish human-annotated evidence on uniform 128-frame pools. AUC is averaged over questions. We also report candidate-pool quality from Appendix D. Ranges cover CLIP, SigLIP, and LongCLIP.

BenchmarkScorer AUCPool metricUniformQUEST
LongVideoBench0.715–0.778UKSS ×10019.2819.44–19.78
VideoMME0.709–0.724UKSS ×10032.4833.03–33.42
CG-Bench-mini0.729–0.733Clue-hit (%)69.9570.62–71.37

The scorers provide an informative signal (AUC > 0.7), and QUEST improves the reported annotation-based pool-quality metrics across all three scorers. To assess downstream benefits, Table 1 changes only candidate construction while keeping the scorer, selector, and frame budgets fixed, yielding a +0.87 pp average QA gain.

(ii) Marginal answer utility. These results support relevance-guided acquisition, but do not imply that relevance scores estimate the marginal answer utility of individual frames. Marginal answer utility depends on existing observations and is not determined by relevance scores alone. Directly modeling it requires an additional utility estimate under current observations. QUEST instead uses relevance as a bounded weight on temporal gaps, without equating relevance with marginal information gain.

AuthorsOpenReview ↗

[Response to Reviewer Kx6o (2/2)]

W3. Magnitude and consistency of improvements. We examined the variation across benchmarks rather than relying on the overall average alone. For all 100 matched Table 1 comparisons, we used 50,000 paired video-cluster bootstrap replicates, preserving shared-video dependencies and the original benchmark metrics. Each benchmark result averages 20 model-selector comparisons. The table reports pointwise 95% CIs and p-values from two-sided video-cluster sign-flip tests.

BenchmarkΔ\Delta Accuracy (pp)95% CI (pp)pWin / Tie / Loss
LongVideoBench+0.920[+0.334, +1.511]0.0023816 / 1 / 3
MLVU+1.031[+0.541, +1.519]0.0000718 / 0 / 2
VideoMME+0.196[-0.277, +0.674]0.4253114 / 0 / 6
LVBench+0.981[+0.114, +1.878]0.0312619 / 0 / 1
CG-Bench+1.243[+0.461, +2.031]0.0021520 / 0 / 0
Overall+0.874[+0.580, +1.175]<0.00187 / 1 / 12

The analysis confirms that the magnitude depends on the evaluation setting. In particular, VideoMME has a smaller gain whose CI includes zero. Nevertheless, the overall mean improvement is statistically supported. Removing any single benchmark retains a positive average gain of +0.782 to +1.044 pp, with all corresponding 95% CIs above zero. The positive average therefore does not depend on one benchmark, while the benchmark-specific results retain the variation and individual regressions noted in the review.

Q1, Q2. Min-max normalization. The identical-score and near-constant-score cases are handled by the same safeguard. Before normalization, we check the range of the observed raw relevance scores (max-min). When this range is at most 10−410^{-4}, we set all interval relevance weights to 1, so acquisition depends only on temporal gaps and refines the largest gap. This avoids division by zero and prevents differences within this small range from being amplified into acquisition priorities. The safeguard is now specified in Section 4.1 and Algorithm 1 in Appendix C.

We have added the evidence-recovery and refinement-depth analyses in Appendices D.1-D.2 and the scorer-annotation alignment analysis in Appendix D.3, with a summary of the candidate-level results in Section 5.3. Section 5.2 summarizes the statistical evidence for the average improvement, and Appendix F provides the benchmark-level results and robustness analyses. We have also clarified the role of relevance and the normalization safeguard in Section 4.1 and updated the pseudocode in Appendix C.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers and Area Chair

We thank all reviewers for their careful evaluation and constructive feedback. We appreciate their recognition of the following strengths of QUEST.

Strengths highlighted by reviewers

  • Practical motivation. Candidate construction addresses an important limitation of video QA pipelines, as evidence omitted at this stage cannot be recovered by downstream selection.
  • Simple and modular design. QUEST is training-free, uses an interpretable relevance-weighted temporal-gap rule, and reuses existing relevance scorers without modifying downstream selectors.
  • Broad evaluation. Experiments span five VLMs, four frame selectors, and five benchmarks under matched budgets, with additional annotation-based candidate pool evaluations.

Main clarifications

  • Methodological contribution. We have clarified that QUEST independently formulates candidate construction as relevance-weighted temporal quantization under a fixed observation budget, directly targeting the worst-case query-aware covering error. We have also clarified its differences from existing adaptive methods and the scope of its one-step minimax guarantee (Sections 1, 2.2, 3.3, and 6).
  • Controlled comparisons. We conducted additional experiments under matched scoring configurations and acquisition budgets. Replacing CSES's acquisition rule with QUEST while retaining the same selector improves accuracy across all five benchmarks (+0.61 pp on average). Separately, matching QUEST's per-sample acquisition-stage scoring count to FOCUS yields a +1.84 pp average gain, with improvements on four of five benchmarks. These results are included in Section 5.2 (Tables 4−5), with detailed settings in Appendix A.6.
  • Statistical reliability. Across 100 matched comparisons, QUEST achieves an average gain of +0.874 pp (95% CI [+0.580, +1.175], p < 0.001). Leave-one-out analyses further confirm that the positive average is not driven by a single benchmark, model, or selector. Section 5.2 summarizes the statistical evidence, and Appendix F provides the full analysis.

We have incorporated these clarifications and additional findings into the revised manuscript. We appreciate the reviewers' feedback and welcome further discussion.

Area_Chair_XhNuOpenReview ↗

Official Comment by Area Chair XhNu

Dear reviewers,

Thank you very much for your time and effort in reviewing this submission. The authors have now submitted their rebuttal, and the discussion period is nearing its end. We would greatly appreciate it if you could take some time to carefully review the authors’ responses and share your assessment of whether your concerns have been adequately addressed. Thank you again for your valuable time and contributions to the review process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.