acceptodds
Under review as a conference paper at ICLR 2027

Q-GPS: Query-Guided Progressive Frame Selection for Long Video Question Answering

11 views

Abstract

Frame selection enables multimodal large language models (MLLMs) to process long videos within limited visual-token budgets by retaining only a small subset of frames. Highly relevant frames may concentrate within the same short interval, leaving other query-relevant moments unrepresented. Effective selection therefore requires assessing both candidate relevance and the additional temporal coverage each frame provides given previous selections. We propose Query-Guided Progressive Frame Selection (Q-GPS), a training-free, model-agnostic method that progressively selects frames according to relevance-calibrated gains in query-weighted temporal coverage. It initializes its residual state with a query-independent path-spectral temporal covariance that models temporal coupling, and uses this state as a surrogate for the temporal coverage remaining after previous selections. At each step, it **selects** the frame with the largest relevance-calibrated coverage gain, **conditions** the covariance on the selected frame's temporal location, and **rescores** the remaining candidates using the updated residual state. Across controlled evaluations with Qwen2.5-VL-7B and frame budgets of 8, 16, and 32, Q-GPS improves average accuracy over uniform sampling by 4.70%, 7.28%, and 11.05% on Video-MME, LongVideoBench, and MLVU, respectively. Code is available at https://anonymous.4open.science/r/Q-GPS-FEB4.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
long video understanding, multimodal large language models, frame selection, video question answering, query-guided frame selection, training-free methods, efficient multimodal inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_41ymOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • Q-GPS introduces a query-aware selection criterion that combines direct candidate relevance with the conditioning-induced reduction of a query-weighted residual coverage objective over currently unselected locations.

  • Q-GPS realizes this criterion with a path-spectral temporal covariance and an exact low-rank implementation, using a closed-form path basis to avoid numerical eigendecomposition and dense covariance storage.

  • Q-GPS demonstrates accuracy gains through controlled comparisons and cross-backbone evaluations, and uses selection-behavior analysis and ablations to characterize the relevance-coverage trade-off and the benefit of progressive updates.

Critical strengths
  • Clear motivation. Q-GPS accounts for both relevance and temporal span, significantly improving accuracy in long-video question answering; it effectively resolves the conflict inherent in traditional methods—namely, the lack of focus in uniform sampling versus the excessive redundancy and clustering often found in relevance-based sampling.
  • The content is comprehensive, providing thorough quantitative and qualitative analysis.
Critical weaknesses
  • If Q-GPS is a training-free method, would it be equally effective for closed-source models or API-based models, such as Gemini.
  • The keyframe selection scheme dates back to early 2025. Given the advent of natively multimodal training and the expansion of context windows to 1M tokens, it remains to be seen whether the Q-GPS approach remains effective for denser frame selection, especially with larger input frame sequences.
  • I find the analysis in Section 4.4 and Figure 6 confusing; how should I interpret the x-axis ("ms/question") and its corresponding logarithmic scale? Additionally, I think it would be helpful to compare latency across Top-k results.
  • In the end-to-end comparisons presented in Appendix C.5 and Figure 10, uniform sampling should also be included to allow readers to clearly discern the additional latency introduced by Q-GPS.
Questions for authors
  • Please refer to the weaknesses.
5 comments
AuthorsOpenReview ↗

Response to Reviewer 41ym [1/3]

We sincerely appreciate the reviewer's careful scrutiny and valuable feedback. We address each concern point by point. All revisions and additions made to the manuscript have been marked in blue to aid in your review.

To Weakness 1:

If Q-GPS is a training-free method, would it be equally effective for closed-source models or API-based models, such as Gemini.

Q-GPS performs frame selection independently of the downstream MLLM, using a separate frozen relevance scorer and temporal coverage updates. It requires no additional training or access to the downstream model’s weights, gradients, or hidden representations. The selected frames can therefore be supplied directly to a multimodal API.

We evaluate Gemini 3.8 Flash and GPT-6.1-sol on the full evaluation splits of Video-MME, LongVideoBench, and MLVU across five frame budgets. For each model and budget, Uniform and Q-GPS use identical prompts, visual preprocessing, and generation settings; only the selected frames differ. Q-GPS uses the shared parameter settings specified in the paper. The results are presented in Figure 3 of the revised manuscript.

Accuracy (%), shown as Uniform → Q-GPS:

API modelFramesVideo-MMELongVideoBenchMLVU
Gemini 3.8 Flash877.19 → 80.2664.92 → 70.3864.91 → 79.88
Gemini 3.8 Flash1680.41 → 82.7468.36 → 73.9769.06 → 82.15
Gemini 3.8 Flash3283.81 → 84.7870.46 → 76.4474.63 → 84.25
Gemini 3.8 Flash6486.33 → 87.1974.72 → 77.3481.54 → 85.30
Gemini 3.8 Flash12888.04 → 88.8577.19 → 78.0185.76 → 87.53
GPT-6.1-sol874.59 → 77.3064.92 → 71.3561.61 → 75.51
GPT-6.1-sol1677.85 → 81.2668.66 → 74.3567.22 → 79.61
GPT-6.1-sol3283.70 → 84.7871.88 → 77.1172.48 → 82.41
GPT-6.1-sol6485.37 → 86.3375.77 → 77.3482.22 → 85.29
GPT-6.1-sol12886.93 → 88.2279.13 → 80.1884.07 → 86.22

Q-GPS improves over Uniform in all 30 comparisons. Averaged equally over the 15 benchmark–budget settings per model, accuracy increases from 76.49% to 81.27% (+4.78%) for Gemini and from 75.76% to 80.48% (+4.72%) for GPT-6.1-sol. On both models, 16 Q-GPS-selected frames outperform 32 uniformly sampled frames on LongVideoBench and MLVU.

AuthorsOpenReview ↗

Response to Reviewer 41ym [2/3]

To Weakness 2:

The keyframe selection scheme dates back to early 2025. Given the advent of natively multimodal training and the expansion of context windows to 1M tokens, it remains to be seen whether the Q-GPS approach remains effective for denser frame selection, especially with larger input frame sequences.

We thank the reviewer for raising this concern. To stress-test Q-GPS with longer input sequences and larger token budgets, we evaluate 8, 16, 32, 64, and 128 frames using a per-frame pixel cap of 602,112602{,}112, matching the default maximum in Qwen2.5-VL’s official video preprocessing (Appendix D.6).

Each cell below reports Uniform → Q-GPS accuracy(%):

FramesVideo-MMELongVideoBenchMLVU
853.89 → 59.6752.88 → 61.4155.38 → 66.15
1657.89 → 62.6756.92 → 63.2858.02 → 68.91
3261.78 → 64.8958.04 → 63.9561.29 → 72.92
6463.41 → 66.5660.51 → 64.3264.20 → 73.66
12865.48 → 67.7463.05 → 65.4568.80 → 74.18

The narrowing advantage over Uniform is consistent with denser sampling reducing the risk of missing relevant evidence. At small budgets, Uniform leaves large temporal gaps, so brief, query-relevant moments can fall between sampled frames. Q-GPS uses query relevance and complementary temporal coverage to allocate these scarce input slots more effectively. As the budget increases, Uniform samples the video more densely and becomes more likely to capture those same moments. Consequently, Q-GPS’s absolute gains over Uniform narrow from 5.78%, 8.53%, and 10.77% at 8 frames to 2.26%, 2.40%, and 5.38% at 128 frames on Video-MME, LongVideoBench, and MLVU, respectively.

This also explains why Q-GPS can reach useful evidence coverage earlier. 32 Q-GPS frames already outperform 64 uniformly sampled frames on all three benchmarks, and exceed 128 uniformly sampled frames on LongVideoBench and MLVU. Once the key evidence is represented, additional frames may contribute overlapping or less relevant information, yielding smaller accuracy improvements. Remaining errors can also involve interpreting the available evidence, which increasing the frame count alone may not resolve.

We also measure downstream inference costs across all five budgets on the full 2,700-question Video-MME split:

FramesMean input tokensInference time (s/question)Maximum sampled GPU memory (GiB)
82,7641.56117.60
165,4052.91418.88
3210,6875.68321.56
6421,25210.67926.94
12842,38123.29237.69

Measurements use Qwen2.5-VL-7B, one model process per GPU, and batch size one. Relevance scoring and frame selection are precomputed.

These diminishing accuracy returns have a substantial computational cost. Increasing Q-GPS from 32 to 128 frames raises input tokens by 3.97×3.97×, inference time by 4.10×4.10×, and sampled GPU memory by 1.75×1.75×, while accuracy improves by 2.85%, 1.50%, and 1.26% across the three benchmarks. Larger context windows expand the available capacity; our results show that allocating that capacity effectively remains valuable, because additional input does not yield proportional accuracy improvements.

AuthorsOpenReview ↗

Response to Reviewer 41ym [3/3]

To Weaknesses 3 and 4:

I find the analysis in Section 4.4 and Figure 6 confusing; how should I interpret the x-axis ("ms/question") and its corresponding logarithmic scale? Additionally, I think it would be helpful to compare latency across Top-k results. In the end-to-end comparisons presented in Appendix C.5 and Figure 10, uniform sampling should also be included to allow readers to clearly discern the additional latency introduced by Q-GPS.

In the revised manuscript, we have updated Sec. 4.4 and Figure 7 to clarify the timing comparison and include Top-K and Uniform.

In Figure 7(a), the x-axis is now labeled “Selector time (ms/question; log).” It shows the average CPU time spent selecting frames for one video–question pair, using precomputed relevance scores. This measures the selection algorithm alone; video decoding, relevance scoring, and downstream MLLM inference are excluded. To make algorithm-level timing differences easier to interpret and avoid differences in process-level parallelism, we measure all methods under the same single-process CPU setting. The tick labels are actual times in milliseconds: 11, 1010, 100100, and 1,0001{,}000 ms are equally spaced, with each interval representing a tenfold increase in time. Thus, points farther left indicate faster frame selection; the logarithmic scale allows methods with widely different selector times to be compared in the same plot. We have also added Top-K under this timing protocol at frame budgets of 88, 1616, and 3232 frames, with selector times of 0.530.53, 0.730.73, and 1.691.69 ms, respectively.

Figure 7(b) now includes Uniform alongside Q-GPS, comparing end-to-end time from frame decoding through downstream inference. At 32 frames on Video-MME, the pipeline times are 1.83 seconds for Uniform and 4.47 seconds for Q-GPS. The additional 2.64 seconds mainly comes from candidate decoding and relevance scoring. Appendix D.5 provides the cost breakdown.

As specified in the main text, these costs are reported per video–question pair without cross-question reuse. For multiple questions about the same video, a dual-encoder scorer such as CLIP can reuse cached frame embeddings to amortize decoding and visual encoding, requiring only text encoding and similarity computation for each new relevance-scoring pass.

Reviewer_P1g5OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Q-GPS is a training-free frame selector for long-video QA. It builds a fixed temporal kernel by projecting onto the first m ≈ 2B low-frequency eigenvectors of a normalized path Laplacian over N = 512 candidates. It then greedily picks frames that maximize relevance × (query-weighted reduction in residual variance), and after each pick it applies a rank-one GP-style conditioning update. Relevance scores come from frozen BLIP-2 ITM. Under a controlled protocol with Qwen2.5-VL-7B (shared candidates and scorer, only the selection rule changes), it beats AKS, MDP3, WFS-SB and DIG on Video-MME, LongVideoBench and MLVU at B = 8/16/32. It also transfers to LLaVA-OV and InternVL3, with a video-clustered bootstrap significance test.

Critical strengths
  • The evaluation is careful. AKS, MDP3, WFS-SB and Q-GPS share the same candidates, scorer and downstream model, so differences come from the selection rule. Q-GPS wins all 9 benchmark–budget cells. The paired video-clustered bootstrap gives +1.57 [0.81, 2.32] over the strongest baseline. Robustness across 9 frozen scorers (Tab. 5) and an honest end-to-end cost breakdown (Tab. 10) make the result credible.

  • The method is simple to deploy: closed form, about 5 ms per question on CPU, one shared hyperparameter setting, and code is released.

Critical weaknesses
  • The claimed core mechanism contributes little, and the simplest competitor is missing. The paper's central idea is the cross-location, query-weighted coverage gain. But "w/o Cross-location Gain" (keeping only the j = i term, with updates) loses only 0.7/0.5/0.9 points. Removing the covariance update loses 3.3. So most of the benefit seems to come from relevance plus local temporal suppression after each pick, which is close to Top-K with temporal non-maximum suppression or a minimum-gap rule. That baseline, plus MMR or greedy DPP with a plain RBF temporal kernel, is not reported. Without it, the path-spectral construction is not shown to be needed.

  • The novelty is thin relative to existing tools. Greedy conditional variance reduction under a GP kernel is the classic sensor placement objective (Krause et al.). Greedy DPP MAP, as used in MDP3, is closely related. The kernel here is query-independent and content-independent: the visual graph does not help (C.1), and query-shaped spectra add +0.11 (C.3). In the end it is a smooth temporal kernel with bandwidth about N/(2B). The textbook closed-form eigenvectors of the path Laplacian (Prop. A.1.1) are presented as a contribution, and the covariance/conditioning vocabulary makes a simple method look more complex than it is.

Questions for authors
  • Please add Top-K + temporal NMS (gap tuned on one benchmark) and greedy variance reduction with an RBF temporal kernel of matched bandwidth. If these come within about 0.5 points of Q-GPS, my rating would drop. If Q-GPS clearly beats them, it would rise.

  • How were c_rank = 2, ν = 0.05 and τ = 0.15 chosen? Tab. 4 is on the test benchmarks. Were the baselines' hyperparameters tuned under the BLIP-2 scorer as well, or left at their defaults?

  • Frames are capped at about 100K pixels and B ≤ 32. Does the gain over Uniform and WFS-SB hold at B = 64 or 128, or at native resolution, where current long-video models usually operate?

  • Because of the normalized Laplacian, endpoint locations get about half the prior variance (d_1 = d_N = 1). Does this bias selection away from the start and end of the video?

5 comments
AuthorsOpenReview ↗

Response to Reviewer P1g5 [1/5]

We sincerely appreciate the reviewer's careful scrutiny and valuable feedback. We address each concern point by point. All revisions and additions made to the manuscript have been marked in blue to aid in your review.

To Weaknesses 1 and 2:

  1. Clarification of the methodological contribution.

The novelty appears limited relative to established GP-based variance reduction and greedy DPP MAP. The kernel is query- and content-independent, with no benefit from visual graphs and little gain from query-shaped spectra. It appears to be a smooth temporal kernel using standard path-Laplacian eigenvectors, while the covariance terminology may overcomplicate a simple method.

Q-GPS builds on established covariance conditioning and greedy variance reduction for GP-based observation selection, as well as analytic path modes. Rather than introducing a new conditioning rule, we contribute a query-guided progressive frame-selection criterion for long-video QA that integrates a path-spectral temporal kernel with relevance-calibrated, query-weighted residual coverage, supported by an exact spectral-coordinate implementation.

The proposed criterion combines two complementary roles of query–frame relevance:

\frac{\sum_{j\in\mathcal U_b}w_j(\Sigma_{ji}^{(b)})^2} {\Sigma_{ii}^{(b)}+\nu}.$$ Here, $r_i$ rewards the direct relevance of candidate $i$, while $w_j$ determines the importance of covering each remaining location $j$. The criterion thus considers both candidate relevance and the reduction in remaining query-weighted coverage demand. After each selection, covariance conditioning updates the residual state, and all remaining candidates’ coverage gains are recomputed **(Sec. 3.2, Eqs. (9)–(11))**. This criterion differs from greedy DPP MAP: DPP evaluates the marginal log-determinant gain of adding a candidate to the **selected set**, whereas Q-GPS evaluates its reduction of query-weighted residual coverage over **unselected locations**. The initial path-spectral kernel $K_T$ provides the temporal covariance structure, while query specificity enters through $r_i$ and $w_j$. To reduce selection overhead, we derive an exact low-rank implementation using the closed-form path eigenvectors. This implementation takes $3.94\sim34.17$ ms per question at frame budgets of 8-32 (**Figure 7(a)**). 2. **Interpretation of the component ablations.** > The claimed core mechanism contributes little, and the simplest competitor is missing. The paper's central idea is the cross-location, query-weighted coverage gain. But "w/o Cross-location Gain" (keeping only the j = i term, with updates) loses only 0.7/0.5/0.9 points. To clarify the roles of target weighting and cross-location aggregation, we summarize the relevant ablations below. All variants retain direct candidate relevance $r_i$ and use the same progressive covariance conditioning. Accuracy (%) is reported at **16 frames**, with changes relative to full Q-GPS in parentheses **(Table 5)**. | Variant | Self term $j=i$ | Cross-location terms $j\neq i$ | Covariance conditioning | Video-MME | LongVideoBench | MLVU | |---|---|---|---|---|---|---| | **Full Q-GPS** | Weighted by $w_i$ | Weighted by $w_j$ | Retained | **61.07** | **60.36** | **66.99** | | w/o Query Weighting | Uniform weight $1/N$ | Uniform weight $1/N$ | Retained | 59.63 (−1.44) | 59.09 (−1.27) | 66.23 (−0.76) | | w/o Cross-location Gain (self-only) | Weighted by $w_i$ | Removed | Retained | 60.37 (−0.70) | 59.84 (−0.52) | 66.09 (−0.90) | Importantly, the **self-only variant retains progressive covariance conditioning**. Although it removes the $j\neq i$ terms from the gain calculation, selecting a frame still updates the residual variances and covariances of other locations. Thus, this ablation does not remove cross-location interactions from the selection process; it isolates the additional contribution of explicitly aggregating coverage gains across unselected locations. Similarly, removing query-derived target weighting reduces accuracy on all three benchmarks, even though direct relevance calibration and covariance conditioning remain unchanged. These results show that **query-derived target weighting and explicit cross-location aggregation each provide additional gains within the progressive selection framework**. The relatively smaller drop from removing cross-location terms should therefore be interpreted as their incremental contribution when the other mechanisms remain active, rather than as a measure of the contribution of the complete query-guided progressive selection criterion.
AuthorsOpenReview ↗

Response to Reviewer P1g5 [2/5]

To Question 1:

Please compare Q-GPS with tuned Top-K + temporal NMS and RBF-based MMR, greedy DPP, and matched-bandwidth variance reduction to establish whether the path-spectral construction provides clear gains beyond simple relevance-based temporal suppression.

Thank you for suggesting these controls. We have added Top-K + temporal NMS, MMR + RBF, greedy DPP + RBF, and two matched-RBF variance-reduction controls. All methods share the same experimental setup. We tune the NMS gap factor and MMR/DPP trade-off parameters on Video-MME at 16 frames and hold them fixed for the remaining eight settings. The RBF bandwidth is matched to the path reference. The added controls are defined in Appendices C.1.2–C.1.3, with tuning details in Appendix C.2 and full comparison results in Appendix C.3 (Table 8) of the revised manuscript.

Each dataset column lists accuracy (%) at 8 / 16 / 32 frames. Mean equally averages the nine dataset–budget settings.

MethodVideo-MMELongVideoBenchMLVUMean
Top-K55.37 / 58.44 / 61.3357.97 / 59.54 / 61.2662.08 / 65.05 / 68.3061.04
Top-K + NMS56.78 / 60.59 / 61.9356.99 / 58.12 / 60.5161.41 / 63.96 / 67.4660.86
MMR + RBF56.52 / 59.96 / 62.5256.92 / 58.41 / 61.0362.77 / 65.63 / 69.3161.45
DPP + RBF56.44 / 60.19 / 62.8557.67 / 59.16 / 62.0861.63 / 67.11 / 69.2161.82
Relevance-VR + RBF56.30 / 59.96 / 62.4158.71 / 59.54 / 61.5661.94 / 65.27 / 69.0161.63
Q-GPS with RBF57.15 / 60.30 / 63.3359.39 / 60.73 / 62.4562.92 / 66.33 / 69.6662.47
Q-GPS57.85 / 61.07 / 64.4460.21 / 60.36 / 63.5064.16 / 66.99 / 70.5863.24

Q-GPS obtains the highest mean accuracy and the best result in seven of nine settings. Its mean gains over NMS, MMR, and DPP are 2.38%, 1.79%, and 1.42%, respectively. These controls capture different notions of temporal diversity: NMS enforces a minimum gap, MMR penalizes temporal similarity to selected frames, and DPP rewards joint temporal diversity of the selected set. Q-GPS combines a candidate’s direct relevance with its reduction of query-weighted residual coverage across remaining locations, recomputing this gain after each selection (Sec. 3.2, Eqs. (9)–(10)).

Q-GPS exceeds Top-K + NMS by 2.38% on average, and improves all nine settings. The gains averaged over frame budgets are 1.35%, 2.82%, and 2.97% on Video-MME, LongVideoBench, and MLVU, respectively. NMS improves over Top-K on Video-MME but falls below it on LongVideoBench and MLVU when the same gap factor is transferred. LongVideoBench emphasizes locating and relating query-specific moments, while MLVU includes tasks requiring different evidence distributions, from brief details to events spread across a video (Appendix B.2). These tasks can require both closely spaced observations and distant events, so greater temporal separation need not preserve the most useful evidence.

Q-GPS exceeds Relevance-VR + RBF by 1.61% on average, and improves all nine settings. The RBF controls separate query-derived target weighting from kernel choice while retaining direct candidate relevance and progressive covariance conditioning. Relevance-VR + RBF weights all remaining target locations equally. Introducing Q-GPS’s query-derived target weights gives Q-GPS with RBF, raising mean accuracy from 61.63% to 62.47% (+ 0.84%). Replacing only the initial RBF kernel with the path-spectral covariance raises it further to 63.24% (+ 0.77%). The truncated path spectrum can produce secondary peaks in correlation magnitude at separated temporal locations, whereas initial RBF correlations decrease monotonically with temporal distance. These comparisons support contributions from both query-weighted target coverage and the initial temporal kernel structure.

We have also uploaded the code for these temporal baselines to our anonymous repository.

AuthorsOpenReview ↗

Response to Reviewer P1g5 [3/5]

To Question 2.1:

How were c_rank = 2, ν = 0.05 and τ = 0.15 chosen? Tab. 4 is on the test benchmarks.

Our study uses the Video-MME test split (2700 questions), LongVideoBench validation split (1337 questions), and MLVU development split (2174 questions), which provide public questions and ground-truth answers. LongVideoBench and MLVU test labels are withheld. Split names follow the respective benchmark protocols, and all three splits serve as evaluation datasets (Appendix B.2).

Prior frame-selection work also examines hyperparameters on these benchmarks: AKS (Table 5) reports joint sweeps on Video-MME and LongVideoBench, while WFS-SB (Sec. 4.3) uses a shared configuration across datasets, backbones, and frame budgets. We select hyperparameters using the public labels of the Video-MME test split at 1616 frames, then apply the configuration to all other settings without further tuning.

We fix the observation-noise variance at the heuristic default ν=0.05\nu=0.05 and evaluate all 2525 combinations of crank∈{0.5,1,2,3,4}c_{\mathrm{rank}}\in\lbrace 0.5,1,2,3,4\rbrace and τ∈{0.05,0.10,0.15,0.20,0.25}\tau\in\lbrace0.05,0.10,0.15,0.20,0.25\rbrace. The selected pair, (crank,τ)=(2,0.15)(c_{\mathrm{rank}},\tau)=(2,0.15), achieves 61.07% accuracy, tied for the highest grid result. All three hyperparameters are then fixed for the remaining eight dataset–budget settings: Video-MME at 88 and 3232 frames, and LongVideoBench and MLVU at 88, 1616, and 3232 frames. The relevance scorer and downstream MLLM remain frozen throughout. The full grid is reported in Appendix C.2.7, Figure 14.

We also conduct a broader hyperparameter sensitivity analysis beyond the search grid on Video-MME and MLVU at 1616 frames, varying one hyperparameter at a time while fixing the other two at the shared configuration. The ranges span crank=0.5∼8c_{\mathrm{rank}}=0.5 \sim 8, ν=0.001∼1\nu=0.001 \sim 1, and τ=0.05∼0.50\tau=0.05 \sim 0.50. Across the 3030 evaluations excluding the shared configuration, the mean absolute accuracy deviation from it is 0.44% (Table 3), indicating limited sensitivity over these broader ranges.

To Question 2.2:

Were the baselines' hyperparameters tuned under the BLIP-2 scorer as well, or left at their defaults?

In the controlled comparisons of Table 1, AKS, MDP³, and WFS-SB use the same frozen BLIP-2-ITM ViT-g scorer, candidate pool, and Qwen2.5-VL-7B inference settings as Q-GPS, while retaining their published default selection hyperparameters. We now additionally report comparisons with AKS and MDP³ tuned under this shared scorer.

Both searches use the full 2700-question Video-MME test split at B=16B=16, matching the calibration setting used for Q-GPS. For each method, we select the configuration with the highest unrounded accuracy and then keep it fixed for the other eight dataset–budget settings, without further dataset-specific or budget-specific tuning.

We search 3030 threshold–depth configurations for AKS and select (t1,d)=(0.6,3)(t_1,d)=(0.6,3). For MDP³, we search 2424 weight–segment-size configurations and select (λMDP,segment size)=(0.8,64)(\lambda_{\mathrm{MDP}},\text{segment size})=(0.8,64). WFS-SB retains its published BLIP-2-ITM configuration without retuning.

Full parameter settings and search details are provided in Appendices C.2.1–C.2.3 of the revised manuscript. The table below summarizes the corresponding video-selector results from Table 8. Each dataset column reports accuracy (%) at 8F / 16F / 32F, and Mean averages all nine dataset–budget settings equally.

MethodVideo-MMELongVideoBenchMLVUMean
AKS, tuned56.19 / 59.74 / 62.2658.04 / 59.24 / 61.7162.26 / 65.43 / 68.8161.52
MDP³, tuned57.22 / 60.26 / 63.7458.34 / 58.79 / 62.2362.93 / 65.74 / 69.0762.04
WFS-SB, published defaults57.22 / 60.59 / 62.8555.42 / 59.09 / 61.3363.77 / 65.41 / 69.3961.67
Q-GPS57.85 / 61.07 / 64.4460.21 / 60.36 / 63.5064.16 / 66.99 / 70.5863.24

Q-GPS outperforms all three baselines in all nine settings. Its mean accuracy is 63.24%, compared with 62.04% for tuned MDP³, the strongest of these baselines, yielding a mean difference of 1.20%.

Reviewer_nDnKOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Q GPS is a training free frame selection method that uses temporal covariance to build a frame set that maximizes need for coverage over the entire input video, balancing relevance and visibility.

Critical strengths
  • The experiments are clear and demonstrate the strength of the selector proposed
  • Thorough ablations that support each component
  • The method is fairly cheap and practical to implement
Critical weaknesses

See questions.

Questions for authors
  • How does this method compare to simpler naive alternatives like Top-K with non-max suppression (ie no two frames within a certain radius) - it is not clear to me that temporal covariance is needed.
  • Method would work well for long video summarization or needle in a haystack task - but not necessarily differentiating between subtle differences over a set of similar looking frames (eg a person eating with a fork vs a spoon.) Could the authors also add performance on EgoSchema?
  • How were the hyperparameters chosen? (eval or train set?)
5 comments
AuthorsOpenReview ↗

Response to Reviewer nDnK [1/3]

We sincerely appreciate the reviewer's careful scrutiny and valuable feedback. We address each concern point by point. All revisions and additions made to the manuscript have been marked in blue to aid in your review.

To Question 1:

How does this method compare to simpler naive alternatives like Top-K with non-max suppression (ie no two frames within a certain radius) - it is not clear to me that temporal covariance is needed.

We added controlled comparisons with Top-K and Top-K + temporal NMS, using the same candidate pool, frozen BLIP-2-ITM scores, frame budgets, and Qwen2.5-VL-7B inference settings. We tuned the NMS gap factor on Video-MME at 1616 frames, selecting α=0.25\alpha=0.25, and kept it fixed across the remaining settings. At the tuning budget, this corresponds to an initial minimum gap of eight candidate steps for 512512 candidates. The search results are reported in Appendix C.2.4 and Figure 11 of the revised manuscript.

Accuracy (%); each dataset column reports results at 8F/16F/32F. Mean averages all nine settings.

MethodVideo-MMELongVideoBenchMLVUMean
Top-K55.37 / 58.44 / 61.3357.97 / 59.54 / 61.2662.08 / 65.05 / 68.3061.04
Top-K + NMS56.78 / 60.59 / 61.9356.99 / 58.12 / 60.5161.41 / 63.96 / 67.4660.86
Q-GPS57.85 / 61.07 / 64.4460.21 / 60.36 / 63.5064.16 / 66.99 / 70.5863.24

Q-GPS outperforms both baselines in all nine settings, with mean gains of +2.20% over Top-K and +2.38% over tuned NMS (Appendix C.3, Table 8). NMS improves Video-MME accuracy at all three budgets, but falls below Top-K on LongVideoBench and MLVU.

NMS excludes nearby candidates through a hard minimum-spacing rule, which can discard frames containing complementary evidence. This is particularly relevant to LongVideoBench: DIG categorizes all questions in this benchmark as localized and finds that including surrounding temporal context is especially beneficial (Appendix B and Sec. 6.4 of DIG[1]).

After each selection, Q-GPS conditions the temporal covariance and recomputes each remaining frame’s query-weighted coverage gain (Sec. 3.2, Eqs. 8–10). Combining this gain with frame relevance allows nearby frames to be selected when they still provide complementary evidence.

[1] Divide, then Ground: Adapting Frame Selection to Query Types for Long-Form Video Understanding (CVPR 2026)

AuthorsOpenReview ↗

Response to Reviewer nDnK [2/3]

To Question 2:

Method would work well for long video summarization or needle in a haystack task - but not necessarily differentiating between subtle differences over a set of similar looking frames (eg a person eating with a fork vs a spoon.) Could the authors also add performance on EgoSchema?

Our benchmarks cover complementary aspects of video understanding: Video-MME (average duration: ~17 min) evaluates diverse abilities, including global comprehension; LongVideoBench (~8 min) emphasizes locating and reasoning over relevant details; and MLVU (~12.6 min) covers holistic, single-detail, and multi-detail understanding. Strong performance across these benchmarks demonstrates the effectiveness of balancing query relevance and temporal coverage.

We additionally evaluate the publicly labeled 500-question subset of EgoSchema. EgoSchema consists of 3-minute egocentric videos and emphasizes long-range temporal reasoning, with a median temporal certificate length of approximately 100 seconds, as reported in the original paper. Q-GPS and all baselines retain their settings and hyperparameters from the main experiments.

Method8F16F32FMean
Uniform53.0056.8058.4056.07
Top-K51.8054.4057.6054.60
AKS50.8058.2057.6055.53
WFS-SB52.0057.6059.6056.40
Q-GPS53.6058.6058.8057.00

Q-GPS achieves the highest mean accuracy (57.00%), outperforming Uniform and Top-K by 0.93% and 2.40%, respectively. Top-K underperforms Uniform at all three budgets. Repetitive egocentric content may lead relevance-only selection to concentrate on visually similar frames and overlook complementary temporal evidence. In contrast, Q-GPS balances query relevance and temporal coverage to preserve evidence across activity stages; Figure 22 in the revised manuscript qualitatively illustrates how Q-GPS preserves discriminative action evidence in Video-MME: it retains the queried handstand pose and supports the correct answer, whereas Uniform and Top-K yield incorrect answers.

AuthorsOpenReview ↗

Response to Reviewer nDnK [3/3]

To Question 3:

How were the hyperparameters chosen? (eval or train set?)

Our study uses the Video-MME test split (2700 questions), LongVideoBench validation split (1337 questions), and MLVU development split (2174 questions), which provide public questions and ground-truth answers. LongVideoBench and MLVU test labels are withheld. Split names follow the respective benchmark protocols, and all three splits serve as evaluation datasets (Appendix B.2).

Prior frame-selection work also examines hyperparameters on these benchmarks: AKS (Table 5) reports joint sweeps on Video-MME and LongVideoBench, while WFS-SB (Sec. 4.3) uses a shared configuration across datasets, backbones, and frame budgets. We select hyperparameters using the public labels of the Video-MME test split at 1616 frames, then apply the configuration to all other settings without further tuning.

We fix the observation-noise variance at the heuristic default ν=0.05\nu=0.05 and evaluate all 2525 combinations of crank∈{0.5,1,2,3,4}c_{\mathrm{rank}}\in\lbrace0.5,1,2,3,4\rbrace and τ∈{0.05,0.10,0.15,0.20,0.25}\tau\in\lbrace0.05,0.10,0.15,0.20,0.25\rbrace. The selected pair, (crank,τ)=(2,0.15)(c_{\mathrm{rank}},\tau)=(2,0.15), achieves 61.07% accuracy, tied for the highest grid result. All three hyperparameters are then fixed for the remaining eight dataset–budget settings: Video-MME at 88 and 3232 frames, and LongVideoBench and MLVU at 88, 1616, and 3232 frames. The relevance scorer and downstream MLLM remain frozen throughout. The full grid is reported in Appendix C.2.7, Figure 14.

We also conduct a broader hyperparameter sensitivity analysis beyond the search grid on Video-MME and MLVU at 1616 frames, varying one hyperparameter at a time while fixing the other two at the shared configuration. The ranges span crank=0.5∼8c_{\mathrm{rank}}=0.5 \sim 8, ν=0.001∼1\nu=0.001 \sim 1, and τ=0.05∼0.50\tau=0.05 \sim 0.50. Across the 30 evaluations excluding the shared configuration, the mean absolute accuracy deviation from it is 0.44% (Table 3), indicating limited sensitivity over these broader ranges.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Author Responses and Revisions

Dear Area Chair and Reviewers,

We thank the reviewers for their constructive feedback and for recognizing Q-GPS's careful evaluation and practical efficiency (P1g5), strong results and thorough ablations (nDnK), and clear motivation and comprehensive analysis (41ym). We have provided additional experiments and clarifications in response to their main concerns, with corresponding changes highlighted in blue in the revised manuscript.

1. Methodological contribution and simpler alternatives. Q-GPS introduces a training-free frame-selection criterion combining direct query relevance with query-weighted residual temporal coverage through progressive covariance conditioning. Our original controlled experiments show statistically supported improvements over established methods. Additional comparisons with simpler temporal selection methods and matched-RBF controls show that Q-GPS achieves the highest mean accuracy across nine settings. These results indicate that its gains cannot be fully explained by simple temporal suppression and further support the contributions of query-weighted coverage and the path-spectral kernel (P1g5, nDnK).

2. Generalization and scalability. Beyond our original cross-backbone and cross-scorer evaluations, we added EgoSchema experiments to further examine the value of complementary temporal evidence for long-range reasoning (nDnK). Evaluations at 64/128 frames and two resolutions show that effective frame allocation remains valuable at larger input budgets (P1g5, 41ym). Experiments on Gemini 3.8 Flash and GPT-6.1-sol show improvements in all 30 comparisons, demonstrating applicability to closed-source MLLMs without internal model access (41ym).

3. Experimental rigor and efficiency. We clarified that hyperparameters were selected on the public Video-MME test split at 16 frames and transferred unchanged to the remaining settings. We also retuned AKS and MDP³ under the same calibration protocol and expanded sensitivity analyses (P1g5, nDnK). Further analyses addressed boundary effects (P1g5) and distinguished selector runtime from end-to-end processing costs (41ym).

Supporting results are summarized in the reply below. We thank the Area Chair for their time and consideration.

Best regards,
The Authors

AuthorsOpenReview ↗

Summary of Additional Experimental Results

We summarize the key additional results below. Complete experimental settings and analyses are provided in the revised manuscript and our individual reviewer responses.

1. Comparisons with Simpler Alternatives (P1g5, nDnK)

MethodMean Accuracy (%)
Top-K61.04
Top-K + NMS60.86
MMR + RBF61.45
DPP + RBF61.82
Relevance-VR + RBF61.63
Q-GPS with RBF62.47
Q-GPS63.24

Mean accuracy across Video-MME, LongVideoBench, and MLVU at 8/16/32 frames (nine settings). All comparisons use matched experimental settings. Full results: Appendix C.3, Table 8.

Q-GPS improves mean accuracy by 2.38% over tuned Top-K + NMS and by 1.61% over Relevance-VR + RBF. The matched-RBF controls further support the separate contributions of query-derived target weighting and the path-spectral kernel.

2. Generalization and Scalability (nDnK, P1g5, 41ym)

Additional EvaluationKey Result
EgoSchema at 8/16/32 frames (nDnK)Highest mean accuracy of 57.00%
64/128 frames at two resolutions (P1g5, 41ym)Outperforms Uniform in 12/12 settings and WFS-SB in 10/12
Gemini 3.8 Flash (8–128 frames) (41ym)Mean accuracy gain of +4.78% over Uniform
GPT-6.1-sol (8–128 frames) (41ym)Mean accuracy gain of +4.72% over Uniform

Full results: Figure 3, Appendix D.6, and our EgoSchema response to Reviewer nDnK.

3. Experimental Rigor and Efficiency (P1g5, nDnK, 41ym)

  • Q-GPS outperforms retuned AKS and MDP³ in all nine settings under the shared BLIP-2-ITM scorer (Table 8).
  • Across 30 non-default evaluations, the mean absolute accuracy deviation from the shared configuration is 0.44% (Table 3).
  • At 16 frames, the first and last 5% of the candidate timeline together account for about 10% of the frames selected by Q-GPS across the three benchmarks. Unit-diagonal normalization yields no consistent accuracy gain (Response to P1g5, Question 4).
  • Figure 7(a) adds Top-K to the CPU selector timing comparison. Figure 7(b) adds Uniform to the end-to-end timing comparison.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.