- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
-
Q-GPS introduces a query-aware selection criterion that combines direct candidate relevance with the conditioning-induced reduction of a query-weighted residual coverage objective over currently unselected locations.
-
Q-GPS realizes this criterion with a path-spectral temporal covariance and an exact low-rank implementation, using a closed-form path basis to avoid numerical eigendecomposition and dense covariance storage.
-
Q-GPS demonstrates accuracy gains through controlled comparisons and cross-backbone evaluations, and uses selection-behavior analysis and ablations to characterize the relevance-coverage trade-off and the benefit of progressive updates.
Critical strengths
- Clear motivation. Q-GPS accounts for both relevance and temporal span, significantly improving accuracy in long-video question answering; it effectively resolves the conflict inherent in traditional methods—namely, the lack of focus in uniform sampling versus the excessive redundancy and clustering often found in relevance-based sampling.
- The content is comprehensive, providing thorough quantitative and qualitative analysis.
Critical weaknesses
- If Q-GPS is a training-free method, would it be equally effective for closed-source models or API-based models, such as Gemini.
- The keyframe selection scheme dates back to early 2025. Given the advent of natively multimodal training and the expansion of context windows to 1M tokens, it remains to be seen whether the Q-GPS approach remains effective for denser frame selection, especially with larger input frame sequences.
- I find the analysis in Section 4.4 and Figure 6 confusing; how should I interpret the x-axis ("ms/question") and its corresponding logarithmic scale? Additionally, I think it would be helpful to compare latency across Top-k results.
- In the end-to-end comparisons presented in Appendix C.5 and Figure 10, uniform sampling should also be included to allow readers to clearly discern the additional latency introduced by Q-GPS.
Questions for authors
- Please refer to the weaknesses.
5 comments
Response to Reviewer 41ym [2/3]
To Weakness 2:
The keyframe selection scheme dates back to early 2025. Given the advent of natively multimodal training and the expansion of context windows to 1M tokens, it remains to be seen whether the Q-GPS approach remains effective for denser frame selection, especially with larger input frame sequences.
We thank the reviewer for raising this concern. To stress-test Q-GPS with longer input sequences and larger token budgets, we evaluate 8, 16, 32, 64, and 128 frames using a per-frame pixel cap of , matching the default maximum in Qwen2.5-VL’s official video preprocessing (Appendix D.6).
Each cell below reports Uniform → Q-GPS accuracy(%):
| Frames | Video-MME | LongVideoBench | MLVU |
|---|---|---|---|
| 8 | 53.89 → 59.67 | 52.88 → 61.41 | 55.38 → 66.15 |
| 16 | 57.89 → 62.67 | 56.92 → 63.28 | 58.02 → 68.91 |
| 32 | 61.78 → 64.89 | 58.04 → 63.95 | 61.29 → 72.92 |
| 64 | 63.41 → 66.56 | 60.51 → 64.32 | 64.20 → 73.66 |
| 128 | 65.48 → 67.74 | 63.05 → 65.45 | 68.80 → 74.18 |
The narrowing advantage over Uniform is consistent with denser sampling reducing the risk of missing relevant evidence. At small budgets, Uniform leaves large temporal gaps, so brief, query-relevant moments can fall between sampled frames. Q-GPS uses query relevance and complementary temporal coverage to allocate these scarce input slots more effectively. As the budget increases, Uniform samples the video more densely and becomes more likely to capture those same moments. Consequently, Q-GPS’s absolute gains over Uniform narrow from 5.78%, 8.53%, and 10.77% at 8 frames to 2.26%, 2.40%, and 5.38% at 128 frames on Video-MME, LongVideoBench, and MLVU, respectively.
This also explains why Q-GPS can reach useful evidence coverage earlier. 32 Q-GPS frames already outperform 64 uniformly sampled frames on all three benchmarks, and exceed 128 uniformly sampled frames on LongVideoBench and MLVU. Once the key evidence is represented, additional frames may contribute overlapping or less relevant information, yielding smaller accuracy improvements. Remaining errors can also involve interpreting the available evidence, which increasing the frame count alone may not resolve.
We also measure downstream inference costs across all five budgets on the full 2,700-question Video-MME split:
| Frames | Mean input tokens | Inference time (s/question) | Maximum sampled GPU memory (GiB) |
|---|---|---|---|
| 8 | 2,764 | 1.561 | 17.60 |
| 16 | 5,405 | 2.914 | 18.88 |
| 32 | 10,687 | 5.683 | 21.56 |
| 64 | 21,252 | 10.679 | 26.94 |
| 128 | 42,381 | 23.292 | 37.69 |
Measurements use Qwen2.5-VL-7B, one model process per GPU, and batch size one. Relevance scoring and frame selection are precomputed.
These diminishing accuracy returns have a substantial computational cost. Increasing Q-GPS from 32 to 128 frames raises input tokens by , inference time by , and sampled GPU memory by , while accuracy improves by 2.85%, 1.50%, and 1.26% across the three benchmarks. Larger context windows expand the available capacity; our results show that allocating that capacity effectively remains valuable, because additional input does not yield proportional accuracy improvements.
Response to Reviewer 41ym [3/3]
To Weaknesses 3 and 4:
I find the analysis in Section 4.4 and Figure 6 confusing; how should I interpret the x-axis ("ms/question") and its corresponding logarithmic scale? Additionally, I think it would be helpful to compare latency across Top-k results. In the end-to-end comparisons presented in Appendix C.5 and Figure 10, uniform sampling should also be included to allow readers to clearly discern the additional latency introduced by Q-GPS.
In the revised manuscript, we have updated Sec. 4.4 and Figure 7 to clarify the timing comparison and include Top-K and Uniform.
In Figure 7(a), the x-axis is now labeled “Selector time (ms/question; log).” It shows the average CPU time spent selecting frames for one video–question pair, using precomputed relevance scores. This measures the selection algorithm alone; video decoding, relevance scoring, and downstream MLLM inference are excluded. To make algorithm-level timing differences easier to interpret and avoid differences in process-level parallelism, we measure all methods under the same single-process CPU setting. The tick labels are actual times in milliseconds: , , , and ms are equally spaced, with each interval representing a tenfold increase in time. Thus, points farther left indicate faster frame selection; the logarithmic scale allows methods with widely different selector times to be compared in the same plot. We have also added Top-K under this timing protocol at frame budgets of , , and frames, with selector times of , , and ms, respectively.
Figure 7(b) now includes Uniform alongside Q-GPS, comparing end-to-end time from frame decoding through downstream inference. At 32 frames on Video-MME, the pipeline times are 1.83 seconds for Uniform and 4.47 seconds for Q-GPS. The additional 2.64 seconds mainly comes from candidate decoding and relevance scoring. Appendix D.5 provides the cost breakdown.
As specified in the main text, these costs are reported per video–question pair without cross-question reuse. For multiple questions about the same video, a dual-encoder scorer such as CLIP can reuse cached frame embeddings to amortize decoding and visual encoding, requiring only text encoding and similarity computation for each new relevance-scoring pass.
Response to Reviewer 41ym [1/3]
We sincerely appreciate the reviewer's careful scrutiny and valuable feedback. We address each concern point by point. All revisions and additions made to the manuscript have been marked in blue to aid in your review.
To Weakness 1:
Q-GPS performs frame selection independently of the downstream MLLM, using a separate frozen relevance scorer and temporal coverage updates. It requires no additional training or access to the downstream model’s weights, gradients, or hidden representations. The selected frames can therefore be supplied directly to a multimodal API.
We evaluate Gemini 3.8 Flash and GPT-6.1-sol on the full evaluation splits of Video-MME, LongVideoBench, and MLVU across five frame budgets. For each model and budget, Uniform and Q-GPS use identical prompts, visual preprocessing, and generation settings; only the selected frames differ. Q-GPS uses the shared parameter settings specified in the paper. The results are presented in Figure 3 of the revised manuscript.
Accuracy (%), shown as Uniform → Q-GPS:
Q-GPS improves over Uniform in all 30 comparisons. Averaged equally over the 15 benchmark–budget settings per model, accuracy increases from 76.49% to 81.27% (+4.78%) for Gemini and from 75.76% to 80.48% (+4.72%) for GPT-6.1-sol. On both models, 16 Q-GPS-selected frames outperform 32 uniformly sampled frames on LongVideoBench and MLVU.