- Rating
- 2
- Confidence
- 3(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PartS, a training-free framework for query-aware keyframe selection in long-video understanding. It uses codec I-frames to initialize a compact candidate pool, adaptively refines query-relevant and visually diverse segments, and combines CLIP-based clustering with relevance-weighted frame allocation to select the final keyframes. Experiments on four benchmarks and four open-source MLLMs demonstrate competitive QA accuracy and 5.1–19.3× faster keyframe selection compared with existing training-free methods.
Critical strengths
S1. Significant efficiency improvements. PartS substantially reduces keyframe selection latency by limiting candidate-frame processing and using a lightweight CLIP encoder. The reported 5.1–19.3× speedups demonstrate practical value for long-video understanding.
S2. Extensive empirical evaluation. The method is evaluated across four long-video benchmarks, four MLLMs, and different frame budgets, showing consistent improvements over uniform sampling.
Critical weaknesses
W1. Limited methodological novelty. PartS primarily combines existing techniques, including codec-based I-frame extraction, heuristic adaptive sampling, CLIP similarity, K-Means clustering, and Softmax-based frame allocation. Notably, QCA: Query- and Content-Aware Keyframe Selection for Long Video Understanding (Peng et al., ECCV 2026) has already explored query-aware frame budget allocation across video segments. The proposed cluster-level allocation follows a similar principle. Overall, PartS appears to be largely an engineering combination of established components, with limited methodological novelty.
W2. Insufficient validation of the core design. Appendix Table A.2 compares adaptive and uniform candidate sampling but lacks sufficient details about candidate counts, initialization strategies, and matched latency conditions, making it difficult to establish the independent contribution of codec-based initialization. Moreover, inconsistencies between Eq. (3) and Algorithm 2 raise concerns about the implementation of soft frame allocation.
Questions for authors
Q1. Contribution of I-frame initialization. Could the authors compare I-frame initialization with an equal number of uniformly sampled initial frames, followed by exactly the same adaptive refinement procedure? Appendix Table A.2 does not provide sufficient details about the matched sampling and latency conditions, making the fairness of the comparison unclear.
Q2. Novelty of frame allocation. QCA has already proposed query-aware frame budget allocation based on semantic relevance and content diversity. What is the distinct contribution of PartS's cluster-level Softmax allocation beyond this existing approach? Could the authors provide a controlled ablation to demonstrate its independent effectiveness?
Q3. Soft allocation implementation. Eq. (3) uses floor rounding, whereas Algorithm 2 uses ceiling rounding. Under the default settings (K=32, c=100, τ=2), the former appears to allocate zero frames to every cluster, while the latter effectively allocates one frame per selected cluster. Could the authors clarify the actual implementation and explain the temperature sensitivity reported in Figure 5?
Q4. Generalization beyond multiple-choice QA. All evaluated benchmarks use multiple-choice questions, and Appendix A incorporates candidate answers into frame relevance scoring. How does PartS perform on open-ended video QA without answer options? Could the authors also evaluate closed-source MLLMs, such as GPT-6, to further validate its plug-and-play transferability?
Q5. Dependence on hand-crafted and dataset-specific hyperparameters. PartS relies on several manually designed thresholds and hyperparameters, whose choices lack sufficient justification and systematic ablation studies, making their individual contributions and interpretability unclear. Moreover, Appendix A determines β=0.95 and M=8 based on LVBench statistics and performance, raising concerns about dataset-specific tuning. Such dependence limits the robustness and practicality of the claimed plug-and-play framework in real-world scenarios. Could the authors provide more comprehensive ablations and demonstrate whether these hyperparameters generalize across datasets without additional tuning?
Q6. Robustness to video encoding configurations. Since I-frame locations depend on video encoding settings, how robust is PartS across different codecs and GOP configurations? Does the reported latency include video decoding, I-frame extraction, and random access to intermediate frames?
4 comments
Q2. Novelty of frame allocation.
QCA does not employ clustering; instead, it divides videos into equally spaced temporal segments and allocates frames based on query relevance and content deviation. In contrast, PartS clusters candidate frames in the visual feature space and distributes the frame budget across visually similar clusters. We emphasize that soft allocation is not, by itself, a claimed contribution of our work. Rather, our novelty lies in the overall pipeline, including the adaptive candidate sampling, to substantially reduce computational cost, achieving up to 19.3x speedup over existing baselines while maintaining competitive accuracy.
To evaluate the independent effectiveness of budget allocation across clusters, we conducted an additional controlled ablation. Without allocation, we partition candidate frames into (K=32) clusters using visual features and select the frame with the highest relevance score within each cluster. With allocation, PartS distributes the same frame budget across clusters using soft frame allocation as detailed in Section 3.3, keeping all other components unchanged. The results presented in the table below show that soft allocation consistently improves accuracy across both MLLMs and benchmarks. These results validate the effectiveness of allocation within PartS. We will include these additional ablation results in the appendix of the revised paper.
| MLLM | Allocation | LongVideoBench | Video-MME |
|---|---|---|---|
| LLaVA-OV | Without | 59.1 | 60.1 |
| With | 60.4 (+1.3) | 61.1 (+1.0) | |
| Qwen2.5-VL | Without | 60.8 | 62.5 |
| With | 63.9 (+3.1) | 63.3 (+0.8) |
Q3. Soft allocation implementation.
We thank the reviewer for identifying this inconsistency. We clarify that the actual implementation uses ceiling rounding, as specified in Algorithm 2, and that the floor operator in Eq. (3) is a typo. We will correct the equation in the revised manuscript to ensure consistency between the mathematical formulation and the implementation.
Algorithm 2 enforces the total frame budget (K) during allocation. Then, in the extreme case where all clusters have identical relevance scores, ceiling rounding assigns one frame per cluster, which would result in 100 frames and exceed the budget K=32. In this case, the algorithm will assign 1 frame to the first 32 visited clusters. It is important to highlight that, in practice, multiple frames are selected from relevant clusters. For instance, on Video-MME, to select 32 keyframes, the PartS samples frames from an average of 15 clusters, allocating approximately 5 frames to the most relevant cluster on average.
Regarding temperature sensitivity, \tau controls how much the allocation favors highly relevant clusters. Lower temperatures produce a more concentrated relevance distribution, prioritizing relevant clusters, whereas higher temperatures encourage a more uniform distribution, promoting visual diversity.
Q4. Generalization beyond multiple-choice QA.
Following common practice, we evaluate keyframe selection indirectly through downstream MLLM performance on multiple-choice QA benchmarks, which allow objective evaluation against ground-truth answers. In contrast, evaluating open-ended QA typically requires an LLM-as-a-judge method, which would introduce additional evaluation noise. Thus, to complement this indirect evaluation, we also report in Figure 5 key frame rate results on LVBench and Video-MME-KFS, which provide frame-level annotations and allow us to directly assess keyframe-selection quality. Furthermore, as described in Appendix A, in open-ended QA settings, frame relevance can be computed using only the question.
Regarding closed-source MLLMs, we acknowledge that evaluating models such as GPT-6 would provide additional evidence of transferability. However, due to API cost constraints, we focus on four open-source MLLMs, spanning different model architectures. Since PartS is a model-agnostic preprocessing step, independent of the downstream MLLM, it can be integrated with closed-source models without modifications. We believe our evaluation across four MLLMs already provides substantial evidence of its plug-and-play transferability.
Q5. Dependence on hand-crafted and dataset-specific hyperparameters.
All hyperparameters are fixed across the four benchmarks and four MLLMs evaluated in our experiments, as specified in Section 4.1 Implementation Details (lines 288-290). Although \beta=0.95 and M=8 are selected based on LVBench, these values are applied unchanged to LongVideoBench, Video-MME, and MLVU, without any dataset or model-specific tuning. Additionally, we provide hyperparameter sensitivity analyses in Section 4.4 Hyperparameter Sensitivity (lines 409-419). Therefore, we believe the consistent performance observed across benchmarks and MLLMs (Table 2) provides empirical evidence that PartS generalizes beyond the dataset used for hyperparameter selection.
Q6. Robustness to video encoding configurations.
Although I-frame placement depends on the video codec and GOP configuration, PartS does not assume a fixed GOP size or I-frame interval. Instead, it uses the available I-frames to initialize its candidate pool and adaptively samples additional frames from intermediate regions based on query relevance and visual diversity. This allows PartS to accommodate different I-frame distributions without requiring codec-specific configurations.
Regarding latency, the candidate-encoding time reported in Table 3 considers the time from opening the video to computing relevance scores. Then, yes, it includes video decoding, I-frame extraction, and random access to intermediate frames.
Response to Authors
I appreciate the clarification on candidate counts, rounding, and hyperparameter settings, along with the new ablation on frame allocation. I still see the method as an incremental contribution, so I’ll keep my original score.
We thank the reviewer for recognizing our extensive empirical evaluation and the efficiency improvements achieved by PartS. We address the concerns below.
W1. Limited methodological novelty.
Although PartS builds upon established techniques, we highlight that our main contribution lies in addressing the computational bottleneck of keyframe selection, as stated in the contributions “efficient adaptive candidate sampling strategy” and “efficient frame selection pipeline” (lines 83 and 86). To the best of our knowledge, PartS is the first method to leverage video codec structure to construct a compact candidate pool for efficient keyframe selection. It reduces the number of candidate frames that need to be scored (Figure 3) and the cost per frame by using the lightweight CLIP ViT-B/32 as frame encoder. As the reviewer acknowledges in S1, PartS achieves significant efficiency improvements. Particularly, we show in Figure 1 and Table 3 that PartS achieves 5.1-19.3x speedups over baselines while maintaining competitive accuracy across four benchmarks and four MLLMs (Table 2). We believe these results demonstrate that PartS contributes more than a combination of existing techniques, it introduces an efficient keyframe selection pipeline that substantially reduces computational overhead without sacrificing downstream performance.
PartS differs from QCA in both candidate-frame construction and frame budget allocation strategies. QCA relies on dense uniformly sampled frames (1 FPS), partitions videos into equally spaced temporal segments, and allocates frames based on query relevance and content deviation. In contrast, PartS constructs its candidate pool using codec information and performs clustering in the visual feature space, allocating frames across visually related clusters rather than predefined temporal segments. More importantly, QCA focuses on selecting informative frames under a fixed budget, whereas PartS targets the computational overhead of keyframe selection.
W2. Insufficient validation of the core design.
We clarify that Table A.2 explicitly compares adaptive and uniform candidate sampling under the same latency budget, as stated in its caption and in line 748, and under the same settings, ensuring a fair comparison of their performance at equivalent computational cost. Furthermore, all hyperparameters are specified in Section 4.1 (Implementation Details, lines 284-292), including the candidate count (K=32). We will further include these details in the appendix to make the comparison easier.
We address the concern about inconsistency between Eq. (3) and Algorithm 2 separately in Q3. Soft allocation implementation.
Q1. Contribution of I-frame initialization.
Regarding the concern that “Appendix Table A.2 does not provide sufficient details about the matched sampling and latency conditions”, Table A.2 precisely provides a controlled comparison between our adaptive sampling strategy and uniform sampling under the same latency budget. This is explicitly stated in the table caption: “We compare PartS using uniformly sampled candidates and our adaptive candidate sampling strategy under the same latency budget.”.
Specifically, to ensure a fair comparison, for each video, we uniformly sample exactly the same number of candidate frames produced by our adaptive sampling strategy. The resulting candidate pool is then passed through the same refinement stage (Step 2, Figure 2) to select K=32 keyframes. Thus, both strategies operate with identical candidate-frame counts and share the same downstream processing pipeline, isolating the effect of the candidate sampling strategy while maintaining comparable computational costs.