acceptodds
Under review as a conference paper at ICLR 2027

Partition and Select: Efficient Training-Free Keyframe Selection for Long-Video Understanding

Abstract

Long-video understanding is challenging because videos may contain tens of thousands of frames, making it impractical to process them entirely with multimodal large language models (MLLMs). Query-aware keyframe selection addresses this issue, but existing methods often require additional training or process large candidate sets with expensive encoders. We introduce **Part**ition and **S**elect (PartS), a training-free framework for efficient query-aware keyframe selection in long videos. Our method improves efficiency by reducing both the number of candidate frames that must be processed and the cost of encoding each candidate. Specifically, PartS exploits video-codec structure to construct a compact set of visual anchors and adaptively enriches it with frames from query-relevant segments. It then encodes candidates using a lightweight CLIP model, clusters them by visual similarity, and applies query-aware soft frame allocation to distribute a fixed frame budget across clusters according to their relevance. We evaluate PartS on four long-video QA benchmarks and four open-source MLLMs. It achieves - speedups over existing training-free approaches, maintaining competitive performance. These results demonstrate that PartS is an efficient plug-and-play module for long-video understanding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Query-Aware Keyframe Selection, Long Video Understanding, Efficient Video Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BScuOpenReview ↗
Rating
2
Confidence
3(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PartS, a training-free framework for query-aware keyframe selection in long-video understanding. It uses codec I-frames to initialize a compact candidate pool, adaptively refines query-relevant and visually diverse segments, and combines CLIP-based clustering with relevance-weighted frame allocation to select the final keyframes. Experiments on four benchmarks and four open-source MLLMs demonstrate competitive QA accuracy and 5.1–19.3× faster keyframe selection compared with existing training-free methods.

Critical strengths

S1. Significant efficiency improvements. PartS substantially reduces keyframe selection latency by limiting candidate-frame processing and using a lightweight CLIP encoder. The reported 5.1–19.3× speedups demonstrate practical value for long-video understanding.

S2. Extensive empirical evaluation. The method is evaluated across four long-video benchmarks, four MLLMs, and different frame budgets, showing consistent improvements over uniform sampling.

Critical weaknesses

W1. Limited methodological novelty. PartS primarily combines existing techniques, including codec-based I-frame extraction, heuristic adaptive sampling, CLIP similarity, K-Means clustering, and Softmax-based frame allocation. Notably, QCA: Query- and Content-Aware Keyframe Selection for Long Video Understanding (Peng et al., ECCV 2026) has already explored query-aware frame budget allocation across video segments. The proposed cluster-level allocation follows a similar principle. Overall, PartS appears to be largely an engineering combination of established components, with limited methodological novelty.

W2. Insufficient validation of the core design. Appendix Table A.2 compares adaptive and uniform candidate sampling but lacks sufficient details about candidate counts, initialization strategies, and matched latency conditions, making it difficult to establish the independent contribution of codec-based initialization. Moreover, inconsistencies between Eq. (3) and Algorithm 2 raise concerns about the implementation of soft frame allocation.

Questions for authors

Q1. Contribution of I-frame initialization. Could the authors compare I-frame initialization with an equal number of uniformly sampled initial frames, followed by exactly the same adaptive refinement procedure? Appendix Table A.2 does not provide sufficient details about the matched sampling and latency conditions, making the fairness of the comparison unclear.

Q2. Novelty of frame allocation. QCA has already proposed query-aware frame budget allocation based on semantic relevance and content diversity. What is the distinct contribution of PartS's cluster-level Softmax allocation beyond this existing approach? Could the authors provide a controlled ablation to demonstrate its independent effectiveness?

Q3. Soft allocation implementation. Eq. (3) uses floor rounding, whereas Algorithm 2 uses ceiling rounding. Under the default settings (K=32, c=100, τ=2), the former appears to allocate zero frames to every cluster, while the latter effectively allocates one frame per selected cluster. Could the authors clarify the actual implementation and explain the temperature sensitivity reported in Figure 5?

Q4. Generalization beyond multiple-choice QA. All evaluated benchmarks use multiple-choice questions, and Appendix A incorporates candidate answers into frame relevance scoring. How does PartS perform on open-ended video QA without answer options? Could the authors also evaluate closed-source MLLMs, such as GPT-6, to further validate its plug-and-play transferability?

Q5. Dependence on hand-crafted and dataset-specific hyperparameters. PartS relies on several manually designed thresholds and hyperparameters, whose choices lack sufficient justification and systematic ablation studies, making their individual contributions and interpretability unclear. Moreover, Appendix A determines β=0.95 and M=8 based on LVBench statistics and performance, raising concerns about dataset-specific tuning. Such dependence limits the robustness and practicality of the claimed plug-and-play framework in real-world scenarios. Could the authors provide more comprehensive ablations and demonstrate whether these hyperparameters generalize across datasets without additional tuning?

Q6. Robustness to video encoding configurations. Since I-frame locations depend on video encoding settings, how robust is PartS across different codecs and GOP configurations? Does the reported latency include video decoding, I-frame extraction, and random access to intermediate frames?

4 comments

We thank the reviewer for recognizing our extensive empirical evaluation and the efficiency improvements achieved by PartS. We address the concerns below.

W1. Limited methodological novelty.

Although PartS builds upon established techniques, we highlight that our main contribution lies in addressing the computational bottleneck of keyframe selection, as stated in the contributions “efficient adaptive candidate sampling strategy” and “efficient frame selection pipeline” (lines 83 and 86). To the best of our knowledge, PartS is the first method to leverage video codec structure to construct a compact candidate pool for efficient keyframe selection. It reduces the number of candidate frames that need to be scored (Figure 3) and the cost per frame by using the lightweight CLIP ViT-B/32 as frame encoder. As the reviewer acknowledges in S1, PartS achieves significant efficiency improvements. Particularly, we show in Figure 1 and Table 3 that PartS achieves 5.1-19.3x speedups over baselines while maintaining competitive accuracy across four benchmarks and four MLLMs (Table 2). We believe these results demonstrate that PartS contributes more than a combination of existing techniques, it introduces an efficient keyframe selection pipeline that substantially reduces computational overhead without sacrificing downstream performance.

PartS differs from QCA in both candidate-frame construction and frame budget allocation strategies. QCA relies on dense uniformly sampled frames (1 FPS), partitions videos into equally spaced temporal segments, and allocates frames based on query relevance and content deviation. In contrast, PartS constructs its candidate pool using codec information and performs clustering in the visual feature space, allocating frames across visually related clusters rather than predefined temporal segments. More importantly, QCA focuses on selecting informative frames under a fixed budget, whereas PartS targets the computational overhead of keyframe selection.

W2. Insufficient validation of the core design.

We clarify that Table A.2 explicitly compares adaptive and uniform candidate sampling under the same latency budget, as stated in its caption and in line 748, and under the same settings, ensuring a fair comparison of their performance at equivalent computational cost. Furthermore, all hyperparameters are specified in Section 4.1 (Implementation Details, lines 284-292), including the candidate count (K=32). We will further include these details in the appendix to make the comparison easier.
We address the concern about inconsistency between Eq. (3) and Algorithm 2 separately in Q3. Soft allocation implementation.

Q1. Contribution of I-frame initialization.

Regarding the concern that “Appendix Table A.2 does not provide sufficient details about the matched sampling and latency conditions”, Table A.2 precisely provides a controlled comparison between our adaptive sampling strategy and uniform sampling under the same latency budget. This is explicitly stated in the table caption: “We compare PartS using uniformly sampled candidates and our adaptive candidate sampling strategy under the same latency budget.”.

Specifically, to ensure a fair comparison, for each video, we uniformly sample exactly the same number of candidate frames produced by our adaptive sampling strategy. The resulting candidate pool is then passed through the same refinement stage (Step 2, Figure 2) to select K=32 keyframes. Thus, both strategies operate with identical candidate-frame counts and share the same downstream processing pipeline, isolating the effect of the candidate sampling strategy while maintaining comparable computational costs.

Q2. Novelty of frame allocation.

QCA does not employ clustering; instead, it divides videos into equally spaced temporal segments and allocates frames based on query relevance and content deviation. In contrast, PartS clusters candidate frames in the visual feature space and distributes the frame budget across visually similar clusters. We emphasize that soft allocation is not, by itself, a claimed contribution of our work. Rather, our novelty lies in the overall pipeline, including the adaptive candidate sampling, to substantially reduce computational cost, achieving up to 19.3x speedup over existing baselines while maintaining competitive accuracy.

To evaluate the independent effectiveness of budget allocation across clusters, we conducted an additional controlled ablation. Without allocation, we partition candidate frames into (K=32) clusters using visual features and select the frame with the highest relevance score within each cluster. With allocation, PartS distributes the same frame budget across clusters using soft frame allocation as detailed in Section 3.3, keeping all other components unchanged. The results presented in the table below show that soft allocation consistently improves accuracy across both MLLMs and benchmarks. These results validate the effectiveness of allocation within PartS. We will include these additional ablation results in the appendix of the revised paper.

MLLMAllocationLongVideoBenchVideo-MME
LLaVA-OVWithout59.160.1
With60.4 (+1.3)61.1 (+1.0)
Qwen2.5-VLWithout60.862.5
With63.9 (+3.1)63.3 (+0.8)

Q3. Soft allocation implementation.

We thank the reviewer for identifying this inconsistency. We clarify that the actual implementation uses ceiling rounding, as specified in Algorithm 2, and that the floor operator in Eq. (3) is a typo. We will correct the equation in the revised manuscript to ensure consistency between the mathematical formulation and the implementation.

Algorithm 2 enforces the total frame budget (K) during allocation. Then, in the extreme case where all clusters have identical relevance scores, ceiling rounding assigns one frame per cluster, which would result in 100 frames and exceed the budget K=32. In this case, the algorithm will assign 1 frame to the first 32 visited clusters. It is important to highlight that, in practice, multiple frames are selected from relevant clusters. For instance, on Video-MME, to select 32 keyframes, the PartS samples frames from an average of 15 clusters, allocating approximately 5 frames to the most relevant cluster on average.

Regarding temperature sensitivity, \tau controls how much the allocation favors highly relevant clusters. Lower temperatures produce a more concentrated relevance distribution, prioritizing relevant clusters, whereas higher temperatures encourage a more uniform distribution, promoting visual diversity.

Q4. Generalization beyond multiple-choice QA.

Following common practice, we evaluate keyframe selection indirectly through downstream MLLM performance on multiple-choice QA benchmarks, which allow objective evaluation against ground-truth answers. In contrast, evaluating open-ended QA typically requires an LLM-as-a-judge method, which would introduce additional evaluation noise. Thus, to complement this indirect evaluation, we also report in Figure 5 key frame rate results on LVBench and Video-MME-KFS, which provide frame-level annotations and allow us to directly assess keyframe-selection quality. Furthermore, as described in Appendix A, in open-ended QA settings, frame relevance can be computed using only the question.

Regarding closed-source MLLMs, we acknowledge that evaluating models such as GPT-6 would provide additional evidence of transferability. However, due to API cost constraints, we focus on four open-source MLLMs, spanning different model architectures. Since PartS is a model-agnostic preprocessing step, independent of the downstream MLLM, it can be integrated with closed-source models without modifications. We believe our evaluation across four MLLMs already provides substantial evidence of its plug-and-play transferability.

Q5. Dependence on hand-crafted and dataset-specific hyperparameters.

All hyperparameters are fixed across the four benchmarks and four MLLMs evaluated in our experiments, as specified in Section 4.1 Implementation Details (lines 288-290). Although \beta=0.95 and M=8 are selected based on LVBench, these values are applied unchanged to LongVideoBench, Video-MME, and MLVU, without any dataset or model-specific tuning. Additionally, we provide hyperparameter sensitivity analyses in Section 4.4 Hyperparameter Sensitivity (lines 409-419). Therefore, we believe the consistent performance observed across benchmarks and MLLMs (Table 2) provides empirical evidence that PartS generalizes beyond the dataset used for hyperparameter selection.

Q6. Robustness to video encoding configurations.

Although I-frame placement depends on the video codec and GOP configuration, PartS does not assume a fixed GOP size or I-frame interval. Instead, it uses the available I-frames to initialize its candidate pool and adaptively samples additional frames from intermediate regions based on query relevance and visual diversity. This allows PartS to accommodate different I-frame distributions without requiring codec-specific configurations.

Regarding latency, the candidate-encoding time reported in Table 3 considers the time from opening the video to computing relevance scores. Then, yes, it includes video decoding, I-frame extraction, and random access to intermediate frames.

Reviewer_BScuOpenReview ↗

Response to Authors

I appreciate the clarification on candidate counts, rounding, and hyperparameter settings, along with the new ablation on frame allocation. I still see the method as an incremental contribution, so I’ll keep my original score.

Reviewer_cxR8OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Partition and Select (PartS), a training-free framework for efficient query-aware long video keyframe selection. The paper uses a set of video anchors and improves it by adding frames from each query-relevant segment. The method efficiently improves end-to-end latency, achieving a ~5.1-19.2x speedup compared to uniform sampling and other recent baselines on four long-video datasets, including LongVideoBench, Video-MME, MLVU, and LVBench.

Even though the improvement is significant, the algorithmic novelty is limited: it is just another training-free key frame selection method and all the existing components, i.e video anchor, partitioning-based query-relevant segmentation and visual clustering, are well established; the paper only connected them efficiently in this work. There are some benchmarks needed to evaluate the core contribution of the paper, which I mentioned in the "Questions For Authors" section. Once we get responses in the rebuttal, I will be happy to update my score.

Critical strengths

A simple training keyframe selection method that uses adaptive candidate sampling and frame clustering based on query relevance to assign the best frames according to the selection budget. PartS improves efficiency by reducing both the number of candidate frames that must be processed and the cost of encoding each candidate. The reported 5.1 - 19.3x speedup over existing baselines is quite significant at the same frame/latency budget.

Critical weaknesses

The paper uses existing methods and connects them well in this work. According to the paper, PartS improves the end-to-end latency by 5.1- 19.3x over the current baseline without improving performance. The main concern is that, as mentioned in Table 3, it seems the speedup mostly comes from the frame scoring model selection, i.e the paper uses a smaller encoder, CLIP-B/32 150m, as compared to FOCUS[1] BLIP-ITM Large 370m model or A.I.R[2] EVA-CLIP-L + Qwen2.5-VL 7b. FOCUS[1] with larger encoder candidate encoding latency is ~80.9s as compared to PartS 10.4s. If it's really because of the encoder selection, can the authors report performance of some of the current baseline approaches with a smaller encoder, i.e FOCUS[1] with CLIP-B/32 150m if they have reproducible code? This will ensure the robustness of the proposed method and position it as a recent SOTA model.

[1] Zirui Zhu et al. FOCUS: Efficient keyframe selection for long video understanding. In International Conference on Learning Representations, 2026. URL: https://arxiv.org/pdf/2510.27280

[2] Yuanhao Zou et al. A.I.R: Adaptive, Iterative, and Reasoning-based Frame Selection For Video Question Answering, URL: https://arxiv.org/html/2510.04428v1

Questions for authors

As mentioned in the critical weakness section, request the author to provide a more detailed assessment of the speedup breakdown for each component and, if possible, evaluate some of the recent baseline models, i.e FOCUS[1], A.I.R[2], QCA[3], WFS-SB[4], with the same smaller Frame-Scoring model CLIP-B/32 (150M) as used in your paper.

[1] Zirui Zhu et al. FOCUS: Efficient keyframe selection for long video understanding. In International Conference on Learning Representations, 2026. URL: https://arxiv.org/pdf/2510.27280

[2] Yuanhao Zou et al. A.I.R: Adaptive, Iterative, and Reasoning-based Frame Selection For Video Question Answering, URL: https://arxiv.org/html/2510.04428v1

[3] Jun Peng et al. QCA: Query- and Content-Aware Keyframe Selection for Long Video Understanding. URL: https://arxiv.org/abs/2607.00983

[4] Wang Chen et al. Wavelet-based Frame Selection by Detecting Semantic Boundary for Long Video Understanding. URL: https://arxiv.org/abs/2603.00512

3 comments

We thank the reviewer for recognizing the efficiency improvements achieved by PartS.

"Can the authors report performance of some of the current baseline approaches with a smaller encoder, i.e FOCUS[1] with CLIP-B/32 150m? This will ensure the robustness of the proposed method and position it as a recent SOTA model."

To address this concern, we conducted additional experiments using CLIP ViT-B/32 for FOCUS, QCA, and PartS (ours). We evaluate all three methods on Video-MME and LVBench across four MLLMs.

The results presented in Table R1 demonstrate that PartS consistently outperforms both FOCUS and QCA across all four MLLMs and both benchmarks. On Video-MME, PartS achieves accuracy gains of 2.4-4.6 points over FOCUS and 1.3-2.8 over QCA. Similarly, on LVBench, PartS improves accuracy by 0.7-2.7 over FOCUS and 7.8-8.3 over QCA. Notably, replacing the original scoring models of the baselines with CLIP ViT-B/32 consistently degrades their performance and PartS achieves better accuracy.

"it seems the speedup mostly comes from the frame scoring model selection"

To isolate the contribution of our adaptive sampling approach, we also measured the keyframe selection latency on LVBench using the same encoder across all methods (Table R2). Under this controlled setting, PartS remains 1.5× faster than FOCUS and 3.4× faster than QCA, demonstrating that its efficiency gains do not come only from the scoring model, although it contributes significantly. Furthermore, to complement this analysis, Figure 3 shows that the number of candidate frames processed by PartS grows more slowly with video duration than both uniform 1-FPS sampling and FOCUS. For instance, for an 8,000-second video, uniform sampling processes 8,000 frames, FOCUS approximately 3,496, and PartS only 1,480, respectively.

This way, we show PartS achieves both higher QA accuracy and lower selection latency under the same encoder configuration, further supporting the effectiveness and robustness of our proposed pipeline. We will include these additional results in the revised manuscript to provide a more comprehensive and controlled comparison with existing methods.

Table R1. Accuracy (%) using CLIP ViT-B/32 for keyframe selection.

MLLMMethodVideo-MMELVBench
LlaVA-OV-7BFOCUS57.1 (-4.0)44.9 (-1.0)
QCA58.3 (-2.8)38.1 (-7.8)
PartS (Ours)61.145.9
Qwen2.5-VL-7BFOCUS60.9 (-2.4)46.7 (-0.7)
QCA61.2 (-2.1)39.1 (-8.3)
PartS (Ours)63.347.4
LLaVA-Video-7BFOCUS60.1 (-4.0)47.3 (-1.4)
QCA62.8 (-1.3)40.7 (-8.0)
PartS (Ours)64.148.7
Qwen3-VL-8BFOCUS61.4 (-4.6)47.8 (-2.7)
QCA64.5 (-1.5)42.2 (-8.3)
PartS (Ours)66.050.5

Table R2. Keyframe selection latency breakdown using CLIP ViT-B/32.

MethodCandidate Encoding (s)Selection (s)Latency (s)Speedup
FOCUS16.10.216.31.5
QCA37.20.437.63.4
PartS (Ours)10.40.611.0-

Dear authors, thank you for your responses in the rebuttal. Just wanted to check a few more points: 1) Does Table R1 in the rebuttal use the same frame budget across all the methods? 2) On the latency breakdown in Table R2, why does FOCUS/PartS have different latency for candidate encoding even after using the same encoder? Did they use the same number of frames?

We thank the reviewer for engaging with our rebuttal and for the follow-up questions.

1) Does Table R1 in the rebuttal use the same frame budget across all methods?

Yes. We evaluate all methods using the same final budget of K=32 keyframes passed to the downstream MLLM and the same frame-scoring encoder (CLIP ViT-B/32), ensuring a fair comparison under identical settings.

2) On the latency breakdown in Table R2, why does FOCUS/PartS have different latency for candidate encoding even after using the same encoder? Did they use the same number of frames?

No. Although all methods use CLIP ViT-B/32 and select the same final budget of K=32 keyframes, we maintain each method's original candidate sampling strategy, resulting in different numbers of candidate frames being processed. Specifically, on LVBench, PartS, FOCUS, and QCA (which uses 1-FPS uniform sampling) process an average of 1,136, 1,749 (1.54$\times$), and 4,046 (3.56$\times$) candidate frames per video, respectively, which explains the candidate encoding latencies in Table R2 (10.4s, 16.1s, and 37.2s).

This controlled setting isolates the contribution of our adaptive candidate sampling: even encoding about 35% fewer candidate frames than FOCUS (and 72% fewer than QCA), PartS still achieves consistently higher QA accuracy across all four MLLMs (Table R1) when CLIP ViT-B/32 is used.

Reviewer_hG6mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces an improved keyframe selection method to address the problem of long video understanding. Instead of densely encoding frames and compute the frame-query similarity scores, the proposed method PartS leverages the I-frames in video-codec to reduce the candidate pool size, and therefore reducing the computational overhead. The method then clusters the candidate frames and applies soft frame allocation to select the final keyframes that go into the MLLM. Consequently, the authors show that PartS achieves a significant reduction in FLOPs while maintaining or even surpassing the accuracy of other keyframe selection baselines, as the main experiments are conducted on four benchmarks and four MLLMs.

Critical strengths
  1. The paper tackles a crucial bottleneck of long video understanding methods that hinders their real-world application: the high computational overhead of keyframe retrieval. From the results shown in table 3 and figure 4, the proposed method significantly reduces the computational cost and inference latency.
Critical weaknesses
  1. Given that the main focus of this paper, as I understand it, is to reduce the computational cost rather than to improve accuracy (as evidenced by the marginal accuracy gains in Table 2), the authors should report comparisons of computational cost and latency across all four benchmarks and MLLMs. However, the paper reports efficiency-related results only on LVBench.
  2. Leveraging I-frame information for long video understanding is not new to the field, yet the authors do not provide an overview or discussion of this line of work in the Related Work section.
Questions for authors
  1. In Section 4.2 (‘Comparison with the State of the Art’), PartS generally yields slight accuracy improvements even over keyframe retrieval methods that rely on a densely sampled candidate pool, such as AKS, and the authors conclude that ‘these results demonstrate that processing a densely sampled candidate pool is not necessary to achieve strong long video QA performance.’ However, it is not self-evident why PartS achieves better accuracy using significantly less compute. Could the authors provide more insight into this result and the conclusion?
1 comment

We thank the reviewer for recognizing the efficiency improvements achieved by PartS. We address the concerns below.

W1: the authors should report comparisons of computational cost and latency across all four benchmarks and MLLMs.

We highlight that our keyframe selection method is intended to be employed before passing frames to the MLLM, as illustrated in Figure 2. Thus, its computational cost is not affected by the choice of MLLM.

Moreover, as our objective is to reduce the cost of keyframe selection, LVBench is the most suitable benchmark because it contains the longest videos among the four benchmarks considered. LVBench includes videos ranging from 30 minutes to several hours, with an average of 68.4 minutes, whereas the average durations in LongVideoBench, Video-MME, and MLVU are 7.9, 17.0, and 15.5 minutes, respectively. Then, LVBench provides the most challenging setting for evaluating the efficiency, where the cost of processing densely sampled candidate frames becomes significant.

W2: the authors do not provide an overview or discussion of I-Frames in the Related Work section.

To the best of our knowledge, PartS is the first method to explicitly leverage I-frames for keyframe selection. Our contribution “efficient adaptive candidate sampling strategy” lies not only in the use of I-frames themselves, but in using them as an initialization to our adaptive candidate sampling algorithm that constructs a compact candidate pool and avoids the computational overhead of dense frame sampling.

Nevertheless, we acknowledge that I-frames have long been exploited in video processing, and we already cite Overview of the H.264/AVC Video Coding Standard (Wiegand et al., 2003). We will revise the Related Work section to include more recent approaches such as ReMoRa (CVPR 2026) [1] and CoPE-VideoLM (NeurIPS 2026) [2], making clear that PartS differs from these approaches by focusing on efficient keyframe selection as a model-agnostic preprocessing step, rather than training MLLMs on compressed-video information.

[1] Yashima et al. ReMoRa: Multimodal Large Language Model based on Refined Motion Representation for Long-Video Understanding. In CVPR 2026. URL https://openaccess.thecvf.com/content/CVPR2026/html/Yashima_ReMoRa_Multimodal_Large_Language_Model_based_on_Refined_Motion_Representation_CVPR_2026_paper.html

[2] Sarkar et al. CoPE-VideoLM: Leveraging Codec Primitives for Efficient Video Language Modeling. In NeurIPS 2026. URL https://neurips.cc/virtual/2026/loc/paris/poster/152600

Question: it is not self-evident why PartS achieves better accuracy using significantly less compute. Could the authors provide more insight into this result and the conclusion?

Our statement that “these results demonstrate that processing a densely sampled candidate pool is not necessary to achieve strong long video QA performance” reflects the main contribution of PartS, which is to reduce the computational cost of keyframe selection, achieving a speedup of up to 19.3x, while maintaining competitive performance. We show that processing more candidate frames does not necessarily lead to a more informative final keyframe set. Long videos typically contain much visual redundancy, which can introduce noise into frame relevance scores and lead to inefficient keyframe allocation. For instance, a one-hour video sampled at 1 FPS produces approximately 3,600 candidate frames, potentially including tens of near-identical frames (cosine similarity > 0.95) with high relevance scores. Such redundancy can cause dense sampling-based methods to waste their budget on redundant frames or overemphasize spurious similarity peaks. PartS mitigates these issues by pruning redundant intervals ($\delta \leq \beta$) during adaptive candidate sampling and clustering visually similar frames before allocating the keyframe budget. This encourages a more diverse and informative selection without requiring dense candidate processing.

We discuss in Section 4.3 Efficiency Analysis that the efficiency gains arise from two factors: (i) processing fewer candidate frames; (ii) reducing the computational cost of encoding each candidate frame, we use a lightweight CLIP ViT-B/32 encoder, whereas competing methods rely on larger models for frame scoring. Figure 3 shows that the number of candidate frames processed by PartS grows consistently more slowly with video duration than the others. Also, we show that PartS matches existing methods across different MLLMs and benchmarks (Table 2) and achieves 5.1-19.3x speedups (Figure 1 and Table 3). These results support our conclusion that densely sampling candidate frames is not essential for achieving strong long-video QA performance.

Comments on the paper

1 comment
Area_Chair_MikaOpenReview ↗

Official Comment by Area Chair Mika

Dear reviewers,

Thank you very much for your time and effort in reviewing this submission. The authors have now submitted their rebuttal, and the discussion period is nearing its end. We would greatly appreciate it if you could take some time to carefully review the authors’ responses and share your assessment of whether your concerns have been adequately addressed. Thank you again for your valuable time and contributions to the review process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.