acceptodds
Under review as a conference paper at ICLR 2027

A Dynamic Temporal Segmentation for Long Video Understanding

Abstract

Multimodal Large Language Models (MLLMs) have achieved remarkable success in video understanding, yet scaling them to long videos remains a major challenge due to severe temporal redundancy and limited frame budgets. Some existing methods rely on uniform temporal segmentation, implicitly assuming that semantic content varies uniformly over time, which misallocates frame budgets and misses critical moments. In this paper, we propose DyQCA, a Dynamic Segmentation framework for keyframe selection in long video understanding. DyQCA detects temporal boundaries by comparing the semantic differences of adjacent frames, partitions the video into dynamic segments, and allocates the frame budget across segments by jointly modeling query relevance and content deviation, followed by anchor-centric keyframe selection within each segment. The proposed DyQCA is training-free, model-agnostic, and can be seamlessly plugged into existing Video-LLMs. Extensive experiments demonstrate that DyQCA consistently outperforms uniform sampling and recent keyframe selection baselines, e.g., 67.3% on LongVideoBench and 76.2% on MLVU under a 64-frame budget, demonstrating that dynamic semantic segmentation offers a simple yet effective path toward more accurate long video understanding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Dynamic Video Segmentation, Long Video Understanding, Multimodal Large Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EkBqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents DyQCA, a training-free, model-agnostic keyframe selection framework for long-video understanding that replaces uniform temporal segmentation with dynamic, content-driven segmentation. It detects boundaries via adjacent-frame semantic difference combined with change-acceleration and change-rate signals, then allocates the frame budget across segments by query relevance and content deviation and performs anchor-centric selection within each segment. On four long-video benchmarks with three Video-LLM backbones under a 64-frame budget, it improves over uniform sampling across the board and surpasses much larger models. However, its improvement over the strongest prior baseline is small and inconsistent, and only the segmentation stage is new relative to prior work.

Critical strengths

S1. A simple, practical, and genuinely plug-and-play framework with clear gains over the ubiquitous uniform-sampling default. DyQCA is training-free, model-agnostic, and requires no architectural change; it improves over uniform sampling consistently across all four benchmarks and all three backbones, and with a small frame budget it surpasses much larger proprietary and open models on several benchmarks. For the widely used "select-then-answer" pipeline, this is an immediately usable improvement, which is the main basis for a positive view of the paper.

S2. The motivation is well articulated and the breakdown analyses support the claimed mechanism. The paper explains why uniform segmentation misallocates frame budgets (splitting events, merging unrelated content), and the evidence aligns with this story: the largest gains appear on the long-video subsets and on temporal-ordering tasks, i.e., exactly where semantic non-uniformity should hurt uniform segmentation most. The authors also report where the method does not help, which adds credibility.

Critical weaknesses

W1. The contribution is incremental over prior work. The budget-allocation stage (query relevance + content deviation) and the anchor-centric keyframe selection stage are taken directly from the prior QCA framework; the sole new component is threshold-based dynamic segmentation using simple difference/acceleration/rate signals. The idea that segment boundaries should align with semantic transitions is intuitively appealing, but as realized here it is a straightforward, heuristic substitution within an existing pipeline rather than a fundamentally new mechanism, which limits the paper's novelty.

W2. The new component does not deliver a consistent or clearly evidenced advantage over the prior pipeline it modifies. Against QCA, DyQCA is worse on most benchmarks with LLaVA-Video-7B, trails on LVBench with Qwen3-VL-8B, and wins by only small margins where it does win; the paper's own ablation further shows that the proposed boundary criterion is not the best choice on every dataset. With differences of this size, the absence of repeated-run statistics or confidence intervals makes it difficult to separate a real gain from noise, and the abstract's claim of "consistently outperforming recent keyframe selection baselines" is not supported by its own main table.

Questions for authors
  1. Comparison with QCA. On LLaVA-Video-7B, DyQCA is below QCA on most benchmarks, and also trails it on LVBench with Qwen3-VL-8B (Table 1). Under what conditions does dynamic segmentation hurt, and should the headline claim be qualified accordingly? What distinguishes the benchmarks where it helps from those where it does not?
  2. Hyperparameter protocol. How are the boundary thresholds and the segment count chosen for each benchmark? Do all reported results use a single frozen setting across datasets? Please report sensitivity (e.g., performance spread across reasonable threshold ranges) so that the gains can be separated from per-dataset tuning.
  3. Statistical reliability. Are the small differences over QCA stable across seeds/runs? Please provide confidence intervals or repeated-run statistics for the main comparisons.
  4. Efficiency. What is the end-to-end selection latency (including 1-FPS feature extraction and segmentation) per video, compared with uniform sampling and QCA? This matters for the claimed practical deployment value.
  5. Claim-evidence alignment. The abstract promises consistent improvement over recent selection baselines, while Table 1 shows mixed outcomes against QCA. Could you align the claims with the table?
  6. Reproducibility. No code or data release is mentioned. Given that all components are training-free, do the authors plan to release the segmentation/allocation code?
  7. Presentation. The budget-allocation and keyframe-selection sections largely restate prior work. Could you mark more explicitly which parts are new, to help readers locate the contribution?
1 comment

Thank you for your detailed reviews and insightful questions. Please see our point-by-point responses.

[W1, Q7] As described in our paper, the budget allocation and the anchor-centric keyframe selection are taken from QCA (Peng et al., 2026). The new component is the training-free, threshold-based dynamic segmentation using DtD_t, AtA_t, and RtR_t, together with its interface to the existing pipeline. Our contribution is a simple, training-free segmentation substitution plus an empirical characterization of when it helps.

[W2, Q1, Q5] We will revise the abstract’s ``consistently outperforms’’ to state that DyQCA is competitive across all benchmarks and backbones, yields clear gains on MLVU, and is comparable or slightly lower elsewhere. As demonstrated in Fig.3, gains concentrate on queries whose answer depends on a short, localized event, e.g., a flag, a biscuit, an egg, where dynamic boundaries let the budget concentrate on the segment containing that evidence. Declines concentrate where evidence is distributed over long ranges or events are already close to uniformly spaced, so uniform segmentation provides a reasonable temporal prior and occasional over-segmentation of stable intervals offsets the benefit.

[Q2] SS is fixed in all experiments and set identically to QCA’s configuration. The threshold rule in Eq.6 does not fix the number of segments by itself. For each video, we compute the three signals DtD_t, AtA_t and RtR_t for all frames and sort each signal independently. The three thresholds are determined as follows:

λ3\lambda_3 is set to the 85th percentile of RtR_t for that video, obtained via np.percentilenp.percentile. λ1\lambda_1 and λ2\lambda_2 are determined by a joint grid search. We sweep both thresholds over the percentile range from 50% to 98% in steps of 2, i.e., over np.arange(50,100,2)np.arange(50, 100, 2) or each signal, and evaluate every (λ1,λ2)(\lambda_1,\lambda_2) combination. For each combination, frames satisfying Eq. (6) and yielding exactly the target S−1S−1 boundaries are taken as candidate boundaries, and a boundary score is computed. The combination with the highest score is selected as the final (λ1,λ2)(\lambda_1,\lambda_2) for that video. γ\gamma and η\eta are fixed at 0.5, inherited unchanged from QCA.

We will state this protocol explicitly in revision.

[Q3] With a fixed random seed and unchanged frame features, the segmentation result for a given video is identical across runs. Consequently, seed-based repeated runs would report zero variance. The relevant source of variability is instead the evaluation set, not the method.

[Q4] On Video-MME, we measured per-stage throughput: BLIP feature extraction 29 FPS, dynamic segmentation 4056 FPS, QCA allocation/keyframe selection 11430 FPS, and MLLM inference 5 FPS. Dynamic segmentation is about two orders of magnitude cheaper than feature extraction and three orders cheaper than MLLM inference, so its added cost over uniform segmentation is negligible relative to the end-to-end pipeline.

[Q6] All components are training-free, and we will release the code for dynamic segmentation, budget allocation, and keyframe selection, together with the SS setting and hyperparameter search procedure described above.

Reviewer_Zr32OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a new method to filter which video frames to give a model in order to reduce compute cost. Their method, named DyQCA, leverages semantic filters as well as change in acceleration to choose frames given a specific frame budget. The model leveraged by the authors to extract the semantic features is BLIP-2. The authors show some examples of dynamic temporal segmentation where we can see that frames are selected around specific semantic meanings such as "building view", "dining", etc. They argue that such semantic temporal segmentation is crucial to reduce information and improve the performance of video models.

On the experimental side, they applied their method to LLaVA-Video, InternVL and Qwen3-VL and used the following benchmarks: Video-MME, LongVideoBench, MLVU and LVBench.

Critical strengths
  • The method is evaluated on three backbones (LLaVA-Video, InternVL and Qwen3-VL) and four long-video benchmarks.
Critical weaknesses
  • This work is mostly based on QCA, which is a method that chooses the most relevant frames given a budget and a target query. The authors show in Table 1 results on DyQCA that are close to the original QCA work. Overall, there does not seem to be a statistically significant difference between DyQCA and QCA on the results that are presented in the paper. It's unfortunate that the authors did not include some analysis of the variance of the method to better characterize the effect of the noise.

  • In addition to not providing experimental evidence from an accuracy point of view, they don't provide a performance analysis of which method is faster/more compute efficient.

  • In addition to not presenting results that are better than the previous approaches, there are a lot of missing related-work comparisons. There have been hundreds of papers tackling this exact approach, and the number of comparison points in Table 1 seems limited.

  • One of the contributions that the authors claim is the identification that uniform temporal sampling is limited, and they propose a dynamic segmentation method, which is clearly not new.

Questions for authors
  • Qwen3-VL-30B-A3B was said to be tested, but I do not see any results in the table. Am I missing something?
  • I am also not sure why only a 64-frame budget was tested. A plot of the method's performance over different budgets could probably help better quantify the noise.
1 comment

Thank you for your detailed reviews and insightful questions. Please see our point-by-point responses.

[W1, Q1, Q2] We will revise the abstract’s ``consistently outperforms’’ to state that DyQCA is competitive across all benchmarks and backbones, yields clear gains on MLVU, and is comparable or slightly lower elsewhere.

The sentence in Section 4.1 stating that we report results with Qwen3-VL-30B-A3B-Instruct is an error; these experiments were not run. We will delete the sentence and keep only the three completed backbones in the text and tables.

We agree that a single 64-frame budget cannot characterize the method's behavior or its noise. We add a budget sweep with Qwen3-VL-8B over N′∈{16,32,64,128}N'\in\{16,32,64,128\} on MLVU and report the results below.

MLVU
MethodN'=16N'=32N'=64N'=128
QCA71.3073.3075.2076.80
DyQCA69.5573.4176.2278.01

The advantage of dynamic segmentation grows with the budget. DyQCA is slightly below QCA at 16 frames, essentially tied at 32, and higher at 64 and 128. This is consistent with our motivation, i.e., under a very tight budget, dynamic segmentation can over-fragment stable intervals, whereas with a larger budget content-driven segments can be covered more fully and the benefit of concentrating frames on query-relevant segments becomes visible. We will add this table and discussion to the revision.

[W2] On Video-MME, we measured per-stage throughput: BLIP feature extraction 29 FPS, dynamic segmentation 4056 FPS, QCA allocation/keyframe selection 11430 FPS, and MLLM inference 5 FPS. Dynamic segmentation is about two orders of magnitude cheaper than feature extraction and three orders cheaper than MLLM inference, so its added cost over uniform segmentation is negligible relative to the end-to-end pipeline.

[W3] Table 1 currently compares against uniform sampling, Top-k, AKS, Q-Frame, OneClip-RAG, and QCA, which cover the three families we discuss: adaptive keyframe selection, segment-level selection, and token pruning. We strengthen this by adding comparisons with recent segment-level and boundary-based methods, specifically WFS-SB (Chen et al., 2026), BOLT (Liu et al., 2025a).

LLaVA-Video-7BLongVideoBenchVideo-MMEMLVU
+ Uniform58.964.470.8
+ WFS-SB (8 Frame)61.161.766.9
+ BOLT62.264.670.3
+ DyQCA62.765.674.7

[W4] We agree that the idea of aligning segment boundaries with semantic transitions is not new. Our contribution is a simple, training-free segmentation substitution plus an empirical characterization of when it helps.

Reviewer_c6yjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes DyQCA, a training-free keyframe selection method that replaces QCA’s uniform temporal segmentation with boundaries detected from adjacent-frame feature changes. It retains QCA’s query/content-aware budget allocation and anchor-centric selection, and evaluates the resulting framework across four benchmarks and three video-language backbones.

Critical strengths
  1. The modification is focused and easy to integrate. It reuses the features and downstream selection machinery of QCA without requiring backbone fine-tuning or an additional boundary detection model.
  2. The results provide meaningful positive evidence on selected temporal tasks. With Qwen3-VL-8B, DyQCA improves over QCA from 58.22 to 60.00 on Video-MME’s long subset and from 59.46 to 66.80 on MLVU Action Order. The boundary-criterion ablation also supports the usefulness of changing segment boundaries while retaining the selection pipeline.
Critical weaknesses
  1. When is dynamic segmentation preferable to QCA’s uniform segmentation? At the precision reported in Table 1, DyQCA improves in four of twelve backbone–benchmark comparisons, declines in seven, and ties in one. All three backbones decline on LVBench. The long-video and action-order gains are encouraging, but the experiments do not establish a consistent relationship between semantic non-uniformity and the benefit of dynamic segmentation. The explanation based on over-segmentation is not supported by a corresponding analysis.
  2. How does the boundary rule produce the prescribed segment count? Equation 6 yields a content-dependent number of threshold crossings, whereas Equations 7–8 assume exactly (S-1) boundaries and describe (S) as user-specified. Table 5 additionally holds the segment count fixed. The procedure connecting threshold detections to this constraint is not specified, although it determines the segmentation and subsequent budget allocation actually evaluated.
Questions for authors
  1. What determines the final boundaries when threshold detection produces more or fewer than (S-1) candidates? How are budgets exceeding the available frames in short segments handled?
  2. Which video or query characteristics explain the improvements over QCA and the consistent declines on LVBench? Is the proposed over-segmentation explanation supported by the observed segmentations?
  3. What evidence connects the detected feature changes to semantic event boundaries? Are the gains explained by improved event coherence or by a useful redistribution of frame budgets?
  4. What benefit does the query-independent feature-change signal offer over WFS-SB’s query-conditioned boundary signal?
  5. Are the boundary thresholds shared across benchmarks or selected separately, and which data split is used for selection?
2 comments

Thank you for your detailed reviews and insightful questions. Please see our point-by-point responses:

[Q1] SS is fixed in all experiments and set identically to QCA’s configuration. The threshold rule in Eq.6 does not fix the number of segments by itself. For each video, we compute the three signals DtD_t, AtA_t and RtR_t for all frames and sort each signal independently. The three thresholds are determined as follows:

λ3\lambda_3 is set to the 85th percentile of RtR_t for that video, obtained via np.percentilenp.percentile. λ1\lambda_1 and λ2\lambda_2 are determined by a joint grid search. We sweep both thresholds over the percentile range from 50% to 98% in steps of 2, i.e., over np.arange(50,100,2)np.arange(50, 100, 2) or each signal, and evaluate every (λ1,λ2)(\lambda_1,\lambda_2) combination. For each combination, frames satisfying Eq. (6) and yielding exactly the target S−1S−1 boundaries are taken as candidate boundaries, and a boundary score is computed. The combination with the highest score is selected as the final (λ1,λ2)(\lambda_1,\lambda_2) for that video. γ\gamma and η\eta are fixed at 0.5, inherited unchanged from QCA.

For budget overflow in short segments: Following QCA, after the proportional allocation in Eq.10, we clip each segment's budget to its available frames, $ q_s = min(q_s, |\mathcal{X}_s|),andredistributethesurplustothehighest−, and redistribute the surplus to the highest-c_s$ segments that still have unselected frames, until N’N’ is exhausted or all segments are fully covered. This will be added to the revision.

[Q2] We will revise the abstract’s ``consistently outperforms’’ to state that DyQCA is competitive across all benchmarks and backbones, yields clear gains on MLVU, and is comparable or slightly lower elsewhere. As demonstrated in Fig.3, gains concentrate on queries whose answer depends on a short, localized event, e.g., a flag, a biscuit, an egg, where dynamic boundaries let the budget concentrate on the segment containing that evidence. Declines concentrate where evidence is distributed over long ranges or events are already close to uniformly spaced, so uniform segmentation provides a reasonable temporal prior and occasional over-segmentation of stable intervals offsets the benefit.

[Q3] DtD_t is sensitive to visual change including gradual within-event motion, while AtA_t and RtR_t respond only to abrupt change, so their conjunction in Eq.6 suppresses spurious boundaries from gradual variation. Tab.5 support the current criterion, i.e., D_t and (A_t or R_t), is best on MLVU and Video-MME, while D_t alone is best on LongVideoBench. We acknowledge the paper does not directly validate detected boundaries against annotated event boundaries.

[Q4] DyQCA's boundary signal is intentionally query-independent. It operates on the same features used for keyframe selection, needs no additional modules or supervision, and keeps the segment structure stable across different queries on the same video, so segmentation can be computed once and reused; query information enters only at allocation and anchor selection. A query-conditioned signal may localize a single queried event more tightly, but it recomputes segments per query and can be unstable when the query is ambiguous. We will cite and add the WFS-SB’s results in the revision.

[Q5] The thresholds are neither shared across benchmarks nor tuned per benchmark on a held-out split. As described in response to Q1, λ1\lambda_1 and λ2\lambda_2 are determined per video by an online search over that video's own distribution. γ\gamma and η\eta are fixed at 0.5, inherited unchanged from QCA. We will state this protocol explicitly in revision.

Thank you for the response and for running the additional experiments! Overall, my concerns have been partially addressed and I will keep my rate.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.