acceptodds
Under review as a conference paper at ICLR 2027

DynVid: Adaptive Dynamism-Aware Token Compression for Efficient Video LLMs

Abstract

Video Large Language Models (VLLMs) have achieved remarkable success in video understanding, yet suffer from prohibitive computational cost due to the massive number of visual tokens from densely sampled frames. While visual token compression offers a promising solution, existing methods are limited in two respects: they either allocate a uniform token budget across all frames, neglecting the inherent variation in scene complexity, or primarily rely on attention scores for token selection, overlooking temporal dynamics unique to video. To address these limitations, we propose DynVid, a training-free, dynamism-aware adaptive token compression framework for VLLMs that follows an allocate-then-select paradigm. In allocation stage, video frames are partitioned into semantically coherent segments via dynamic segmentation, and each segment receives an adaptive token budget proportional to its visual complexity. In selection stage, we introduce a token-level dynamism score as a temporally-aware importance signal that complements attention scores, and employ a Facility Location objective to select anchor tokens that are both individually important and collectively representative. Extensive experiments on eight video understanding benchmarks demonstrate that DynVid outperforms training-free baselines in both accuracy and inference efficiency. Notably, on LLaVA-OneVision, DynVid retains 99.3% relative accuracy with 10% average retention ratio of visual tokens preserved, and still maintains 96.4% relative accuracy at an aggressive 5% retention ratio. On Qwen2.5-VL, DynVid achieves 97.4% and 96.0% relative accuracy at 10% and 7.5% retention ratios, respectively, while delivering up to 7.53 and 9.49 prefilling speedups.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Large Language Models (VLLMs), training-free, dynamism-aware adaptive token compression framework

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_LhhMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies training-free visual token compression for efficient VLLMs. It addresses the limitations of uniform token allocation and attention-based token selection, which overlook variations in scene complexity and temporal dynamics. It proposes DynVid, which combines adaptive budget allocation with dynamism-aware importance scoring and Facility Location-based token selection. Experiments across 8 benchmarks and 4 VLLMs show that DynVid preserves 99.3% relative accuracy at 10% average token retention on LLaVA-OneVision and achieves up to 9.49× prefilling speedup on Qwen2.5-VL.

Critical strengths

The paper addresses an important practical challenge in efficient video understanding. The proposed framework provides a systematic approach to reducing visual token redundancy through adaptive token budget allocation and representative token selection.

DynVid is training-free and applicable across different VLLM architectures, making it a flexible and practical solution with strong potential for integration into existing video understanding systems.

The experimental evaluation is extensive, covering 8 benchmarks, 4 VLLMs, and a wide range of compression ratios. The evaluation is further supported by component-wise ablation studies, efficiency analyses, and additional experiments with increased numbers of input frames.

Critical weaknesses

The core ideas behind DynVid have been explored in several closely related studies. Specifically, InfoMerge[1] already explores spectral entropy-based adaptive token allocation, MeToM[2] dynamically allocates token budgets based on video complexity, and FLoC[3] employs Facility Location for video token compression. These studies substantially overlap with the motivation and methodology of DynVid. This paper should clarify the key differences and provide comparisons with these methods.

The proposed dynamism score has an unexpected algebraic simplification. Eqs. (6) to (8) are equivalent to 1−min⁡(s−,s+)1-\min(s^{-},s^{+}), meaning that the novelty and asymmetry terms reduce to a single minimum similarity measure. This raises questions about the necessity of the proposed two-component design. This paper should clarify its motivation and provide separate ablations for novelty and asymmetry.

The Facility Location formulation may suffer from negative importance weights. Since the attention and dynamism scores are normalized using z-score normalization, the resulting importance scores can be negative. Consequently, the objective may assign zero coverage to similar low-importance tokens while rewarding negative similarities. This paper should clarify how negative weights are handled and evaluate alternatives with nonnegative importance scores.

The soft fusion strategy may compromise temporal information. In Eqs. (13) and (14), tokens are aggregated based on feature similarity without explicit temporal constraints. This may mix information from different time steps and obscure temporal ordering, which is inconsistent with the paper's motivation of preserving temporal dynamics. This paper should clarify how temporal information is preserved during fusion

The computational scalability of DynVid is insufficiently evaluated. Appendix B shows that Facility Location selection requires O(∑ini2(d+Bi))O(\sum_i n_i^2(d+B_i)) time and O(ni2)O(n_i^2) memory per segment. Since temporal segmentation has no explicit maximum length, long continuous scenes may form a single large segment, leading to cubic worst-case time complexity under a fixed retention ratio. However, Table 2 only reports efficiency for 32-frame inputs, while the 160-frame experiments lack latency and memory measurements. This paper should provide compression latency, TTFT, peak GPU memory, and segment length statistics at larger frame counts, including stress tests on long continuous scenes.

The effectiveness of DynVid as a standalone compression method remains unclear. The main experiments adopt the inner-LLM query-guided pruning strategy from FlashVID. Although Figure 6 shows lower performance when this stage is removed, the number of tokens entering the LLM also changes. This paper should provide controlled comparisons with and without inner-LLM pruning under matched input token budgets.

[1] Liu, Xinxin, et al. "InfoMerge: Information-aware Token Compression for Efficient Video Large Language Models." arXiv 2026.

[2] Wu, Zhuojie, et al. "MeToM: Metadata-Guided Token Merging for Efficient Video LLMs." CVPR 2026.

[3] Cho, Janghoon, et al. "Floc: Facility location-based efficient visual token compression for long video understanding." ICLR 2026.

Questions for authors

Plz refer to the weaknesses.

10 comments
AuthorsOpenReview ↗

Response to Reviewer LhhM (Part I)

Thank you for your constructive comments. We have addressed all your comments below with additional experiments.


W1. More comparison with recent related studies

Comparison with closely related works. InfoMerge combines segment uniqueness and spectral-entropy-based representational richness for adaptive budget calculation, together with temporal redundancy estimation, attention-based token selection, and density-based merging. MeToM uses codec residuals and GoP packet sizes to guide spatial merging and per-frame budget allocation, followed by attention-guided merging inside the LLM. FLoC employs similarity-based Facility Location selection within temporal blocks using lazy greedy optimization.

Our contribution lies in the specific allocate-then-select design that couples complexity-adaptive allocation with dynamism-aware coverage selection, aiming to make every retained token count. ERA uses PR as an efficient surrogate for effective rank to determine each segment's budget from its representational complexity, without requiring codec metadata. DATS combines temporal novelty and forward/backward asymmetry with attention to estimate token importance, then performs FL-based anchor selection under the allocated budget to optimize importance-weighted content coverage. We use FL as an optimization tool, rather than claiming it as a novel algorithm. Together, ERA and DATS address two crucial questions in token compression: how many tokens to allocate to each segment and which tokens to retain within it.

Additional experimental comparisons. We could not locate publicly available official implementations of InfoMerge, MeToM, and FLoC, and therefore do not report direct experimental comparisons with these methods. To broaden our empirical evaluation, we reproduced OTPrune[1] and AOT[2] and compared them with DynVid on both backbones at retention ratios of 10% and 5%, as reported in Tables 8 and 9 of our response to Reviewer kxtr. DynVid achieves the highest average score in all four settings, providing additional evidence of its effectiveness across backbones and compression levels.

[1] OTPrune: Distribution-Aligned Visual Token Pruning via Optimal Transport, CVPR2026

[2] Token reduction via local and global context optimization for efficient video large language models, CVPR2026


W2. The motivation and more ablations for novelty and asymmetry.

Algebraic simplification and motivation. For tokens with both adjacent frames available, Eqs. (6)–(8) indeed simplify to dyn(vj)=1−min⁡(s−,s+)\mathrm{dyn}(v_j)=1-\min(s^-,s^+). We use the two-component decomposition to distinguish different temporal-change patterns, rather than claiming that their sum is algebraically distinct from this expression. For example, Novelty can measure the absence of a strong match in either adjacent frame, whereas asymmetry can measure differences between forward and backward temporal correspondence. When both similarities are low, novelty is high but asymmetry can be low; when one similarity is high and the other is low, asymmetry is high while novelty can be low. The decomposition thus provides an interpretable motivation and enables separate evaluation of the two signals.

Ablation settings. To separate their contributions, we evaluate the weighted dynamism score dynα(vj)=α⋅novelty(vj)+(1−α)⋅asym(vj)\mathrm{dyn}_{\alpha}(v_j)=\alpha\cdot\mathrm{novelty}(v_j)+(1-\alpha)\cdot\mathrm{asym}(v_j), varying α\alpha from 0 to 1 while keeping the remaining settings unchanged. We evaluate four benchmarks on LLaVA-OneVision and Qwen2.5-VL using 32 frames and an average retention ratio of 10%. Here, α=0\alpha=0 uses asymmetry only, α=1\alpha=1 uses novelty only, and α=0.5\alpha=0.5 gives the equal-weight combination.

Results. As shown in Tables 22 and 23 below, the default equal-weight combination achieves higher average scores than either component alone on both backbones. On LLaVA-OneVision, it achieves 54.18, compared with 53.70 for asymmetry alone and 53.50 for novelty alone; the highest average score is 54.28 at α=0.25\alpha=0.25. On Qwen2.5-VL, the equal-weight combination achieves the highest average score of 56.38, compared with 55.93 for asymmetry alone and 55.88 for novelty alone. These results support the complementary value of the two signals rather than a single universally dominant component. We use equal weights by default to avoid additional tuning.

AuthorsOpenReview ↗

Response to Reviewer LhhM (Part II)


Table 22. LLaVA-OneVision (average retention ratio of 10% for compressed methods)

Variantα\alpha (Novelty weight)Asymmetry weightVideo-MMELongVideoBenchLVBenchTempCompassAvg. scoreRel. Acc (%)
Asymmetry only0.01.056.856.737.164.253.7098.3
Weighted combination0.10.957.256.437.364.253.7898.4
Weighted combination0.250.7557.756.938.064.554.2899.3
DynVid (equal weights)0.50.557.756.538.264.354.1899.1
Weighted combination0.750.2557.556.138.363.953.9598.7
Weighted combination0.90.157.256.037.663.653.6098.1
Novelty only1.00.057.156.137.163.753.5097.9

Table 23. Qwen2.5-VL (average retention ratio of 10% for compressed methods)

Variantα\alpha (Novelty weight)Asymmetry weightVideo-MMELongVideoBenchLVBenchTempCompassAvg. scoreRel. Acc (%)
Asymmetry only0.01.058.257.637.370.655.9396.8
Weighted combination0.10.958.357.537.671.056.1097.1
Weighted combination0.250.7558.257.737.870.956.1597.2
DynVid (equal weights)0.50.558.458.237.871.156.3897.6
Weighted combination0.750.2558.257.537.170.955.9396.8
Weighted combination0.90.158.257.637.570.655.9896.9
Novelty only1.00.058.257.537.270.655.8896.8

W3. Negative importance weights.

Handling negative importance scores. We apologize for omitting the sigmoid step from the normalization description in the manuscript. Our implementation applies a sigmoid after z-score normalization, rather than using z-score alone. Specifically, in our anonymous code repository, line 52 of dynvid/compression_unified.py contains out = torch.sigmoid((x - mu) / sigma). The actual normalization is N(x)=sigmoid⁡((x−μ)/σ)\mathcal{N}(x)=\operatorname{sigmoid}((x-\mu)/\sigma), which maps the scores to values between 0 and 1. Both attention and dynamism are normalized in this manner before combination, so the resulting importance weights are nonnegative. Consequently, multiplying an importance weight by a negative cosine similarity cannot produce a positive contribution; negative weighted similarities contribute zero through the clipping in the FL objective.

Normalization ablation. To evaluate the effect of normalization on importance-weighted selection, we compare z-score alone, which can produce negative importance weights, min–max normalization, and our implemented z-score plus sigmoid on four benchmarks using both backbones. As shown in Tables 24 and 25 below, z-score plus sigmoid improves the average score over z-score alone by 0.90 points on LLaVA-OneVision and 0.63 points on Qwen2.5-VL. With z-score alone, negative importance weights can reverse the intended similarity-based reward: negative cosine similarities may yield positive contributions, whereas positive similarities may yield negative contributions. This provides a possible explanation for its lower performance. Min–max normalization also outperforms z-score alone, matching the average score of our default normalization on LLaVA-OneVision (54.18) and achieving a slightly lower average score on Qwen2.5-VL (56.30 vs. 56.38). These results support the use of nonnegative normalized importance weights and suggest low sensitivity to the choice between min–max normalization and z-score plus sigmoid in the evaluated settings.

Table 24. Normalization ablation on LLaVA-OneVision (average retention ratio of 10%)

NormalizationVideo-MMELongVideoBenchLVBenchTempCompassAvg. scoreRel. Acc (%)
Z-score only57.155.936.263.953.2897.5
Min–max58.156.437.864.454.1899.1
Z-score + sigmoid (DynVid)57.756.538.264.354.1899.1

Table 25. Normalization ablation on Qwen2.5-VL (average retention ratio of 10%)

NormalizationVideo-MMELongVideoBenchLVBenchTempCompassAvg. scoreRel. Acc (%)
Z-score only58.157.337.270.455.7596.5
Min–max58.557.938.070.856.3097.5
Z-score + sigmoid (DynVid)58.458.237.871.156.3897.6
AuthorsOpenReview ↗

Response to Reviewer LhhM (Part III)


W4. Temporal information

Temporal information during soft fusion. Soft fusion updates the anchor features without changing their original positional information or temporal order. Each anchor remains associated with its original frame and position, while absorbing complementary information from non-anchor tokens through the similarity-weighted aggregation in Eqs. (13) and (14). Moreover, fusion is performed independently within each temporally coherent segment defined in Section 4.1, rather than across the entire video, preventing aggregation across segment boundaries. Thus, fusion retains the temporal structure of the selected anchors while incorporating additional within-segment context without increasing the token count.


W5. Scalability

Observed segment lengths. Table 26 below reports segmentation statistics over 900 unique Video-MME videos. As the sampled frame count increases from 64 to 160, the average number of segments per video increases from 29.75 to 70.72, while the mean segment length remains approximately 2.15–2.26 frames. The median segment length is 1 frame, and the 90th-percentile (P90) segment length is 4–5 frames, indicating that most segments remain short while some group multiple consecutive frames. Nevertheless, the maximum segment lengths are 54, 69, and 87 frames, respectively, showing that long segments can still occur.

Table 26. Temporal segment statistics on Video-MME with LLaVA-OneVision at different frame counts.

Sampled framesUnique videosTotal segmentsAvg. segments per videoMean segment length (frames)Median segment length (frames)P90 segment length (frames)Maximum segment length (frames)
6490026,77429.752.151454
12890051,52857.252.241569
16090063,64870.722.261587

Efficiency at larger frame counts. Table 27 below reports efficiency at 64, 128, and 160 frames. Under normal temporal segmentation, DynVid's compression latency is 65.7, 108.0, and 129.4 ms, respectively, lower than FlashVID's at all three frame counts. Its TTFT is 811.1, 1579.5, and 1963.4 ms, and the reported overall peak GPU memory remains close to FlashVID's. Together with the segment statistics, these results demonstrate practical efficiency when most segments remain short, without implying a hard bound on segment length or worst-case computational cost.

Table 27. Efficiency on LLaVA-OneVision at larger frame counts (average retention ratio of 10% for compressed methods).

FramesMethodVision encoding time (ms)Compression time (ms)LLM Forward time (ms)Prefilling time (ms)TTFT (ms)Peak GPU memory (MB)TFLOPs
64Vanilla584.80.01659.61659.62244.422735.4113.4
64FlashVID584.876.7163.9240.6825.422737.88.73
64DynVid with single-segment FL584.82113.2160.32273.52858.322738.48.73
64DynVid (single segment, seed-and-block selection)584.8104.2160.3264.5849.322738.48.73
64DynVid584.865.7160.6226.3811.122738.28.73
128Vanilla1167.40.04003.34003.35170.730126.8290.0
128FlashVID1167.4135.3308.4443.71611.130131.318.2
128DynVid with single-segment FL1167.415948.0304.416252.417419.830131.718.2
128DynVid (single segment, seed-and-block selection)1167.4456.3304.4760.71928.130131.718.2
128DynVid1167.4108.0304.1412.11579.530131.918.2
160Vanilla1454.60.05432.55432.56887.133819.5402.0
160FlashVID1454.6162.1382.7544.81999.433825.923.2
160DynVid with single-segment FL1454.630798.7380.631179.332633.933826.423.2
160DynVid (single segment, seed-and-block selection)1454.6750.5380.61131.12585.733826.423.2
160DynVid1454.6129.4379.4508.81963.433826.423.2

Forced single-segment stress test. To examine the large-segment bottleneck directly, we also force all sampled frames into one segment and directly use FL-based seletion. Compression latency increases to 2113.2, 15,948.0, and 30,798.7 ms, confirming the substantial cost of full greedy FL selection on large segments. This occurs because each greedy step evaluates marginal coverage gains over the large segment-wide token set, and the number of selection steps also grows with the token budget, yielding cubic worst-case selection time at a fixed retention ratio.

We additionally evaluate an exploratory variant using a small number of exact greedy seeds followed by blockwise selection (seed-and-block selection), reducing compression latency to 104.2, 456.3, and 750.5 ms, respectively. These results show that full greedy FL is costly in the single-segment extreme, but computationally optimized selection strategies can substantially mitigate this bottleneck.

Reviewer_nKDsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DynVid is a training-free method for compressing visual tokens in VLLMs. It first splits a video into temporal segments and assigns a token budget to each segment according to the participation ratio of its feature matrix. Within a segment, token importance combines attention scores with temporal novelty and asymmetry; a Facility Location objective then picks representative tokens, and the remaining tokens are softly fused into the selected anchors. A short spectral argument is given to motivate the allocation strategy. Results on four VLLMs and eight benchmarks at 5%–20% retention show that DynVid's advantage over baselines is largest at low retention ratios, though its margin over FlashVID is typically within 1–2 points. Prefilling speedups, ablations, and experiments with different frame counts are also reported.

Critical strengths
  1. At 5%–7.5% retention, DynVid generally outperforms FlashVID by around 1–2 points across four VLLMs, and similar trends appear in the 64-frame and fixed-budget experiments. On Qwen2.5-VL, DynVid at 7.5% slightly exceeds FlashVID at 10% in accuracy (96.0% vs. 95.6%) with a higher prefilling speedup (9.49× vs. 7.49×). This is the main reason I do not consider the paper a clear reject.
  2. The evaluation covers four VLLMs, eight benchmarks, several retention ratios, component ablations, hyperparameter sensitivity analyses, different frame counts, and both successful and failed examples. The use of the same token-budget protocol as FlashVID makes the comparisons easier to interpret, although the small gains still need stronger statistical support.
Critical weaknesses
  1. The method largely combines existing components from FastVID, FlashVID, and prior token merging methods. The main additions are a participation-ratio-based allocation rule and a novelty/asymmetry dynamism score. Prop. 1 concerns rank-b approximation rather than token selection and does not derive the proportional allocation in Eq. 2. I find the novelty limited: the contribution rests on two heuristics layered on FlashVID's pipeline.
  2. The gains over FlashVID are small (mostly 0.4–1.9 points), from single runs without paired confidence intervals or significance tests. DynVid also falls below FlashVID or FastVID on MVBench and TempCompass in several settings, leaving its benefit for temporal understanding unclear. In addition, the importance score in Eq. 9 can be negative after z-score normalization. Multiplying it by cosine similarity in Eq. 10 means a dissimilar anchor can contribute positively to the Facility Location objective while a similar one contributes nothing. The paper does not clarify how negative scores are handled.
Questions for authors
  1. How are negative importance scores handled in Eq. 10? Does the implementation use the raw z-normalized scores or apply any non-negative transformation?
  2. What value of τs\tau_s was used for each model in the main experiments, and were the hyperparameters selected independently of the evaluation benchmarks?
  3. Can the authors provide paired confidence intervals or significance tests against FlashVID, particularly at 5% and 7.5% retention? Can they also explain the less consistent results on MVBench and TempCompass?
  4. How does ERA compare with simpler allocation rules, such as feature variance, or with global Facility Location selection under a comparable computational budget?
  5. Fig. 6 at fe=1.0f_e=1.0 appears to correspond to before-LLM compression only. Can the authors report FlashVID under the same setting and across all retention ratios, so that the two before-LLM compression strategies can be compared directly?
7 comments
AuthorsOpenReview ↗

Response to Reviewer nKDs (Part I)

Thank you for your constructive comments. We have addressed all your comments below with additional experiments.


W1. The contribution

Methodological contribution. FastVID partitions videos into temporally coherent segments and combines density-based token merging with attention-based token selection. However, under a fixed retention ratio, its segment budget is proportional to the number of frames, rather than explicitly adapted to the segment's representational complexity. FlashVID combines frame-wise attention and diversity-based token selection with tree-based spatiotemporal merging. However, such post-hoc merging does not explicitly account for differences in token budget requirements across segments and does not guarantee that the retained tokens maximize importance-weighted content coverage under a fixed budget.

Our contribution is not temporal segmentation, token merging, or inner-LLM pruning in isolation, but a specific allocate-then-select paradigm that couples complexity-adaptive allocation with dynamism-aware coverage selection to make every retained token count. Specifically, ERA adaptively determines each segment's token budget according to its representational complexity, using PR as an efficient surrogate for effective rank. This assigns more tokens to segments with higher estimated complexity and compresses lower-complexity segments more aggressively. DATS then combines temporal novelty and forward/backward asymmetry with attention to estimate token importance and performs FL-based anchor selection under the allocated budget, jointly considering token importance and content coverage. Together, ERA and DATS explicitly address two complementary questions: how many tokens to allocate to each segment, and which tokens to retain within it.

Scope of Proposition 1. Proposition 1 provides theoretical motivation for using effective rank as a complexity signal for adaptive budget allocation: within the specified power-law spectral family, higher effective rank implies greater relative reconstruction error under the same rank budget. This indicates that representations with higher effective rank are harder to approximate under a fixed rank budget, motivating the allocation of more resources to segments with higher representational complexity. It does not establish an equivalence between low-rank approximation and token selection or derive the proportional allocation rule in Eq. 2. As explicitly stated in the accompanying remark, Eq. 2 is a simple practical allocation rule that we designed. Proposition 1 motivates the use of effective rank in this rule, while Section 5.4 empirically evaluates its effectiveness. Our contribution rests on the proposed allocation and selection mechanisms and their experimental validation, rather than a claim that Proposition 1 proves optimal token allocation.


W2&Q3. Paired confidence intervals or significance tests against FlashVID

Paired significance tests. We conduct question-level paired comparisons against FlashVID using two-sided exact McNemar tests at retention ratios of 7.5% and 5%, and report unadjusted p-values. EgoSchema requires online submission for evaluation, so it is temporarily excluded from the paired statistical analysis because per-question ground-truth labels are unavailable. For MLVU, this analysis uses pooled question-level accuracy rather than the official category-macro metric. The results are shown in Table 12-15 below.

Several observed improvements are supported by small p-values. For example, DynVid improves MVBench on LLaVA-OneVision at 5% retention by 2.18%, with p = 0.0004. On Qwen2.5-VL, its MLVU-Dev gains are 2.16% and 1.98% at 7.5% and 5% retention, with p = 0.0012 and p = 0.0017, respectively. These results provide strong evidence against equal marginal accuracy in these settings.

Conversely, the settings where FlashVID slightly outperforms DynVid have large p-values. These include MLVU-Dev on LLaVA-OneVision at 7.5% retention (−0.32%, p = 0.7321) and LVBench at 5% retention (−0.32%, p = 0.8310). Thus, these small observed differences do not provide sufficient evidence of a systematic advantage for FlashVID.

AuthorsOpenReview ↗

Response to Reviewer nKDs (Part II)

Table 12. LLaVA-OneVision (average retention ratio of 7.5%)

BenchmarkTotal queriesFlashVID correctDynVid correctOnly FlashVID correctOnly DynVid correctBoth correctBoth wrongΔ\Delta Acc (%)p-value
Video-MME27001516152711212314041061+0.410.5143
EgoSchema5031N/AN/AN/AN/AN/AN/AN/AN/A
LongVideoBench13377297427285657523+0.970.3382
MVBench40002251229619323820581511+1.130.0339
MLVU-Test5022312451428217243+2.790.0436
MLVU-Dev2174135713501571501200667-0.320.7321
LVBench1549566590120144446839+1.550.1568
TempCompass1580100410073538969538+0.190.8151

Table 13. LLaVA-OneVision (average retention ratio of 5%)

BenchmarkTotal queriesFlashVID correctDynVid correctOnly FlashVID correctOnly DynVid correctBoth correctBoth wrongΔ\Delta Acc (%)p-value
Video-MME27001466148916218513041049+0.850.2376
EgoSchema5031N/AN/AN/AN/AN/AN/AN/AN/A
LongVideoBench13377137317795636529+1.350.1947
MVBench40002187227022430719631506+2.180.0004
MLVU-Test5022282401931209243+2.390.1189
MLVU-Dev2174132713311291331198714+0.180.8530
LVBench1549564559178173386812-0.320.8310
TempCompass15809819893341948558+0.510.4160

Table 14. Qwen2.5-VL (average retention ratio of 7.5%)

BenchmarkTotal queriesFlashVID correctDynVid correctOnly FlashVID correctOnly DynVid correctBoth correctBoth wrongΔ\Delta Acc (%)p-value
Video-MME27001535154010210714331058+0.190.7821
EgoSchema5031N/AN/AN/AN/AN/AN/AN/AN/A
LongVideoBench13377507614253708534+0.820.3049
MVBench40002595259714514724501258+0.050.9533
MLVU-Test5022222251720205260+0.600.7428
MLVU-Dev217412461293781251168803+2.160.0012
LVBench154954456877101467904+1.550.0844
TempCompass15801089110228411061450+0.820.1480

Table 15. Qwen2.5-VL (average retention ratio of 5%)

BenchmarkTotal queriesFlashVID correctDynVid correctOnly FlashVID correctOnly DynVid correctBoth correctBoth wrongΔ\Delta Acc (%)p-value
Video-MME27001513152311812813951059+0.370.5662
EgoSchema5031N/AN/AN/AN/AN/AN/AN/AN/A
LongVideoBench13377367595982677519+1.720.0635
MVBench40002548255019719923511253+0.050.9599
MLVU-Test5022112241730194261+2.590.0789
MLVU-Dev217412351278691121166827+1.980.0017
LVBench15495355458090455924+0.650.4901
TempCompass15801062108442641020454+1.390.0409

MVBench and TempCompass. Tables 4 and 5 in our paper show that DynVid does not achieve the best performance on MVBench and TempCompass in several retention settings. These benchmarks emphasize temporal details within relatively short video clips. MVBench covers diverse tasks such as object tracking, causal reasoning, and other temporal reasoning abilities, while TempCompass specifically tests speed, direction, and event order, including videos with similar static content but different temporal properties.

Fine-grained tasks such as causal reasoning and event ordering may depend on a small set of critical temporal cues. Although DynVid considers both importance and content coverage during selection, this objective may favor broad visual coverage without fully preserving those fine-grained task-specific cues in some cases. We will further investigate these temporal task categories and improve allocation and selection to better preserve fine-grained temporal reasoning cues and event-transition evidence under aggressive compression.

AuthorsOpenReview ↗

Response to Reviewer nKDs (Part III)


W2&Q1. Negative importance scores

Handling negative importance scores. We apologize for omitting the sigmoid step from the normalization description in the manuscript. Our implementation applies a sigmoid after z-score normalization, rather than using z-score alone. Specifically, in our anonymous code repository, line 52 of dynvid/compression_unified.py contains out = torch.sigmoid((x - mu) / sigma). The actual normalization is N(x)=sigmoid⁡((x−μ)/σ)\mathcal{N}(x)=\operatorname{sigmoid}((x-\mu)/\sigma), which maps the scores to values between 0 and 1. Both attention and dynamism are normalized in this manner before combination, so the resulting importance weights are nonnegative. Consequently, multiplying an importance weight by a negative cosine similarity cannot produce a positive contribution; negative weighted similarities contribute zero through the clipping in the FL objective.

Normalization ablation. To evaluate the effect of normalization on importance-weighted selection, we compare z-score alone, which can produce negative importance weights, min–max normalization, and our implemented z-score plus sigmoid on four benchmarks using both backbones. As shown in Table 16 and Table 17 below, our z-score plus sigmoid improves the average score over z-score alone by 0.90 points on LLaVA-OneVision and 0.63 points on Qwen2.5-VL. With z-score alone, negative importance weights can reverse the intended similarity-based reward: negative cosine similarities may yield positive contributions, whereas positive similarities may yield negative contributions. This provides a possible explanation for its lower performance. Min–max normalization also outperforms z-score alone, matching the average score of our default normalization on LLaVA-OneVision (54.18) and achieving a slightly lower average score on Qwen2.5-VL (56.30 vs. 56.38). These results support the use of nonnegative normalized importance weights and indicate low sensitivity to the choice between these two nonnegative normalization strategies in the evaluated settings.

Table 16. Normalization ablation on LLaVA-OneVision (32 frames, average retention ratio of 10%)

NormalizationVideo-MMELongVideoBenchLVBenchTempCompassAvg. scoreRel. Acc (%)
Z-score only57.155.936.263.953.2897.5
Min–max58.156.437.864.454.1899.1
Z-score + sigmoid (DynVid)57.756.538.264.354.1899.1

Table 17. Normalization ablation on Qwen2.5-VL (32 frames, average retention ratio of 10%)

NormalizationVideo-MMELongVideoBenchLVBenchTempCompassAvg. scoreRel. Acc (%)
Z-score only58.157.337.270.455.7596.5
Min–max58.557.938.070.856.3097.5
Z-score + sigmoid (DynVid)58.458.237.871.156.3897.6

Q2. The hyperparameters

Segmentation threshold τs\tau_s and hyperparameter selection. In the main experiments, we used τs=0.6\tau_s = 0.6 for LLaVA-OneVision and τs=0.4\tau_s = 0.4 for Qwen2.5-VL. The different optimal values may reflect architectural differences across backbones, while comparable performance at nearby parameter values suggests that the method does not rely on a precisely tuned optimum. To limit the tuning scope, we selected these values separately for each backbone based on the average score across four benchmarks: Video-MME, LongVideoBench, LVBench, and TempCompass, using a single configuration of 32 frames and an average retention ratio of 10%. Once selected, each backbone's threshold was held fixed across all eight benchmarks and all evaluated retention ratios, without further tuning for individual benchmarks or retention ratios.

Reviewer_kxtrOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

To tackle the video LLM visual token compression task, this paper attempts to solve this task in two aspects: 1) dynamically allocating the visual token budget across various frame segment clips, conditioned on their visual complexity, and 2) enhancing the token importance measurement with a Facility Location to cover more complete and distinct visual regions. First, the authors compute the normalized similarity between mean-pooled features of adjacent frames through a thresholding method to segment the video clip boundaries. Then, they compute the effective rank of the clip-level visual complexity to determine its token budget, which complex and dynamic changing visual content obtains higher budget while other static ones obtain lower budget. This paper further estimates temporal visual token importance through a temporal & asymmetric score and a Facility Location-Based Anchor Selection, which maximizes token importance and content coverage. In the end, the authors conduct extensive video benchmark evaluations across various video LLMs to demonstrate their effectiveness.

Critical strengths
  1. The paper writing is clear and easy to follow.

  2. The motivation sounds reasonable, and the proposed method to employ visual token compression through dynamic token budget allocation and visual content maximization makes sense.

  3. The experimental evaluations and model coverage are extensive, and performance is competitive compared with existing work.

Critical weaknesses
  1. Regarding the video clip segmentation, I am not sure how this will make a difference by following FastVID. I would suggest that the authors to provide with some cases/samples demonstration in terms of the clip segmentation quality.

  2. Though the high-level modeling from temporal dynamic budget allocation and enhanced visual token estimation sounds reasonable, the method proposed in this paper, like the effective rank and Facility Location-Based Anchor Selection seems complicated with existing token selection metrics, like the diversity or similarity. Do the authors conduct a computational overhead analysis in terms of these parts, GPU memory consumption, computation time, etc.?

  3. There are some recent and meaningful related works should be included in the paper to make discussions or experimental comparisons which will strengthen this paper.

    [1] OTPrune: Distribution-Aligned Visual Token Pruning via Optimal Transport, CVPR2026

    [2] Token reduction via local and global context optimization for efficient video large language models, CVPR2026

    [3] S-Prune: Stability-Aware Token Budgeting for Long-Form Video-Language Models, ECCV2026

  4. How about increasing the number of sampled video frames to 126 or more?

Questions for authors

I would like to see more explanations and demonstrations in the authors' response, and please refer to my detailed questions above. I will expect convinced response which addresses my questions and concerns well.

5 comments
AuthorsOpenReview ↗

Response to Reviewer kxtr (Part I)

Thank you for your constructive comments. We have addressed all your comments below with additional experiments.


W1. Regarding the video clip segmentation

As stated in Section 4.1 (Lines 192–196) in the original paper, we follow FastVID for temporal segmentation, placing segment boundaries where the normalized similarity between mean-pooled features of adjacent frames falls below τs\tau_s. We do not claim temporal segmentation itself as a novel contribution. Rather, it provides temporally coherent units for our subsequent allocation and selection stages. Our contributions lie in adaptively allocating token budgets according to the visual complexity of each segment and selecting representative anchors through an importance-weighted Facility Location objective that accounts for both token importance and coverage.

To illustrate the behavior of temporal segmentation, we have added visualization examples to Appendix F Visualization of the revised paper. These examples show sampled frames in chronological order, with red dashed lines marking segment boundaries, illustrating how visually similar consecutive frames are grouped into segments of varying lengths.


W2. Computational overhead analysis in terms of each part

Budget allocation overhead. Table 6 below compares ERA with feature variance allocation under the same temporal segmentation and subsequent selection procedure. Feature variance measures the mean squared Euclidean distance of token features from their segment centroid. We choose it as a simple and computationally inexpensive baseline that allocates token budgets according to within-segment feature dispersion. ERA requires 2.3 ms on LLaVA-OneVision and 4.6 ms on Qwen2.5-VL, adding 1.1 ms and 3.2 ms, respectively, over feature variance. Considering the resulting changes in selection time, the total TTFT increases by only 3.0 ms (0.56%) and 7.1 ms (0.75%), while the average score improves from 54.8 to 55.6 and from 55.3 to 56.0, respectively. ERA increases allocation extra peak memory by 37.8 MB and 136.5 MB, but the reported overall peak GPU memory remains unchanged. These results show that ERA improves allocation effectiveness with a small additional latency cost in the evaluated setting.

Table 6. Computational overhead of budget allocation rules (average retention ratio of 10%).

BackboneAllocation ruleSegmentation time (ms)Allocation time (ms)Selection time (ms)Compression time (ms)TTFT (ms)Allocation extra peak memory (MB)Selection extra peak memory (MB)Overall peak GPU memory (MB)Avg. scoreRel. Acc (%)
LLaVA-OneVisionFeature variance0.81.246.948.9534.5112.6272.519042.654.897.8
LLaVA-OneVisionERA (DynVid)0.82.348.851.9537.5150.4238.519042.655.699.3
Qwen2.5-VLFeature variance1.01.443.646948.7198.0572.525389.255.396.2
Qwen2.5-VLERA (DynVid)1.04.647.553.1955.8334.5591.125389.256.097.4

Token selection overhead. Table 7 below compares our FL-based selection with importance top-k while retaining the same temporal segmentation and ERA allocation. Importance top-k only selects the highest-scoring tokens within each segment under the same segment budget without explicitly optimizing content coverage. FL-based selection increases selection time from 16.7 to 48.8 ms on LLaVA-OneVision and from 16.4 to 47.5 ms on Qwen2.5-VL. The corresponding TTFT increases are 32.1 ms and 31.1 ms. Thus, selection accounts for most of DynVid's compression latency, whereas segmentation and allocation contribute relatively little. FL-based selection also increases selection extra peak memory by 24.7 MB and 184.9 MB, respectively, while the reported overall peak GPU memory remains unchanged. In return for this additional overhead, it improves the average score from 54.7 to 55.6 on LLaVA-OneVision and from 55.3 to 56.0 on Qwen2.5-VL. These results support the benefit of explicitly optimizing importance-weighted content coverage beyond selecting tokens by importance alone.

Table 7. Computational overhead of token selection rules (average retention ratio of 10%).

BackboneSelection ruleSegmentation time (ms)Allocation time (ms)Selection time (ms)Compression time (ms)TTFT (ms)Allocation extra peak memory (MB)Selection extra peak memory (MB)Overall peak GPU memory (MB)Avg. scoreRel. Acc (%)
LLaVA-OneVisionTop-k0.82.316.719.8505.4150.4213.819042.654.797.7
LLaVA-OneVisionFL (DynVid)0.82.348.851.9537.5150.4238.519042.655.699.3
Qwen2.5-VLTtop-k1.04.616.422.0924.7334.5406.225389.255.396.1
Qwen2.5-VLFL (DynVid)1.04.647.553.1955.8334.5591.125389.256.097.4
AuthorsOpenReview ↗

Response to Reviewer kxtr (Part II)


W3. More related works

More related works. OTPrune formulates visual token pruning as distribution alignment under optimal transport and optimizes a tractable log-determinant surrogate based on a second-order Gaussian approximation. AOT establishes attention-guided local and global token anchors within each frame, then uses optimal transport to aggregate information within frames and across frames within temporal clips, while retaining temporally distinct tokens. S3S^3-Prune models spatial uncertainty and segment transitions, stabilizes estimated token demand through Kalman filtering, and applies hierarchical selection before and within the LLM. DynVid shares their goal of efficient visual representation but adopts a different allocate-then-select paradigm: ERA allocates token budgets according to each segment's representational complexity, using PR as an efficient surrogate for effective rank. DATS combines temporal novelty and forward/backward asymmetry into a dynamism score, integrates it with attention score to estimate token importance, and performs FL-based anchor selection that jointly considers token importance and content coverage.

Performance comparison. To compare their performance, we reproduced OTPrune and AOT based on their publicly available official implementations. For OTPrune, we used γ=0.01\gamma=0.01, as specified in the official implementation. For AOT, we used a global anchor ratio of 0.5, a temporal keep ratio of 0.3, a Sinkhorn regularization parameter of 0.1, and fixed clips of four frames for 32-frame inputs. We did not include a quantitative comparison with S3S^3-Prune because we could not locate a publicly available official implementation.

Tables 8 and 9 below compare OTPrune, AOT, and DynVid using 32 frames at average retention ratios of 10% and 5%. DynVid achieves the highest average score in all four settings. On LLaVA-OneVision, DynVid scores 55.6 at 10%, compared with 53.7 for OTPrune and 53.3 for AOT; at 5%, the corresponding scores are 54.0, 51.6, and 51.1. On Qwen2.5-VL, DynVid scores 56.0 at 10%, compared with 52.8 for OTPrune and 54.0 for AOT; at 5%, the corresponding scores are 54.4, 50.9, and 52.7. These results support the effectiveness of DynVid across both backbones under aggressive token compression.

Table 8. LLaVA-OneVision

MethodAverage retention ratioVideo-MMEEgoSchemaLongVideoBenchMVBenchMLVU-TestMLVU-DevLVBenchTempCompassAvg. scoreRel. Acc (%)
Vanilla100%58.560.356.658.347.763.138.764.856.0100.0
OTPrune10%55.958.255.956.743.061.036.462.153.795.8
AOT10%54.159.453.556.742.160.436.263.953.395.2
DynVid10%57.761.056.557.147.462.238.264.355.699.3
OTPrune5%53.957.953.054.739.459.036.558.151.692.1
AOT5%52.058.150.254.741.458.533.660.551.191.3
DynVid5%55.260.354.756.845.860.536.162.654.096.4

Table 9. Qwen2.5-VL

MethodAverage retention ratioVideo-MMEEgoSchemaLongVideoBenchMVBenchMLVU-TestMLVU-DevLVBenchTempCompassAvg. scoreRel. Acc (%)
Vanilla100%61.358.358.968.043.659.637.473.457.5100.0
OTPrune10%54.553.856.963.639.155.534.564.352.891.8
AOT10%55.854.654.665.638.856.636.069.654.093.8
DynVid10%58.456.158.265.542.558.637.871.156.097.4
OTPrune5%51.452.154.361.939.353.833.760.850.988.5
AOT5%54.454.152.463.839.955.234.867.352.791.7
DynVid5%56.454.256.863.842.457.635.268.654.494.6
AuthorsOpenReview ↗

Response to Reviewer kxtr (Part III)


W4. Increasing the number of sampled video frames to 126

More sampled frames. To evaluate DynVid with more sampled video frames, we increased the input to 126 frames and compared it with FlashVID across all eight benchmarks at average retention ratios of 10% and 5% on both backbones. As shown in Table 10 and Table 11 below, DynVid achieves higher average scores in all settings. On LLaVA-OneVision, the average gains over FlashVID are 0.59 and 1.57 points at 10% and 5%, respectively; on Qwen2.5-VL, the corresponding gains are 0.90 and 1.55 points. The larger gains at 5% suggest that DynVid more effectively selects and preserves useful visual information under tighter token budgets. DynVid retains 102.43% and 101.93% of the Vanilla average score on LLaVA-OneVision, and 96.65% and 94.59% on Qwen2.5-VL. These results demonstrate that its average performance advantage persists with 126-frame inputs.

Table 10. LLaVA-OneVision with 126 sampled frames

FramesMethodAvg. retention ratioVideo-MMEEgoSchemaLongVideoBenchMVBenchMLVU-TestMLVU-DevLVBenchTempCompassAvg. scoreRel. Acc (%)
126Vanilla100%58.959.757.556.744.164.940.562.955.65100.00
126FlashVID10%59.260.457.757.345.566.140.264.956.41101.37
126DynVid10%60.060.658.258.047.366.839.765.457.00102.43
126FlashVID5%57.859.455.556.444.764.839.163.655.1699.12
126DynVid5%59.660.658.557.847.865.939.164.556.73101.93

Table 11. Qwen2.5-VL with 126 sampled frames

FramesMethodAvg. retention ratioVideo-MMEEgoSchemaLongVideoBenchMVBenchMLVU-TestMLVU-DevLVBenchTempCompassAvg. scoreRel. Acc (%)
126Vanilla100%65.959.861.568.650.866.542.974.161.26100.00
126FlashVID10%61.858.258.966.745.862.440.472.358.3195.19
126DynVid10%62.858.759.367.547.664.240.673.059.2196.65
126FlashVID5%59.457.155.865.544.560.437.371.256.4092.06
126DynVid5%61.057.657.865.748.262.538.971.957.9594.59
Reviewer_hoHxOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses the costly inference of video LLMs caused by the number of visual tokens, due to the quadratic cost of self-attention. The authors argue that existing training-free compression methods allocate a uniform token budget across frames, ignoring variation in scene complexity, and rely principally on attention scores, ignoring temporal dynamics. They propose DynVid, a training-free method that follows an allocate-then-select paradigm.

In the allocation stage (ERA), the video is temporally segmented into visually coherent segments, using a procedure adapted from FastVID (a threshold on the similarity between consecutive frames). Each segment then receives a budget proportional to its complexity, measured by the participation ratio (PR) of its feature matrix, which serves as an efficient surrogate for the effective rank; PR is computed from the trace and Frobenius norm of the Gram matrix, without eigendecomposition. Proposition 1 motivates this choice. In the selection stage (DATS), token importance is defined as an attention-based saliency plus a dynamism score (temporal novelty plus forward/backward asymmetry, based on nearest-neighbor similarity in adjacent frames). Anchors are selected via a weighted Facility Location objective, solved greedily, to balance importance and coverage. Non-anchor tokens are then softly fused into their most similar anchors, keeping the token count unchanged. The method is evaluated on 8 benchmarks and 4 VLLMs (LLaVA-OneVision, LLaVA-Video, Qwen2.5-VL, Qwen3-VL), together with an acceleration analysis.

Critical strengths
  • Well-motivated problem, grounded in a clear analysis of the literature: adapting the token budget to segment complexity is intuitive, and Table 3 validates it against a uniform budget (+0.4 and +2.2 points of Avg. score on LLaVA-OneVision and Qwen2.5-VL).

  • ERA is cheap (no eigendecomposition) and theoretically motivated (Proposition 1), constituting an elegant contribution in a field that is usually more "incremental", making this method a meaningful contribution to the area.

  • The Facility Location formulation adds coverage to importance-based selection, with a standard greedy approximation guarantee.

  • Broad evaluation: 4 models with different architectures, 8 benchmarks; strong results in both accuracy and acceleration: at a retention ratio of 7.5%, DynVid keeps 98.2% (LLaVA-OneVision-7B), 95.7% (LLaVA-Video-7B) and 96.0% (Qwen2.5-VL-7B) of the full-token accuracy (Table 1), with 6.44× and 9.49× prefilling speedups on LLaVA-OneVision and Qwen2.5-VL, respectively.

  • Appendix figures (8-13) qualitatively show what is captured by token importance, which mitigates artifacts in the [CLS] attention score.

  • Code available.

Critical weaknesses
  • Gains are moderate. Without a naive baseline such as random token selection, it is difficult to assess whether this added complexity is fully justified.

  • Hyperparameter dependencies may limit portability across models and robustness: the sensitivity analyses (Figures 4–7) show that the optima of tau_s (0.6 vs. 0.4) and gamma (0.1 vs. 0.25) differ per model and, for gamma, per benchmark, and these values are selected on benchmarks that are also used for the final evaluation.

  • The ablation of the dynamism score (Table 3) evaluates Dyn as a whole, but does not separate the contributions of its two components (temporal novelty and forward/backward asymmetry), so it remains unclear which one drives the gain.

Questions for authors

Q1. Wen et al. (2025), “Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem?”, show that random token selection can outperform carefully designed pruning methods on image benchmarks, notably FastV. Could you report random token selection (uniformly sampled across the sequence) as a before-LLM baseline on Qwen2.5-VL with 32 frames, at an average retention ratio of 10%, across all eight benchmarks, with both accuracy and efficiency?

Q2. How were the default values of tau_s and gamma selected, given that their optima vary across models and benchmarks?

Q3. There is already an ablation for the contribution of Dyn (Table 3), but would it be possible to see more precisely whether the novelty, the asymmetry, or both contribute the most (at least for 1 benchmark)?

If the authors address these concerns, I would be willing to increase my score.

7 comments
AuthorsOpenReview ↗

Response to Reviewer hoHx (Part I)

Thank you for your constructive comments. We have addressed all your comments below with additional experiments.


W1 & Q1. Naive baseline: random token selection

We evaluate two random-selection baselines in terms of accuracy and efficiency on all eight benchmarks using 32 frames and an average retention ratio of 10%: (i) Random within segments, which preserves temporal segmentation and ERA's segment budgets but replaces FL-based selection with uniform random sampling within each segment; and (ii) Global random, which uniformly samples the total token budget from the entire video token sequence without temporal segmentation and budget allocation.

Accuracy. The accuracy comparison results (average retention ratio of 10%) are shown in Tables 1 and 2 below. Random selection baselines indeed preserve some useful visual information and retains a substantial fraction of the original performance. Moreover, random selection within segments consistently outperforms global random selection, supporting the joint value of temporal segmentation and ERA's segment-wise budget allocation. DynVid further outperforms both random baselines on every benchmark across both backbones. On Qwen2.5-VL, its average score is 56.0, compared with 53.2 for global random selection and 55.2 for random selection within segments. The corresponding scores on LLaVA-OneVision are 55.6, 53.4, and 54.9. These further improvements support the value of importance-guided, coverage-aware selection beyond segmentation and budget allocation alone.

Table 1. LLaVA-OneVision

MethodVideo-MMEEgoSchemaLongVideoBenchMVBenchMLVU-TestMLVU-DevLVBenchTempCompassAvg. scoreRel. Acc (%)
Vanilla58.560.356.658.347.763.138.764.856.0100.0
Random within segments56.060.356.356.546.461.837.664.154.998.0
Global random53.958.954.555.745.459.236.263.253.495.3
DynVid57.761.056.557.147.462.238.264.355.699.3

Table 2. Qwen2.5-VL

MethodVideo-MMEEgoSchemaLongVideoBenchMVBenchMLVU-TestMLVU-DevLVBenchTempCompassAvg. scoreRel. Acc (%)
Vanilla61.358.358.968.043.659.637.473.457.5100.0
Random within segments57.655.558.064.042.358.235.570.555.296.0
Global random56.153.654.162.739.256.235.168.653.292.5
DynVid58.456.158.265.542.558.637.871.156.097.4

Efficiency. Table 3 below shows that random selection incurs less compression overhead, particularly global random selection, which avoids temporal segmentation, segment-wise budget allocation, and FL optimization. Nevertheless, DynVid retains substantial acceleration over Vanilla, achieving prefilling speedups of 5.23 ×\times and 7.53 ×\times and TTFT speedups of 2.13 ×\times and 2.37 ×\times on LLaVA-OneVision and Qwen2.5-VL, respectively. Compared with random selection within segments, DynVid adds 29.8 ms and 29.5 ms to TTFT, corresponding to increases of 5.9% and 3.2%, while improving the average accuracy across eight benchmarks. Compared with global random selection, the additional TTFT is 50.4 ms and 52.1 ms, or 10.4% and 5.8%, respectively, alongside higher average scores of 55.6 vs. 53.4 and 56.0 vs. 53.2. These results quantify the tradeoff: DynVid incurs additional compression cost to preserve more useful visual information while maintaining substantial inference acceleration.

Table 3. Efficiency comparison on Video-MME

ModelMethodAvg. retention ratioVision encoding (ms)Compression (ms)LLM forward (ms)Total prefilling (ms)Prefilling speedupTTFT (ms)TTFT speedup
LLaVA-OneVisionVanilla100%393.70.0752.3752.31.00 ×\times1146.01.00 ×\times
Random within segments10%393.722.191.91146.60 ×\times507.72.26 ×\times
Global random10%393.71.4691.993.368.06 ×\times487.062.35 ×\times
DynVid10%393.751.991.9143.85.23 ×\times537.52.13 ×\times
Qwen2.5-VLVanilla100%755.40.01509.71509.71.00 ×\times2265.11.00 ×\times
Random within segments10%755.423.6147.3170.98.83 ×\times926.32.45 ×\times
Global random10%755.40.98147.3148.2810.2 ×\times903.682.51 ×\times
DynVid10%755.453.1147.3200.47.53 ×\times955.82.37 ×\times
AuthorsOpenReview ↗

Response to Reviewer hoHx (Part II)

W2 & Q2. Hyperparameter selection

Segmentation threshold τs\tau_s. This parameter controls the granularity of temporal segmentation. LLaVA-OneVision peaks at τs=0.6\tau_s = 0.6, whereas Qwen2.5-VL peaks at τs=0.4\tau_s = 0.4. The variation may reflect architectural differences across backbones, such as the spatial token merging in Qwen2.5-VL, which can affect visual embeddings and their temporal similarity distributions. The sensitivity results also show comparable performance at nearby parameter values, suggesting that the method does not rely on a precisely tuned optimum.

We selected τs\tau_s separately for LLaVA-OneVision and Qwen2.5-VL using four benchmarks, namely VideoMME, LongVideoBench, LVBench, and TempCompass, with 32 frames and an average retention ratio of 10%. For each backbone, the selected value was then held fixed across all eight benchmarks and all evaluated retention ratios, without further tuning for individual benchmarks or retention ratios. This allows us to choose an appropriate value for different architectures without tuning it separately for each benchmark or retention ratio.

Soft fusion coefficient γ\gamma. This parameter controls the strength of information incorporated from unselected tokens into selected anchors. LLaVA-OneVision peaks at γ=0.1\gamma = 0.1, with γ=0.25\gamma = 0.25 giving the second-best result, whereas Qwen2.5-VL shows the reverse ordering. This difference may reflect how the backbones respond to feature aggregation. Incorporating additional information can improve coverage, but an overly large γ\gamma may dilute the selected anchor features. A relatively small γ\gamma therefore balances information recovery with preserving the anchor representations. The comparable performance at nearby parameter values further suggests that the method does not require precise tuning of γ\gamma.

We also selected γ\gamma separately for LLaVA-OneVision and Qwen2.5-VL using four benchmarks, namely VideoMME, LongVideoBench, LVBench, and TempCompass, with 32 frames and an average retention ratio of 10%. For each backbone, the selected value was then held fixed across all eight benchmarks and all evaluated retention ratios. This allows us to choose an appropriate value for each model architecture without further tuning for individual benchmarks or retention ratios.


W3 & Q3. The ablation of the novelty and asymmetry

To separate the contributions of temporal novelty and forward/backward asymmetry, we evaluate the weighted dynamism score dynα(vj)=α⋅novelty(vj)+(1−α)⋅asym(vj)\mathrm{dyn}_{\alpha}(v_j) = \alpha \cdot \mathrm{novelty}(v_j) + (1-\alpha) \cdot \mathrm{asym}(v_j), varying α\alpha from 0 to 1 while keeping the remaining settings unchanged. We evaluate four benchmarks on LLaVA-OneVision and Qwen2.5-VL backbones using 32 frames and an average retention ratio of 10%. Here, α=0\alpha=0 uses asymmetry only, α=1\alpha=1 uses novelty only, and α=0.5\alpha=0.5 corresponds to DynVid's equal-weight combination. The results are shown in Table 4 and 5 below.

Results. Combining novelty and asymmetry yields higher average scores than using either component alone on both backbones. On LLaVA-OneVision, asymmetry alone achieves 53.70, compared with 53.50 for novelty alone. Their combination achieves the highest average score of 54.28 at α=0.25\alpha=0.25, while the default equal-weight combination achieves 54.18. On Qwen2.5-VL, the equal-weight combination achieves the highest average score of 56.38, compared with 55.93 for asymmetry alone and 55.88 for novelty alone. Across the tested settings, the highest average scores occur when novelty receives equal or slightly lower weight than asymmetry.

These results support the complementary value of the two signals rather than a single universally dominant component. Temporal novelty is high when neither adjacent frame provides a strong feature match, as may occur for transient visual content or an appearance that differs markedly from both temporal neighbors. In contrast, asymmetry is high when a token has a strong match in one temporal direction but a weak match in the other, as may occur during object appearance or disappearance. Their combination can therefore identify distinct patterns of temporal change. We use equal weights by default to avoid additional tuning.

AuthorsOpenReview ↗

Response to Reviewer hoHx (Part III)

Table 4. LLaVA-OneVision (average retention ratio of 10% for compressed methods)

Variantα\alpha (Novelty weight)Asymmetry weightVideo-MMELongVideoBenchLVBenchTempCompassAvg. scoreRel. Acc (%)
Asymmetry only0.01.056.856.737.164.253.7098.3
Weighted combination0.10.957.256.437.364.253.7898.4
Weighted combination0.250.7557.756.938.064.554.2899.3
DynVid (equal weights)0.50.557.756.538.264.354.1899.1
Weighted combination0.750.2557.556.138.363.953.9598.7
Weighted combination0.90.157.256.037.663.653.6098.1
Novelty only1.00.057.156.137.163.753.5097.9

Table 5. Qwen2.5-VL (average retention ratio of 10% for compressed methods)

Variantα\alpha (Novelty weight)Asymmetry weightVideo-MMELongVideoBenchLVBenchTempCompassAvg. scoreRel. Acc (%)
Asymmetry only0.01.058.257.637.370.655.9396.8
Weighted combination0.10.958.357.537.671.056.1097.1
Weighted combination0.250.7558.257.737.870.956.1597.2
DynVid (equal weights)0.50.558.458.237.871.156.3897.6
Weighted combination0.750.2558.257.537.170.955.9396.8
Weighted combination0.90.158.257.637.570.655.9896.9
Novelty only1.00.058.257.537.270.655.8896.8

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We sincerely thank the Program Chairs, Senior Area Chairs, Area Chair, and all reviewers for their time and constructive feedback. We summarize below the additional evidence, key clarifications, and current discussion status.

Reviewer hoHx. We added global and within-segment random-selection baselines with accuracy and efficiency comparisons, clarified backbone-specific hyperparameter selection, and separately ablated novelty and asymmetry. DynVid outperformed both random baselines on all eight benchmarks across both backbones. The reviewer acknowledged that the ablations and hyperparameter clarifications addressed several initial concerns, recognized technical soundness and consistent improvements, and maintained Weak Accept, considering the contribution alongside related work and the observed gain magnitudes.

Reviewer kxtr. We provided segmentation visualizations, component-level latency and memory analyses, comparisons with OTPrune and AOT, and 126-frame evaluations across all eight benchmarks on both backbones. These experiments quantified the allocation and selection overhead and showed higher average scores than FlashVID at both evaluated retention ratios with 126-frame inputs. The reviewer's initial recommendation was Weak Accept; no follow-up assessment has been posted in response to these additions.

Reviewer nKDs. We clarified the ERA-DATS contribution, Proposition 1's scope, nonnegative importance normalization, and hyperparameter selection. We added normalization ablations, paired significance tests, and comparisons with simpler allocation, global FL, and FlashVID without inner-LLM pruning, while discussing limitations on temporal-detail benchmarks. These comparisons support allocation and standalone compression effectiveness; no follow-up assessment has been posted in response to these additions.

Reviewer LhhM. We added FLoC-style and MeToM-signal controlled comparisons, clarified the dynamism simplification and fusion behavior, and reported larger-frame efficiency, long-segment stress tests, and seed-and-block accuracy at 32 frames. The reviewer acknowledged that several implementation and controlled-comparison concerns were addressed and that the comparisons supported the design, with remaining discussion centered on novelty, gain magnitudes, and long-segment costs. We provided further clarifications; no further reviewer response has been posted following our latest reply.

Overall summary. The additional evidence supports DynVid's specific ERA-DATS coupling for deciding how many tokens to allocate to each segment and which tokens to retain. The controlled comparisons demonstrate consistent average-score gains and standalone effectiveness, while the efficiency measurements show average-case advantages over FlashVID across the evaluated frame settings. If accepted, we will incorporate all these additional experimental results and clarifications into the camera-ready manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.