acceptodds
Under review as a conference paper at ICLR 2027

CompVID: Which Video Regions Deserve More Tokens? Complexity-Aware Token Compression for Video LLMs

Abstract

Existing Video Large Language Model (VLLM) token pruning methods typically perform importance- or diversity-based visual token compression at the video, segment, or frame level. However, they overlook an inherent property of video data: video complexity is unevenly distributed across temporal and spatial dimensions, leading to suboptimal token allocation across spatiotemporal regions. Through empirical analysis, we find that allocating tokens to higher-complexity regions yields larger logit margin gains and contributes to improving video understanding capabilities. Inspired by this, we introduce a new perspective: regional complexity determines how much representation budget a region requires, whereas dimension-wise complexity determines how the region should be partitioned. Building on this insight, we propose CompVID, a training-free framework for complexity-aware video token allocation. Specifically, CompVID employs dimension-wise singular value decomposition (SVD) to reveal the rank structure along each dimension and quantify its intrinsic complexity through singular-value decay. Based on these complexity estimates, we introduce SVDtree, an SVD-guided adaptive partition tree that recursively refines the video feature volume along the complexity-dominant dimension and allocates token budgets to the resulting regions according to their regional complexity. Evaluations on VLLMs across video understanding benchmarks demonstrate that CompVID achieves a state-of-the-art efficiency–accuracy trade-off. By pruning of visual tokens on LLaVA-OneVision-7B, CompVID reduces TFLOPs by and accelerates the LLM prefill stage by , while preserving of the full-token performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Large Language Models, Visual Token Pruning, Efficient Inference.

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_8XuuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a region-wise token compression technique that allocates token budgets according to the complexity of each region. The authors report better preservation of output quality compared with prior image and video token compression methods.

Critical strengths
  1. The experimental evaluation is extensive, with comparisons against a broad range of baselines.
  2. The paper is generally well written and easy to follow.
Critical weaknesses
  1. My primary concern is the motivation for region-wise compression and its claimed advantage over existing video-wise, segment-wise, or frame-wise approaches. The author tried to state that “these methods generally operate at coarse compression levels, overlooking the varying representation demands of local spatiotemporal regions.” in Line 83. However, to my knowledge, such methods often also perform token-level compression within individual frames. This can implicitly achieve region-wise compression, with the regions receiving different levels of compression varying across frames. Therefore, to me, the proposed approach does not introduce a fundamentally different compression framework or addresses a limitation that prior methods actually exhibit. I would appreciate a more precise discussion of this distinction and remain open to clarification from the authors.

  2. Another major concern is whether a video can consistently be partitioned into distinct regions with different levels of complexity. For spatially continuous content, such as rivers or skies, or scenes whose structure changes continuously over time, can the proposed partitioning remain meaningful and temporally consistent? In these cases, the approach appears likely to reduce to token-wise compression. The authors should clarify how regions are defined and maintained under these conditions, and whether region-wise compression retains a meaningful distinction from token-wise compression.

  3. Figure 2,3,5 should specify the relevant experimental settings, including the prompts used and the definitions of the reported metrics. Without this information, it is difficult to interpret the results and assess how well they support the paper’s claims.

  4. I suggest adding a brief statement of the main takeaway to the captions of Figures 2, 3, and 5. In addition, Figure 4 appears somewhat blurry, and the text in panel (d) overlaps. Improving the image quality and layout would make the figure easier to read.

Questions for authors
  1. Can the author show that a video can consistently be partitioned into distinct regions with different levels of complexity? Does this partition fundamentally different than token-wise method?

  2. What is the specific experimental settings in Figure 2,3,5?

7 comments

We appreciate your recognition of the extensive experimental evaluation and that our paper is well written, as well as your valuable feedback. Below, we address your concerns in detail.

W1. Distinction Between Region-Wise and Video-Wise, Segment-Wise, and Frame-Wise Compression

We thank you for this valuable comment. We clarify the distinction between CompVID and existing video-wise, segment-wise, and frame-wise compression methods from two perspectives: methodological differences and qualitative comparisons.

1. Methodological Differences.

We agree that existing methods can perform token-level compression within individual frames, implicitly achieving region-wise compression with different compression ratios across regions and frames. However, their token selection is primarily guided by attention or diversity, rather than regional visual complexity. These criteria are different from complexity-based allocation, as they focus on token importance or redundancy without measuring the complexity of local regions. Specifically, VisionZip (frame-wise) selects tokens based on attention scores and feature similarity, potentially overlooking locally complex regions with low token importance. FastVID (segment-wise) performs temporal segmentation followed by density-based pruning, without explicitly considering complexity differences among spatial regions. FlashVID (video-wise) employs attention- and diversity-based token selection and tree-based merging, focusing on reducing spatiotemporal redundancy without accounting for the varying complexity of local regions. The common limitation of these methods is that regional token retention is an implicit outcome of token selection and merging, rather than explicit allocation based on the complexity of individual 3D spatiotemporal regions. Consequently, locally complex regions may receive insufficient tokens, while simpler regions may retain unnecessary tokens, despite differences in compression ratios across regions and frames.

In contrast, CompVID explicitly partitions the 3D spatiotemporal feature volume along the temporal, height, and width dimensions and allocates token budgets according to the complexity of each region. Thus, the key distinction lies in complexity-guided regional budget allocation. This allows token budgets to directly reflect the varying representation demands of local spatiotemporal regions.

2. Qualitative Comparison. To further examine this distinction, we provide a qualitative comparison with VisionZip (frame-wise), FastVID (segment-wise), and FlashVID (video-wise) in Fig. 14. Although these methods retain different numbers of tokens across spatial regions, the visualizations show that some simple regions retain excessive tokens, while certain visually complex regions receive insufficient coverage. In contrast, CompVID retains more tokens around fine visual details while reducing token retention in simpler regions, consistent with the visual content complexity maps in the bottom row. This comparison further highlights the distinction between complexity-guided regional token allocation and existing video-wise, segment-wise, and frame-wise compression methods.

We have clarified the methodological differences on Page 2, Lines 75–85, and added the qualitative comparison on Page 30, Lines 1566–1592 (Appendix F.4), with the corresponding visualization on Page 33 (Fig. 14).

W2 & Q1. Adaptive Partitioning of Continuous Visual Content and Distinction from Token-Wise Compression

We thank you for this valuable comment. We provide additional qualitative analyses in Figs. 12–14 to address these concerns.

1. Partitioning Videos into Regions with Different Complexity Levels. As shown in the third to fifth rows of Figs. 12 and 13, CompVID partitions videos into regions with different complexity levels. Regions containing richer visual details, such as people and buildings in Case 1 and dense jungle and rough trees in Case 2, generally have higher complexity scores and receive more tokens, whereas simple regions, such as the sky and stream, exhibit lower complexity and receive fewer tokens. Figure 14 further compares the retained tokens across different compression methods, showing that CompVID allocates more tokens to regions with higher visual complexity. These visualizations qualitatively demonstrate that CompVID can distinguish regions with different complexity levels.

2. Distinction from Token-Wise Compression. As shown in the second and third rows of Fig. 12, token-wise compression and CompVID produce different token retention patterns. Specifically, we use Max-Min Diversity as the token-wise compression method to select tokens globally under a shared token budget, allowing all tokens to compete directly. Consequently, it retains excessive tokens in simple regions such as the sky while allocating insufficient tokens to visually complex regions. In contrast, CompVID first partitions the video into 3D spatiotemporal regions, determines the token budget of each region based on its complexity, and then performs token selection within each region, allowing tokens to compete locally under the assigned regional budget.

We have added these analyses on Page 29, Lines 1519–1533 (Appendix F.1), and Page 30, Lines 1566–1592 (Appendix F.4), with visualizations on Pages 31–33 (Figs. 12–14).

W3 & Q2. Clarification of Experimental Settings and Metrics in Figs. 2, 3, and 5

We thank you for this comment. We clarify the experimental settings, axis definitions, and metric calculations for each figure below.

1. Figure 2: Regional Complexity and Token Allocation. We uniformly divide the feature volume of a video from VideoMME into eight spatiotemporal regions, using LLaVA-OneVision-7B at a 10% token retention ratio. The horizontal axis represents the eight regions (R1–R8), while the vertical axis represents normalized regional complexity and token budgets. The Complexity Value Curve is calculated using Eq. (4), whereas the Token Budget Curves represent the numbers of tokens retained by FastVID, FlashVID, and CompVID. Specifically, mean-based normalization is applied jointly to the 24 token counts (8 regions × 3 methods) and the eight regional complexity values. The normalization procedure is detailed in Appendix E.4.

2. Figure 3: Token Distribution and Accuracy. The experiments are conducted on LLaVA-OneVision-7B under a 10% token retention ratio. The horizontal axis compares FastVID, FlashVID, and CompVID across VideoMME, LongVideoBench, and MLVU. The bars represent three regional complexity levels: Low, Medium, and High. The left vertical axis (Token Count (%)) indicates the proportion of retained tokens belonging to each complexity level, while the right vertical axis (Accuracy (%)) reports the corresponding benchmark accuracy. As detailed in Appendix B, regional complexity levels are determined using the 1/3 and 2/3 quantiles of complexity scores within each video, and token proportions are calculated by dividing the retained tokens in each category by the total number of retained tokens.

3. Figure 5: Regional Complexity and Logit Margin Gain. The horizontal axis represents normalized regional complexity, while the vertical axis represents normalized logit margin gain. As detailed in Appendix E.2, we analyze 900 VideoMME videos, each divided into eight spatiotemporal regions, yielding 7200 region-level samples. Under a fixed budget of 640 tokens, we increase the budget of one region from 80 to 150 tokens while reducing the budgets of the remaining seven regions from 80 to 70 tokens each. Using LLaVA-OneVision-7B on VideoMME, logit margin gain is calculated as the change in the difference between the correct answer logit and the highest incorrect answer logit, averaged over all questions associated with each video. The Spearman correlation is 0.56, indicating a positive association between regional complexity and the benefit of additional tokens.

These experimental settings and metric definitions are detailed on Page 2, Lines 87–93; Page 3, Lines 120–124; and Page 4, Lines 183–188, with additional calculations provided in Pages 21–22, Lines 1145–1162 (Appendix B); Page 26, Lines 1364–1403 ( Appendix E.2), and Page 28, Lines 1463–1473 (Appendix E.4).

W4. Improvements to Figure Captions and Figure Quality

We thank you for these helpful suggestions. We have revised the captions of Figs. 2, 3, and 5 and improved the visual quality of Fig. 4.

1. Main Takeaways in Figure Captions. We have added brief statements highlighting the main findings of Figs. 2, 3, and 5. Specifically:

  • Fig. 2: "The results show that the Token Budget Curves of CompVID align more closely with the Complexity Value Curve, demonstrating better complexity–budget matching across different video regions."
  • Fig. 3: "The results show that CompVID retains more tokens in high-complexity regions and fewer in low-complexity regions while consistently achieving higher accuracy across all three benchmarks, demonstrating the effectiveness of complexity-aware token allocation."
  • Fig. 5: "A positive correlation (Spearman ρ = 0.56) indicates that higher-complexity regions tend to yield larger logit margin gains."

These statements have been added to the figure captions on Page 2, Lines 64–69 (Figs. 2 and 3), and Page 4, Lines 175–178 (Fig. 5).

2. Figure 4 Quality and Layout. We have replaced the original PNG image of Fig. 4 with a PDF to improve its readability. We have also adjusted the layout of panel (d) to eliminate the overlapping text.

We hope these responses address your concerns comprehensively. Feel free to let us know if you have any additional questions, and we are more than happy to provide further clarification on any aspect of our work.

Respectfully and sincerely,
Your Authors

Reviewer_oS9GOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates a training-free visual token compression for VLLMs. It addresses a practical problem, inefficient token allocation caused by uneven spatiotemporal complexity. This paper proposes CompVID, a training free framework that uses SVD to estimate regional and dimension wise complexity, followed by adaptive region partitioning and token allocation through SVDtree. Experiments on 4 VLLMs and 7 benchmarks demonstrate competitive performance, preserving 99.7% of the original accuracy with only 10% of visual tokens and achieving a 3.7× prefill speedup on LLaVA-OneVision-7B.

Critical strengths
  1. This paper addresses a practical challenge in efficient video understanding.

  2. The proposed framework, CompVID, is training-free and requires no modifications to the underlying VLLMs, making it practical and potentially applicable to different model architectures and inference pipelines.

  3. The experiments cover 4 VLLMs, 7 video understanding benchmarks, and various token retention ratios, providing a comprehensive evaluation of the proposed framework.

Critical weaknesses

This paper omits several closely related studies on content-adaptive visual token compression. MeToM[1] addresses uneven spatiotemporal information density through metadata-guided spatial token merging and adaptive frame-level token allocation, sharing a similar motivation with CompVID despite methodological differences. STTM[2] explores adaptive spatial partitioning and temporal merging, while InfoMerge[3] employs spectral entropy for content-aware token allocation. None of these closely related works are mentioned in the related work section or included in the experimental comparisons. This omission makes it difficult to properly position CompVID within existing research and assess its methodological contributions relative to prior approaches. This paper should explicitly discuss these works and provide experimental comparisons to establish a clearer understanding of the differences and advantages of CompVID.

Figure 5 provides useful evidence for the relationship between regional complexity and token demand. However, as I understand it, the reported correlation appears to be calculated across regions pooled from different videos, whereas CompVID performs token allocation within each individual video. It is therefore unclear whether the observed correlation also holds within individual videos. A per-video correlation analysis between regional complexity and logit margin gains would provide more direct evidence supporting the proposed token allocation strategy.

The definition of complexity in the paper is somewhat ambiguous. It is unclear whether complexity refers to the complexity of the original video content or that of the features extracted by the visual encoder. If the former is intended, the proposed complexity measure is not sufficiently validated. Since SVD is applied to visual encoder features rather than raw video content, the resulting spectral entropy does not necessarily reflect the actual visual complexity of the corresponding regions. The paper should clarify the definition of complexity and provide further justification or empirical evidence to establish the relationship between the proposed measure and the complexity of the original visual content.

The proposed complexity measure has an undefined edge case. In Eq. (3), the normalization term becomes zero when the feature matrix has rank one, leading to an undefined value. This can occur when Eq. (6) splits a region at its boundary, producing a child region with size one along a particular dimension. However, the paper does not specify any minimum region size constraints or how such cases are handled during recursive partitioning. Further clarification is needed to ensure the mathematical validity and robustness of the proposed method.

The necessity of using SVD to determine partition boundaries is unclear. In Equation (6), the Euclidean distances between adjacent SVD representations are mathematically equivalent to those between the original visual encoder features under the given formulation. Therefore, the same partition boundaries could be obtained directly from the original feature differences without applying SVD. The paper should clarify what additional benefits SVD provides for identifying partition boundaries and justify its necessity in this process.

Although Tables 3 and 4 provide efficiency measurements, the scalability of the compression algorithm to longer video inputs remains unclear. Figure 10 evaluates up to 320 input frames but only reports accuracy. Reporting the compression latency under these settings would help determine whether the efficiency gains can be maintained as the number of input frames increases.

The handling of positional information and DeepStack features after token compression is unclear. Section 4.3 states that selected tokens from different regions are concatenated before entering the LLM, but does not explain how their original spatiotemporal positions are preserved. This is particularly important for Qwen3-VL, which uses spatiotemporal positional encoding and injects DeepStack visual features into multiple LLM layers. Since token compression is performed after the vision encoder, it is unclear how the retained tokens are aligned with the corresponding DeepStack features and positional indices. The paper should clarify how token ordering, positional encoding, and DeepStack feature alignment are handled, and whether DeepStack remains enabled during evaluation.

This paper lacks qualitative visualizations to demonstrate whether the proposed regional complexity measure accurately reflects the visual complexity of actual video content. Although Figures 5, 6, and 8 provide statistical analyses, they do not illustrate how the estimated complexity varies across different regions in video frames. It remains unclear whether regions assigned higher complexity scores contain more informative visual content or are simply dominated by background textures and noise. Qualitative examples showing regional complexity maps, adaptive partition boundaries, and retained tokens on representative videos would provide more direct evidence of the effectiveness of the proposed complexity estimation and token allocation strategies. This concern is also related to my Weakness 3.

[1] Wu, Zhuojie, et al. "MeToM: Metadata-Guided Token Merging for Efficient Video LLMs." CVPR 2026.

[2] Hyun, Jeongseok, et al. "Multi-granular spatio-temporal token merging for training-free acceleration of video llms." ICCV 2025.

[3] Liu, Xinxin, et al. "InfoMerge: Information-aware Token Compression for Efficient Video Large Language Models." arXiv 2026.

Questions for authors

Plz refer to the weaknesses.

8 comments

We appreciate your recognition of the practical significance and comprehensive experimental evaluation of CompVID. For the concerns, we have provided additional experimental comparisons, quantitative and qualitative analyses, and detailed methodological clarifications in the revised manuscript. We hope these additions can address your concerns.

W1. Comparison with Related Content-Adaptive Compression Methods

We thank you for highlighting these closely related studies. We have added experimental comparisons with MeToM, STTM, and InfoMerge in Table 12 and expanded the discussion in Appendix D (Related Work).

Following the different token retention ratios adopted in the these studies, we conduct comparisons under their respective settings. As shown in Table 12, our method achieves the highest average relative accuracy of 102.1% at 30% retention, outperforming MeToM and STTM by 0.2% and 0.8%, respectively. At 10% retention, our method achieves 99.4%, exceeding InfoMerge by 2.2%.

These three methods focus on content-adaptive video token compression. However, MeToM and InfoMerge primarily allocate token budgets at the frame and segment level, while STTM relies on similarity-based spatial partitioning and temporal merging. Although these methods adapt token compression to variations in video content, they do not explicitly allocate token budgets according to the varying complexity of local 3D spatiotemporal regions. In contrast, CompVID adaptively partitions videos into 3D spatiotemporal regions and allocates token budgets based on regional complexity, preserving more tokens in complex regions while reducing redundancy in simpler ones.

Table 12: Comparison of content-adaptive visual token compression methods on LLaVA-OneVision-7B. Results are expressed as percentages of the full-token baseline performance.

MethodRetention RatioVideoMMELongVideoBenchEgoSchemaMLVUAvg.
MeToM (CVPR26)30%102.7100.9101.6102.2101.9
STTM (ICCV25)30%102.6100.5100.7101.4101.3
Ours30%101.9102.1102.6101.7102.1
InfoMerge (arXiv26)10%98.597.9–95.197.2
Ours10%99.5101.2100.297.499.4

We have added the experimental comparisons to Page 18, Lines 928–964 (Table 12 and Appendix A.8), and expanded the related work discussion on Pages 24–25, Lines 1292–1304 (Appendix D).

W2. Within-Video Correlation Analysis

We thank you for this valuable suggestion. We conduct a within-video correlation analysis on 2700 video-question samples (900 videos × 3 questions) from VideoMME using LLaVA-OneVision-7B at a 10% token retention ratio. Specifically, we compute the Spearman correlation (ρ\rho) between regional complexity and logit margin gains separately within each video-question sample.

As shown in Table 16 and Fig. 11, 2509 out of 2700 samples (92.9%) exhibit positive correlations, with 1716 samples (63.6%) achieving ρ≥0.5\rho\geq0.5, indicating that higher-complexity regions generally yield larger logit margin gains within individual videos. Samples with non-negative correlations achieve 58.5% Acc, exceeding the 53.9% achieved by those with negative correlations by 4.6%. These results show that regional complexity is positively associated with token demand within individual videos, further supporting our complexity-guided token allocation strategy.

Table 16: Accuracy across different within-video Spearman correlation intervals on VideoMME.

Spearman ρ\rhoSamplesAcc (%)
[−1,−0.5)[-1, -0.5)4553.3
[−0.5,0)[-0.5, 0)14654.1
[0,0.5)[0, 0.5)79358.3
[0.5,1][0.5, 1]171658.6

We have added this analysis to the revised manuscript on Pages 21–22, Lines 1080–1124 (Appendix A.13 and Fig. 11), and Page 22, Lines 1134–1143 (Table 16).

W3. Definition of Regional Complexity and Relationship with Visual Content Complexity

We thank you for this valuable comment. We clarify the definition of regional complexity and provide additional visualizations to examine its relationship with the original video content.

1. Definition of Regional Complexity. In CompVID, regional complexity is defined based on visual encoder features. Specifically, we apply SVD to the visual features of each spatiotemporal region along the temporal, height, and width dimensions. Dimension-wise complexity is quantified based on the singular-value distributions, where slower singular-value decay indicates higher complexity (Eq. (3)). Regional complexity is then calculated by combining dimension-wise complexity with region size (Eq. (4)).

2. Relationship with Visual Content Complexity. To examine the relationship between regional complexity and the complexity of the original visual content, we compare the Region Complexity Maps and Visual Content Complexity Maps in Figs. 12 and 13. The two maps exhibit broadly consistent complexity distributions. Regions containing richer visual details, such as people and buildings in Case 1 and dense jungle and rough trees in Case 2, generally exhibit higher complexity scores. In contrast, relatively homogeneous regions, such as the sky and stream, exhibit lower complexity. These observations indicate that our feature-based complexity measure reflects variations in the visual complexity of the original video content. The Visual Content Complexity Maps are computed using the Sobel operator (Sobel & Feldman, 1968) [1], which measures local spatial variations through image gradients.

We have added the detailed analysis to Page 29, Lines 1549–1565 (Appendix F.3), with the corresponding visualizations on Pages 31–32 (Figs. 12–13). The calculation of visual content complexity is provided on Page 28, Lines 1476–1505 (Appendix E.5).

[1] Sobel, I., and Feldman, G. (1968). A 3×3 Isotropic Gradient Operator for Image Processing. Stanford Artificial Intelligence Project.

W4. Clarification of Edge Cases in Recursive Partitioning

We thank you for raising this concern. First, when a dimension has size one or its unfolded feature matrix has rank at most one, we set cd(R)=0c_d(\mathcal{R})=0 (Eq. (3)). Second, during recursive partitioning, dimensions of size one are excluded from further splitting, while dimensions larger than one can be further partitioned. If all three spatiotemporal dimensions have size one, the region is treated as a terminal region, retaining a single visual token.

We have added these clarifications on Page 6, Lines 276–277 and Lines 309–313.

W5. Role of SVD in Partition Boundary Identification

We thank you for raising this concern. We agree that, under the full SVD formulation, the Euclidean distances between adjacent SVD-derived representations are mathematically equivalent to those between the original visual features. Specifically, according to Eq. (2), we have:

X(d∗)=U(d∗)Σ(d∗)(V(d∗))⊤=Z(d∗)(V(d∗))⊤.\mathbf{X}^{(d^*)}=\mathbf{U}^{(d^*)}\mathbf{\Sigma}^{(d^*)}(\mathbf{V}^{(d^*)})^\top=\mathbf{Z}^{(d^*)}(\mathbf{V}^{(d^*)})^\top.

Here, Z(d∗)\mathbf{Z}^{(d^*)} denotes the SVD-derived representation, while X(d∗)\mathbf{X}^{(d^*)} denotes the original visual encoder features unfolded along the selected dimension. Their i-th rows, zi(d∗)\mathbf{z}_i^{(d^*)} and xi(d∗)\mathbf{x}_i^{(d^*)}, represent the corresponding positions along that dimension. Since the columns of V(d∗)\mathbf{V}^{(d^*)} are orthonormal, we have:

(V(d∗))⊤V(d∗)=I.(\mathbf{V}^{(d^*)})^\top\mathbf{V}^{(d^*)}=\mathbf{I}.

Therefore, the squared Euclidean distance between adjacent original features can be expressed as:

∥xi+1(d∗)−xi(d∗)∥22=(zi+1(d∗)−zi(d∗))(V(d∗))⊤V(d∗)(zi+1(d∗)−zi(d∗))⊤=∥zi+1(d∗)−zi(d∗)∥22.\lVert\mathbf{x}_{i+1}^{(d^*)}-\mathbf{x}_i^{(d^*)}\rVert_{2}^{2}=(\mathbf{z}_{i+1}^{(d^*)}-\mathbf{z}_i^{(d^*)})(\mathbf{V}^{(d^*)})^\top\mathbf{V}^{(d^*)}(\mathbf{z}_{i+1}^{(d^*)}-\mathbf{z}_i^{(d^*)})^\top=\lVert\mathbf{z}_{i+1}^{(d^*)}-\mathbf{z}_i^{(d^*)}\rVert_{2}^{2}.

Thus,

∥zi+1(d∗)−zi(d∗)∥2=∥xi+1(d∗)−xi(d∗)∥2.\lVert\mathbf{z}_{i+1}^{(d^*)}-\mathbf{z}_i^{(d^*)}\rVert_{2}=\lVert\mathbf{x}_{i+1}^{(d^*)}-\mathbf{x}_i^{(d^*)}\rVert_{2}.

Therefore, both representations yield the same partition point in Eq. (6).

However, the unique advantage of SVD lies in its ability to characterize the rank structure and complexity distribution of visual features. Specifically, the singular-value distributions quantify regional and dimension-wise complexity (Eqs. (3)–(4)), allowing CompVID to identify which region requires further partitioning and along which dimension (Eq. (5)). In contrast, adjacent original feature differences only determine the partition point along a given dimension and do not directly characterize these complexity distributions. Thus, SVD enables complexity-guided region and dimension selection, while Euclidean distance determines the partition point.

W6. Efficiency Analysis on Longer Video Inputs

Thanks for your this valuable suggestion. To evaluate the efficiency scalability of CompVID, we conduct additional efficiency experiments using 128, 256, and 320 input frames, with corresponding token retention ratios of 25%, 12.5%, and 10%. We report compression latency, LLM forward latency, and total prefill latency on LLaVA-OneVision-7B using an NVIDIA A100 GPU.

As shown in Table 15, CompVID achieves compression latencies of 204.4 ms, 410.4 ms, and 480.7 ms at 128, 256, and 320 frames, respectively. Although its compression latency is slightly higher than FlashVID, the reduced LLM forward latency offsets this overhead. Consequently, CompVID achieves the lowest total prefill latency of 379.0 ms, 727.3 ms, and 913.8 ms, outperforming FlashVID by 4.3 ms, 29.7 ms, and 8.8 ms, respectively. CompVID also achieves the highest accuracy across all three settings, demonstrating that its efficiency advantages are maintained as the number of input frames increases.

Table 15: Comparison of prefill latency and accuracy under different numbers of input frames. Acc (%) denotes the accuracy measured on VideoMME.

MethodCompression (ms)LLM Forward (ms)Total (ms)Acc (%)
128 Frames / 25% Retention
HoliTom205.4200.5405.960.7
FastVID221.2174.6395.857.5
FlashVID198.0185.3383.358.0
Ours204.4174.6379.061.8
256 Frames / 12.5% Retention
HoliTom413.5384.6798.160.0
FastVID453.8316.9770.755.9
FlashVID394.3362.7757.060.4
Ours410.4316.9727.360.5
320 Frames / 10% Retention
HoliTom496.4478.3974.759.9
FastVID511.3433.1944.454.7
FlashVID468.1454.5922.659.5
Ours480.7433.1913.861.2

We have added the detailed efficiency analysis to the revised manuscript on Pages 19–20, Lines 1018–1056 (Appendix A.12), with the corresponding results on Page 20, Lines 1059–1078 (Table 15).

W7. Clarification of Positional Information and DeepStack Features

For all VLLMs, after token compression, the retained visual tokens are reordered according to their original spatiotemporal indices before being passed to the LLM. This ensures that the compressed visual token sequence preserves the original temporal and spatial ordering.

For Qwen3-VL, the MRoPE positional indices corresponding to the retained visual tokens are preserved. This maintains the original spatiotemporal positional information of each retained token. For DeepStack, the same token selection indices are applied to the corresponding DeepStack visual features, ensuring that the visual features injected into different LLM layers remain consistent with the compressed visual token sequence. DeepStack remains enabled during evaluation.

These implementation details are clarified on Page 7, Lines 350–353 (Section 5.1).

W8. Qualitative Analysis of Regional Complexity and Token Allocation

To address this concern, we provide two representative examples in Figs. 12 and 13, visualizing regional complexity maps, visual content complexity maps, partition boundaries, and retained visual tokens.

1. Relationship Between Regional Complexity and Visual Content Complexity. As shown in the fourth and fifth rows of Figs. 12 and 13, the Region Complexity Maps and Visual Content Complexity Maps exhibit broadly consistent complexity distributions. Regions containing richer visual details, such as people and buildings in Case 1 and dense jungle and rough trees in Case 2, generally have higher complexity scores. In contrast, relatively homogeneous regions, such as the sky and stream, exhibit lower complexity. This indicates that our feature-based regional complexity measure reflects variations in the original visual content.

2. Adaptive Partitioning and Token Allocation. As shown in the third row of Figs. 12 and 13, CompVID adapts partition boundaries and token allocation according to regional complexity. Visually complex areas, such as people and buildings in Case 1 and dense jungle and rough trees in Case 2, are divided into finer regions and allocated more tokens. In contrast, relatively homogeneous areas, such as the sky and stream, are grouped into larger regions with fewer tokens. These examples demonstrate that CompVID adaptively partitions video regions and allocates token budgets based on their varying visual complexity.

We have added these analyses on Page 29, Lines 1519–1565 (Appendix F.1–F.3), with visualizations on Pages 31–32 (Figs. 12–13).

We hope these responses address your concerns comprehensively. Feel free to let us know if you have any additional questions, and we are more than happy to provide further clarification on any aspect of our work.

Respectfully and sincerely,
Your Authors

Reviewer_A5kwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CompVID, a training-free framework for complexity-aware video token compression. CompVID estimates temporal and spatial complexity using the singular-value distributions of region features. It then uses SVDtree to recursively partition the video feature volume along the complexity-dominant dimension and allocates more tokens to regions with higher estimated complexity. Representative tokens within each region are retained using Max-Min Diversity selection. Experiments across multiple VLLMs and video understanding benchmarks show that CompVID generally achieves a strong accuracy–efficiency trade-off.

Critical strengths
  • This paper is clearly written and easy to follow.
  • The proposed algorithm is intuitive and well motivated.
  • Extensive experiments across multiple models, benchmarks, and retention ratios show the effectiveness of the proposed algorithm.
Critical weaknesses
  • More direct analysis supporting the paper’s central claim regarding complexity-aware token allocation would further strengthen the work.
Questions for authors
  1. Although the ablation studies show that regional allocation improves over uniform allocation, they do not provide a direct validation that high-complexity regions are genuinely more important. It would be good to have some more analysis regarding this. For example, a comparison against random, shuffled, and reverse-complexity allocation (while fixing the partition, token selector, and total budget) would be helpful.

  2. The proposed algorithm estimates complexity only from visual features and does not consider the input question. How does the method behave when a visually complex region is irrelevant to the question, while a relatively simple region contains the answer-critical evidence? A qualitative visualization or quantitative analysis related to this issue would help clarify this potential failure mode. A comparison with a simple query-aware allocation variant may also be informative.

  3. In Eq. (1), U is introduced as the amount of information preserved by the compressed visual tokens. However, U is neither operationally defined nor directly optimized. Is there any theoretical connection showing that the proposed SVD-based partitioning and budget allocation maximize, or approximately maximize, a concrete instantiation of U? Otherwise, Eq. (1) appears to serve only as a conceptual formulation, and further clarification of what is meant by “information preservation” would be helpful.

4 comments

We are grateful to you for recognizing the clear writing, the intuitive and well-motivated design of our approach, and the extensive experimental evaluations across multiple models and benchmarks, alongside your valuable feedback and constructive suggestions. Below, we address your concerns in turn.

Q1. Ablation Study on Token Allocation Strategies

To further validate the effectiveness of complexity-guided token allocation, we compare our method with Random, Shuffled, and Reverse-Complexity strategies under the same regional partitioning, token selection method, and token budget. The results are presented in Table 8.

As shown in Table 8, our method achieves the best performance across all four benchmarks, reaching 99.7% Acc. Compared with Random, Shuffled, and Reverse-Complexity, our method improves Acc by 4.3%, 2.7%, and 5.8%, respectively. Random allocation ignores regional representation demands, while Shuffled disrupts the correspondence between complexity and token budgets. Reverse-Complexity performs worst because it allocates fewer tokens to complex regions, potentially discarding important visual information. These limitations lead to a mismatch between regional representation demands and allocated token budgets, resulting in suboptimal compression performance. In contrast, our method assigns more tokens to high-complexity regions while reducing redundancy in simpler regions, demonstrating the effectiveness of complexity-guided token allocation.

Table 8: Ablation study of different token allocation strategies on LLaVA-OneVision-7B at a 10% token retention ratio.

Allocation MethodsVideoMMELongVideoBenchEgoSchemaMLVUAvg. ScoreAcc (%)
Random56.955.256.859.657.195.4
Shuffled56.756.158.061.658.197.0
Reverse-Complexity55.354.456.658.756.393.9
Ours58.257.160.463.059.599.7

We have added this experiment to the revised manuscript on Page 16, Lines 800–809, 837-850 (Table 8 and Appendix A.3).

Q2. Qualitative Analysis of Query-Aware Compression

We thank you for highlighting this potential limitation. To address this concern, we provide a qualitative comparison with a simple query-aware compression. Specifically, before feeding visual tokens into the LLM, we compute the similarity between visual features and the input question, and allocate token budgets according to the resulting similarity scores, assigning more tokens to question-relevant regions.

As shown in Fig. 13 of the revised manuscript, for the question "How many streams did the cat cross in the video?", CompVID incorrectly predicts one stream, whereas query-aware compression correctly predicts two. Query-aware compression preserves more answer-critical tokens around the cat and streams, while CompVID allocates more tokens to visually complex but less relevant regions. This indicates that visual complexity does not always correspond to question relevance, suggesting that incorporating question relevance could further improve regional token allocation.

We have added this qualitative analysis to the revised manuscript on Page 29, Lines 1535–1544 (Appendix F.2), with the corresponding visualization in Fig. 13 on Page 32.

Q3. Definition of UU

We thank you for this valuable comment. To clarify this, the UU is defined as a measure of information-preservation surrogate utility based on regional complexity and token allocation:

U(X~(P,b))=∑k=1KC(Rk)log⁡(1+bkC(Rk)).U(\widetilde{\mathbf{X}}(\mathcal{P},\mathbf{b})) = \sum_{k=1}^{K} C(\mathcal{R}_k)\log\left(1+\frac{b_k}{C(\mathcal{R}_k)}\right).

Here, C(Rk)C(\mathcal{R}_k) denotes the SVD-derived regional complexity, bkb_k represents the allocated token budget, and the logarithmic function models diminishing gains in information-preservation utility as more tokens are allocated. It shows that UU is strictly increasing and concave with respect to regional token budgets. Under a fixed partition and the global budget constraint ∑k=1Kbk=B\sum_{k=1}^{K}b_k=B, maximizing UU yields the optimal continuous allocation:

bk∗=BC(Rk)∑j=1KC(Rj).b_k^* = \frac{BC(\mathcal{R}_k)}{\sum_{j=1}^{K}C(\mathcal{R}_j)}.

This exactly recovers the complexity-proportional allocation in Eq. (7), demonstrating that our SVD-derived regional complexity-guided token allocation maximizes the defined surrogate utility under a fixed partition.

We have added the detailed definition and theoretical derivation to the revised manuscript on Pages 27–28, Lines 1404–1461 (Appendix E.3).

We hope these responses address your concerns comprehensively. Feel free to let us know if you have any additional questions, and we are more than happy to provide further clarification on any aspect of our work.

Respectfully and sincerely,
Your Authors

Dear Reviewer A5kw,

We sincerely appreciate your time and effort in reviewing our manuscript. As the discussion period is nearing its end, we would like to ensure whether we have addressed your concerns. If there are any additional feedback you would like us to consider, please let us know. We greatly value your insights and are willing to address any remaining concerns to further improve our work.

Thank you again for your valuable feedback and consideration.

Best regards, Authors

I appreciate the authors’ response and will maintain my original positive rating

Dear Reviewer A5kw,

We sincerely thank you for your encouraging feedback. We are truly grateful for your time and thoughtful consideration of our manuscript. We are pleased to know that our responses have addressed your concerns. We deeply appreciate your valuable comments, which have helped us further improve the quality of our work.

Respectfully and sincerely, All Authors

Comments on the paper

1 comment

General Response to All Reviewers

We sincerely thank the Area Chair and all reviewers for their constructive feedback and valuable suggestions. We have strengthened the manuscript through additional experiments, quantitative and qualitative analyses, and methodological clarifications. We have uploaded a revised version with all changes highlighted in blue. The major updates are summarized below.


(i) Clarification of CompVID's motivation and methodological distinctions. (Section 1; Appendix A.8, D, F.1, and F.4)

To address concerns about the distinction between region-wise compression and existing video-wise, segment-wise, frame-wise, and token-wise approaches, we clarified that existing methods implicitly produce diverse regional token distributions through importance- and diversity-based token selection, without considering the complexity of local 3D spatiotemporal regions. In contrast, CompVID uses the measured complexity to guide 3D spatiotemporal partitioning and assigns region-specific token budgets before token selection, enabling finer allocation according to regional complexity. We expanded the comparisons with VisionZip, FastVID, and FlashVID and added experiments with content-adaptive methods MeToM, STTM, and InfoMerge (Table 12). CompVID outperforms MeToM and STTM by 0.2% and 0.8% at 30% token retention, respectively, and surpasses InfoMerge by 2.2% at 10% retention. Additional visualizations in Figs. 12 and 14 illustrate the differences between complexity-guided regional allocation and existing compression strategies.

(ii) Validation of complexity-guided token allocation. (Appendix A.3, A.13, and E.2; Tables 8 and 16; Fig. 11)

To examine whether higher-complexity regions benefit from receiving more tokens, we added comparisons against Random, Shuffled, and Reverse-Complexity allocation, keeping the partitioning, token selection, and total budget fixed. CompVID achieves 99.7% Acc, outperforming these alternatives by 4.3%, 2.7%, and 5.8%, respectively. We further conducted a within-video correlation analysis on 2700 video-question samples from VideoMME. Among them, 92.9% exhibit positive correlations between regional complexity and logit margin gain, and 63.6% achieve Spearman ρ ≥ 0.5. These results further demonstrate the effectiveness of complexity-guided regional token allocation.

(iii) Regional complexity interpretation and qualitative validation. (Appendix E.5 and F.1–F.4; Figs. 12–14)

We clarified that regional complexity is estimated from visual encoder features using dimension-wise SVD and provided qualitative comparisons with the original video content. The Region Complexity Maps and Visual Content Complexity Maps exhibit broadly consistent distributions, with higher complexity around detailed objects and lower complexity in simpler areas. Additional visualizations demonstrate CompVID’s adaptive partitioning of visual content and complexity-guided token budget allocation across regions. We also introduced a query-aware comparison highlighting a limitation of complexity-based allocation: visually complex regions are not necessarily the most relevant to the input question.

(iv) Methodological and implementation clarifications. (Sections 4.2, 4.3, and 5.1; Appendix E.3)

We provided a definition of the information-preservation surrogate utility in Eq. (1) and derived its relationship with complexity-proportional token allocation. The analysis establishes that the allocation in Eq. (7) maximizes the defined surrogate utility under a fixed partition and continuous token budgets. We also clarified the role of SVD in selecting regions and partition dimensions, the clarification of size-one and rank-one cases in recursive partitioning, and the preservation of spatiotemporal positional information. For Qwen3-VL, we specified how MRoPE indices and DeepStack features remain aligned with the retained visual tokens.

(v) Long-video efficiency analysis. (Appendices A.11 and A.12; Table 15)

To evaluate efficiency scalability, we added experiments with 128, 256, and 320 input frames at retention ratios of 25%, 12.5%, and 10%, respectively. CompVID achieves total prefill latencies of 379.0 ms, 727.3 ms, and 913.8 ms, outperforming FlashVID by 4.3 ms, 29.7 ms, and 8.8 ms, while maintaining higher VideoMME accuracy across all three settings.

(vi) Experimental clarifications and figure improvements. (Sections 1 and 3; Appendices B, E.2, and E.4; Figs. 2–5)

We clarified the experimental settings, axis definitions, normalization procedures, and metric calculations for Figs. 2, 3, and 5, and added their main takeaways to the captions. Furthermore, we replaced the original PNG of Fig. 4 with a PDF and corrected the overlapping text in panel (d) to improve readability.


We hope these revisions can address the main concerns raised during review. We are grateful for the reviewers’ guidance, which materially enhanced the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.