- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a training-free visual token compression framework for Video-LLMs under aggressive token budgets. Its core component is set-aware anchor selection: each token is selected according to how much its addition reduces the total importance-weighted projection error of the original visual features onto the selected anchor subspace. This is a consequence-aware measurement rather than previous compression methods that is based on a score prior. The framework further incorporates segment-wise selection, selective token merging, proportional attention calibration, and text-guided compression within the LLM. The author reports the highest average accuracy among the evaluated methods at layer-average token retention ratios of 2%, 5%, and 10%.
Critical strengths
-
The paper identifies a meaningful limitation of individual importance and pairwise similarity criteria. Existing methods do not directly measure how the selected tokens jointly represent the original features.
-
The evaluation covers three backbones, four benchmarks, and several retention ratios. AVTC achieves the best average accuracy in the main comparisons, with larger advantages over competing methods as the token budget becomes tighter. Although FlashVID is also competitive for Qwen-2.5-VL.
Critical weaknesses
-
A major concern is whether the proposed method truly improves information preservation. One possible explanation for the results is that existing baselines require a minimum token budget or additional hyperparameter tuning to operate effectively at an extremely low retention ratio such as 2%, whereas the proposed method can converge to a representative set quicker under such tight budgets. How can the authors establish that the gains reflect better information preservation rather than differences in suitability for aggressive compression? Additional experiments under milder compression would help clarify this distinction.
-
In my understanding, the proposed “set-aware” selection is a reconstruction-based pruning strategy, which has precedents in visual token pruning. The paper acknowledges that its objective instantiates generalized column subset selection and discusses closely related methods such as SPARE and ResPrune. Could the authors clarify the methodological novelty beyond adapting this objective to video compression and provide direct comparisons with these methods?
-
I still find the ablation studies insufficient to isolate the source of the improvements. Table 4 replaces several components with alternatives, but these comparisons do not fully establish which components drive the overall gains. Given that the framework contains four to five components, stronger controlled evidence is needed to show that reconstruction-based selection itself substantially improves the information retained by the tokens, rather than the gains primarily arising from jointly tuning the components.
To reconsider my score, I would encourage the authors to provide the evidence outlined above to substantiate the central claim of improved information preservation and clarify the methodological novelty.
Questions for authors
-
How does token set reconstruction relate to downstream information preservation? As far as I understand, a token may be reconstructable as a linear combination of anchors, but the LLM receives the retained or merged anchors without the reconstruction coefficients. What evidence shows that minimizing this projection error preserves information directly to the LLM? Could the authors examine the relationship between reconstruction error and downstream accuracy?
-
How substantial is the distinction from existing reconstruction-based methods?
-
How sensitive is the method to the configuration of temporal segmentation?
5 comments
Response to Weakness 2 and Question 2 (Novelty beyond Existing Reconstruction-based Pruning)
Thank you for the comment.
The key distinction between AVTC and other subspace reconstruction methods, such as SPARE and ResPrune, lies in how the benefit of selecting an anchor is evaluated. SPARE[1] and ResPrune[2] score each candidate using its own projection residual. This may favor a candidate that mainly adds its own unique feature over one that captures information missing from many other important tokens. Under a fixed budget, selecting the former spends a token slot mainly on reconstructing that candidate itself, rather than reducing the reconstruction errors of many other important tokens. This can leave useful visual cues insufficiently represented in the selected set. AVTC addresses this limitation by selecting the candidate that most reduces the total importance-weighted reconstruction error of the original token set, thereby improving feature coverage under a limited token budget.
SPARE and ResPrune are developed and evaluated for image token compression. We could not locate publicly available official implementations, which limits faithful reproduction and adaptation of their complete pipelines to video. We therefore compare their residual-based selection principle within our video compression framework. Specifically, we replace AVTC’s selection criterion with a Self Residual baseline while keeping all remaining components unchanged. As shown in Table 4, this replacement reduces average accuracy from 57.78 to 57.05 on LLaVA-OneVision-7B at 5% layer-average retention, a drop of 0.73 percentage points. This controlled comparison isolates the benefit of AVTC’s selection criterion. Beyond this ablation, the complete AVTC framework achieves state-of-the-art average accuracy among the evaluated methods across all three backbones.
[1] Jaeyeon Lee, Shunjie Wen, and Dong-Wan Choi. Moving beyond diversity: Visual token pruning as subspace reconstruction for efficient VLMs. arXiv preprint arXiv:2606.18681, 2026. URL https://arxiv.org/abs/2606.18681.
[2] Xu Li, Yi Zheng, Yuxuan Liang, Zhe Liu, Xiaolei Chen, Haotian Chen, Rui Zhu, and Xiangyang Xue. ResPrune: Text-conditioned subspace reconstruction for visual token pruning in large vision-language models. arXiv preprint arXiv:2603.21105, 2026. URL https://arxiv.org/abs/2603.21105.
Response to Weakness 3 (Component-wise Ablation Studies)
Thank you for the comment.
Table 4 reports controlled ablations of AVTC’s components. We summarize these results below to clarify their contributions within the framework. All experiments use LLaVA-OneVision-7B at 5% target layer-average retention. Each ablation modifies the indicated component, with the remaining pipeline components fixed and the layer-average token budget matched.
| Component | Ablation or alternative | Avg. accuracy (%) ↑ | Drop (pp) |
|---|---|---|---|
| Full framework | AVTC | 57.78 | — |
| Importance estimation | Incoming Attention | 57.12 | 0.66 |
| Anchor selection | Importance Top-k | 55.40 | 2.38 |
| Anchor selection | Self Residual | 57.05 | 0.73 |
| Anchor selection | Facility Location | 56.89 | 0.89 |
| Anchor selection | Without importance weighting in selection | 57.47 | 0.31 |
| Temporal segmentation | Global selection | 57.43 | 0.35 |
| Token merging | Without merging | 56.78 | 1.00 |
| Token merging | Without importance-weighted fusion | 57.38 | 0.40 |
| Token merging | Without selective-merging threshold | 57.55 | 0.23 |
| LLM-side refinement | Without attention calibration | 56.89 | 0.89 |
| LLM-side refinement | Without text-guided compression | 57.35 | 0.43 |
These experiments separately examine importance estimation, anchor selection, temporal segmentation, token merging, and LLM-side refinement. Reconstruction-based selection yields the largest observed gain among the evaluated component changes: replacing it with Importance Top-k reduces average accuracy by 2.38 percentage points, exceeding the drop from removing any individual supporting component.
Response to Weakness 1 (Information Preservation across Compression Budgets)
Thank you for raising this concern.
Our evaluation is not limited to extremely low retention ratios. Table 5 reports results at 15%, 20%, and 25% retention, covering commonly evaluated budgets of existing methods. AVTC achieves the highest average accuracy in eight of the nine model–budget settings across three backbones. Its advantage therefore extends beyond the extreme-compression regime.
We further assess feature preservation independently of downstream QA accuracy. At 10% retention, AVTC achieves the lowest Sinkhorn divergence and Gaussian W22 among the compared methods (Table 3). These measurements provide complementary evidence that its compressed representations better preserve the original visual feature distribution.
We additionally compare our experimental results with Unified Spatiotemporal Token Compression for Video-LLMs at Ultra-Low Retention (CVPR2026), a method specifically designed for extremely small token budgets. We recompute its average over the four shared benchmarks using its reported “w/o M” variant, which maintains the stated retention ratio throughout the LLM.
This comparison provides additional support that AVTC’s gains extend to a competing approach explicitly designed for ultra-low retention.