acceptodds
Withdrawn from ICLR 2027

AVTC: Aggressive Visual Token Compression via Set-Aware Selection for Video Large Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual token compression offers a practical way to reduce the inference cost of video large language models (Video-LLMs), but preserving model performance under aggressive compression remains challenging. Existing methods often select tokens by individual importance or prune them based on pairwise similarity, which can lead to redundant selections and incomplete coverage of the original visual content under tight budgets. To address this limitation, we introduce AVTC, a training-free framework that performs set-aware selection by evaluating how well the selected tokens jointly reconstruct the original visual features. Specifically, AVTC scores each candidate by how much its addition reduces the total importance-weighted projection error of all original tokens onto the subspace spanned by the selected anchors. This criterion favors tokens that contribute important information not yet captured by the current selection. Experiments on three Video-LLMs and four benchmarks show that AVTC achieves the highest average accuracy among the compared methods at layer-average token retention ratios of 2%, 5%, and 10%. On LLaVA-OneVision-7B, AVTC retains 94.6% of the uncompressed model’s average accuracy at just 2% token retention. At 10% retention, it achieves 100.2% of the uncompressed model’s average accuracy while delivering a 6.6 prefill speedup and a 2.0 speedup in time to first token (TTFT). Code is available at https://anonymous.4open.science/r/AVTC-8F87.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Video Large Language Models, Token Compression, Video Understanding, Inference Efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xRE5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training-free visual token compression framework for Video-LLMs under aggressive token budgets. Its core component is set-aware anchor selection: each token is selected according to how much its addition reduces the total importance-weighted projection error of the original visual features onto the selected anchor subspace. This is a consequence-aware measurement rather than previous compression methods that is based on a score prior. The framework further incorporates segment-wise selection, selective token merging, proportional attention calibration, and text-guided compression within the LLM. The author reports the highest average accuracy among the evaluated methods at layer-average token retention ratios of 2%, 5%, and 10%.

Critical strengths
  1. The paper identifies a meaningful limitation of individual importance and pairwise similarity criteria. Existing methods do not directly measure how the selected tokens jointly represent the original features.

  2. The evaluation covers three backbones, four benchmarks, and several retention ratios. AVTC achieves the best average accuracy in the main comparisons, with larger advantages over competing methods as the token budget becomes tighter. Although FlashVID is also competitive for Qwen-2.5-VL.

Critical weaknesses
  1. A major concern is whether the proposed method truly improves information preservation. One possible explanation for the results is that existing baselines require a minimum token budget or additional hyperparameter tuning to operate effectively at an extremely low retention ratio such as 2%, whereas the proposed method can converge to a representative set quicker under such tight budgets. How can the authors establish that the gains reflect better information preservation rather than differences in suitability for aggressive compression? Additional experiments under milder compression would help clarify this distinction.

  2. In my understanding, the proposed “set-aware” selection is a reconstruction-based pruning strategy, which has precedents in visual token pruning. The paper acknowledges that its objective instantiates generalized column subset selection and discusses closely related methods such as SPARE and ResPrune. Could the authors clarify the methodological novelty beyond adapting this objective to video compression and provide direct comparisons with these methods?

  3. I still find the ablation studies insufficient to isolate the source of the improvements. Table 4 replaces several components with alternatives, but these comparisons do not fully establish which components drive the overall gains. Given that the framework contains four to five components, stronger controlled evidence is needed to show that reconstruction-based selection itself substantially improves the information retained by the tokens, rather than the gains primarily arising from jointly tuning the components.

To reconsider my score, I would encourage the authors to provide the evidence outlined above to substantiate the central claim of improved information preservation and clarify the methodological novelty.

Questions for authors
  1. How does token set reconstruction relate to downstream information preservation? As far as I understand, a token may be reconstructable as a linear combination of anchors, but the LLM receives the retained or merged anchors without the reconstruction coefficients. What evidence shows that minimizing this projection error preserves information directly to the LLM? Could the authors examine the relationship between reconstruction error and downstream accuracy?

  2. How substantial is the distinction from existing reconstruction-based methods?

  3. How sensitive is the method to the configuration of temporal segmentation?

5 comments
AuthorsOpenReview ↗

Response to Weakness 1 (Information Preservation across Compression Budgets)

Thank you for raising this concern.

Our evaluation is not limited to extremely low retention ratios. Table 5 reports results at 15%, 20%, and 25% retention, covering commonly evaluated budgets of existing methods. AVTC achieves the highest average accuracy in eight of the nine model–budget settings across three backbones. Its advantage therefore extends beyond the extreme-compression regime.

We further assess feature preservation independently of downstream QA accuracy. At 10% retention, AVTC achieves the lowest Sinkhorn divergence and Gaussian W22W_2^2 among the compared methods (Table 3). These measurements provide complementary evidence that its compressed representations better preserve the original visual feature distribution.

We additionally compare our experimental results with Unified Spatiotemporal Token Compression for Video-LLMs at Ultra-Low Retention (CVPR2026), a method specifically designed for extremely small token budgets. We recompute its average over the four shared benchmarks using its reported “w/o M” variant, which maintains the stated retention ratio throughout the LLM.

RetentionUnified Spatiotemporal Compression (w/o M)AVTCDifference
2%53.1055.26+2.16
5%56.1557.78+1.63
10%57.9858.55+0.57

This comparison provides additional support that AVTC’s gains extend to a competing approach explicitly designed for ultra-low retention.

AuthorsOpenReview ↗

Response to Weakness 2 and Question 2 (Novelty beyond Existing Reconstruction-based Pruning)

Thank you for the comment.

The key distinction between AVTC and other subspace reconstruction methods, such as SPARE and ResPrune, lies in how the benefit of selecting an anchor is evaluated. SPARE[1] and ResPrune[2] score each candidate using its own projection residual. This may favor a candidate that mainly adds its own unique feature over one that captures information missing from many other important tokens. Under a fixed budget, selecting the former spends a token slot mainly on reconstructing that candidate itself, rather than reducing the reconstruction errors of many other important tokens. This can leave useful visual cues insufficiently represented in the selected set. AVTC addresses this limitation by selecting the candidate that most reduces the total importance-weighted reconstruction error of the original token set, thereby improving feature coverage under a limited token budget.

SPARE and ResPrune are developed and evaluated for image token compression. We could not locate publicly available official implementations, which limits faithful reproduction and adaptation of their complete pipelines to video. We therefore compare their residual-based selection principle within our video compression framework. Specifically, we replace AVTC’s selection criterion with a Self Residual baseline while keeping all remaining components unchanged. As shown in Table 4, this replacement reduces average accuracy from 57.78 to 57.05 on LLaVA-OneVision-7B at 5% layer-average retention, a drop of 0.73 percentage points. This controlled comparison isolates the benefit of AVTC’s selection criterion. Beyond this ablation, the complete AVTC framework achieves state-of-the-art average accuracy among the evaluated methods across all three backbones.

[1] Jaeyeon Lee, Shunjie Wen, and Dong-Wan Choi. Moving beyond diversity: Visual token pruning as subspace reconstruction for efficient VLMs. arXiv preprint arXiv:2606.18681, 2026. URL https://arxiv.org/abs/2606.18681.

[2] Xu Li, Yi Zheng, Yuxuan Liang, Zhe Liu, Xiaolei Chen, Haotian Chen, Rui Zhu, and Xiangyang Xue. ResPrune: Text-conditioned subspace reconstruction for visual token pruning in large vision-language models. arXiv preprint arXiv:2603.21105, 2026. URL https://arxiv.org/abs/2603.21105.

AuthorsOpenReview ↗

Response to Weakness 3 (Component-wise Ablation Studies)

Thank you for the comment.

Table 4 reports controlled ablations of AVTC’s components. We summarize these results below to clarify their contributions within the framework. All experiments use LLaVA-OneVision-7B at 5% target layer-average retention. Each ablation modifies the indicated component, with the remaining pipeline components fixed and the layer-average token budget matched.

ComponentAblation or alternativeAvg. accuracy (%) ↑Drop (pp)
Full frameworkAVTC57.78—
Importance estimationIncoming Attention57.120.66
Anchor selectionImportance Top-k55.402.38
Anchor selectionSelf Residual57.050.73
Anchor selectionFacility Location56.890.89
Anchor selectionWithout importance weighting in selection57.470.31
Temporal segmentationGlobal selection57.430.35
Token mergingWithout merging56.781.00
Token mergingWithout importance-weighted fusion57.380.40
Token mergingWithout selective-merging threshold57.550.23
LLM-side refinementWithout attention calibration56.890.89
LLM-side refinementWithout text-guided compression57.350.43

These experiments separately examine importance estimation, anchor selection, temporal segmentation, token merging, and LLM-side refinement. Reconstruction-based selection yields the largest observed gain among the evaluated component changes: replacing it with Importance Top-k reduces average accuracy by 2.38 percentage points, exceeding the drop from removing any individual supporting component.

Reviewer_HAwWOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

AVTC compresses video tokens using importance-weighted set reconstruction. It greedily selects anchors by their reduction of the projection error of the original token set, performs selection within temporal segments under a shared budget, and selectively merges discarded features. Proportional attention and text-guided compression provide additional refinement inside the LLM. Evaluation covers three backbones and four benchmarks, including very low layer-average retention ratios, component ablations, feature-distribution diagnostics, and measured inference latency.

Critical strengths

AVTC reports consistent improvements under aggressive compression, exceeding the strongest evaluated baseline by 1.80–2.79 percentage points in average QA accuracy across three backbones at 2% layer-average retention.

Critical weaknesses
  1. The methodological novelty is incremental, and the selection criterion’s benefit is not fully isolated. The reconstruction objective and greedy rule instantiate established generalized column subset selection; proportional attention and text-guided pruning also follow prior methods. In Table 4, unweighted AVTC scores 57.15 versus 57.05 for Self Residual. Compare equally weighted selection criteria with segmentation, merging, and LLM-side processing fixed to establish the value of the proposed adaptation.

  2. Selection ignores redundancy across temporal segments. Eq. (14) restricts reconstruction to each eight-frame segment, so features selected in one segment do not reduce the gains of duplicates elsewhere. Compare with the existing global-selection variant on repeated/interleaved scenes and shifted segment boundaries. Extend the current 32/64-frame evaluation up to 768 frames with intermediate settings to assess accuracy and computational scaling.

  3. Extend accuracy–efficiency evaluation to Qwen3-VL and test deployment with vLLM, comparing against a representative subset of baselines such as VidCom2 [1], MMG-Vid [2], CDPruner [3], and other relevant methods. Compare accuracy, compression-inclusive TTFT, and peak memory under matched hardware, inputs, and comparable budgets.

  4. Generalization is limited to 7B dense models and multiple-choice QA. Include larger dense and MoE backbones, such as Qwen3.5-27B and Qwen3.5-35B-A3B. Temporal grounding and video captioning on suitable benchmarks are also needed to assess whether compression preserves event boundaries and descriptive detail beyond answer selection.

[1] Xuyang Liu, Yiyu Wang, Junpeng Ma, and Linfeng Zhang. “Video Compression Commander: Plug-and-Play Inference Acceleration for Video Large Language Models.” EMNLP, 2025.

[2] Junpeng Ma, Qizhe Zhang, Ming Lu, Zhibin Wang, Qiang Zhou, Jun Song, and Shanghang Zhang. “MMG-Vid: Maximizing Marginal Gains at Segment-level and Token-level for Efficient Video LLMs.” AAAI, 2026.

[3] Qizhe Zhang, Mengzhen Liu, Lichen Li, Ming Lu, Yuan Zhang, Junwen Pan, Qi She, and Shanghang Zhang. Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs. NeurIPS, 2025.

Questions for authors

See weeknesses.

4 comments
AuthorsOpenReview ↗

Response to Comment 1 (Selection Criterion and Controlled Ablations)

Thank you for the comment.

Our anchor-selection ablations replace only the selection criterion, keeping temporal segmentation, token merging, and LLM-side processing fixed. Importance Top-k selects individually important tokens. Self Residual ranks candidates by their own projection residual. Facility Location measures coverage through each token's best-matching anchor. AVTC instead evaluates how much adding a candidate reduces the joint reconstruction error of the original token set:

Δ(j∣A)=∑iwi⟨ri,rj⟩2∥rj∥22.\Delta(j\mid\mathcal A)=\sum_i w_i\frac{\langle r_i,r_j\rangle^2}{\|r_j\|_2^2}.

Thus, a candidate is evaluated by its contribution to reconstructing the token set, including other tokens whose residuals align with its new subspace direction. The selected anchors jointly represent the original features through linear combinations.

The 57.15 result cited in the review corresponds to Self Residual without importance weighting, rather than unweighted AVTC. This entry has been corrected in the revised manuscript. The comparison between 57.15 and 57.05 therefore concerns variants of Self Residual; it does not measure the difference between AVTC's joint-reconstruction criterion and Self Residual.

Table R1 combines the main anchor-selection ablation with the additional anchor-selection experiments from the supplementary material. Scores are mean accuracy (%) over EgoSchema, LongVideoBench, MVBench, and Video-MME without subtitles. All retention percentages denote layer-average visual-token retention.

Table R1. Anchor-selection ablations with the rest of the pipeline fixed.

Selection criterionLLaVA-OneVision-7B, 2%LLaVA-OneVision-7B, 5%LLaVA-OneVision-7B, 10%Qwen3.5-27B, 5%
Importance Top-k52.5855.4057.2364.17
Self Residual54.5757.0557.9465.27
Facility Location55.0056.8958.2164.86
AVTC55.2657.7858.5565.75

AVTC exceeds Self Residual by 0.69, 0.73, and 0.61 points on LLaVA-OneVision-7B at 2%, 5%, and 10% retention, respectively, and by 0.48 points on Qwen3.5-27B at 5% retention. It also outperforms Importance Top-k and Facility Location in all four settings. The benefit therefore appears across compression budgets and extends to a larger dense backbone.

The reconstruction-error measurements provide a complementary comparison under the same 5% retention setting on LLaVA-OneVision-7B:

Table R2. Weighted reconstruction error and downstream QA accuracy.

Selection criterionReconstruction error ↓Mean QA accuracy (%) ↑
Importance Top-k0.1283355.40
Self Residual0.0920957.05
Facility Location0.0915556.89
AVTC0.0771257.78

AVTC reduces reconstruction error by 16.26% relative to Self Residual and 15.76% relative to Facility Location, accompanied by accuracy gains of 0.73 and 0.89 points. These measurements connect the selection criterion's reconstruction benefit with downstream performance while holding the remaining pipeline components fixed.

AuthorsOpenReview ↗

Response to Comment 2 (Evaluation with 768 Input Frames)

Thank you for the suggestion to evaluate longer video inputs.

We evaluate Qwen3-VL-8B-Instruct with 768 input frames on all four video-QA benchmarks at 2%, 5%, and 10% retention. This setting uses eight frames per temporal segment, resulting in 96 segments sharing a single anchor-token budget.

Table R3. Video-QA accuracy (%) with 768 input frames.

RetentionMethodEgoSchemaLongVideoBenchMVBenchVideo-MMEMean
100%Uncompressed71.0866.9468.5372.3769.73
2%FlashVID65.5157.0758.0859.8160.12
2%GSTEP66.9460.4362.6564.3363.59
2%AVTC68.5063.1364.7066.5965.73
5%FlashVID66.7759.8463.6264.2263.61
5%GSTEP68.6363.2864.2067.4165.88
5%AVTC70.0765.0067.2568.4167.68
10%FlashVID68.2862.3066.0368.0066.15
10%GSTEP69.9565.2264.9069.0067.27
10%AVTC70.6265.3768.0370.5268.63

AVTC achieves the highest accuracy on every benchmark at all three retention ratios. Its mean accuracy exceeds the strongest compared baseline by 2.14, 1.80, and 1.36 points at 2%, 5%, and 10% retention, respectively. At 10% retention, AVTC achieves 68.63 mean accuracy compared with 69.73 for the uncompressed model, retaining 98.4% of its mean accuracy with 768 input frames.

AuthorsOpenReview ↗

Response to Comment 3 (Results on Qwen3-VL)

Thank you for the suggestion to evaluate Qwen3-VL.

We compare AVTC with VidCom2, FlashVID, and GSTEP on Qwen3-VL-8B-Instruct, using 32 uniformly sampled frames and matched nominal layer-average retention budgets. At the time of evaluation, MMG-Vid and CDPruner did not provide publicly available implementations for Qwen backbones. We therefore compare with baselines that have available Qwen adaptations. Table R4 reports mean accuracy across the four QA benchmarks.

Table R4. Mean video-QA accuracy (%) on Qwen3-VL-8B-Instruct.

Method2% retention5% retention10% retention
VidCom249.0654.1858.09
FlashVID52.1757.8561.69
GSTEP57.2159.8962.18
AVTC59.8062.5364.02

The uncompressed model achieves 65.68 mean accuracy. AVTC outperforms the strongest compared baseline by 2.59, 2.64, and 1.84 points at 2%, 5%, and 10% retention, respectively, preserving 91.0%, 95.2%, and 97.5% of the uncompressed model's mean accuracy. Together with the 768-frame results in Table R3, these comparisons demonstrate the effectiveness of AVTC on Qwen3-VL under both standard and substantially longer visual inputs.

AuthorsOpenReview ↗

Response to Comment 4 (Larger Dense/MoE Backbones, Temporal Grounding, and Video Captioning)

Thank you for the suggestion to evaluate larger backbones and tasks beyond multiple-choice QA.

Our evaluation includes Qwen3.5-27B, a larger dense backbone, and Qwen3.5-35B-A3B, an MoE backbone with 35B total and 3B activated parameters. Both use 32 uniformly sampled frames. Table R5 reports mean accuracy over the same four QA benchmarks at matched nominal layer-average retention budgets.

Table R5. Mean video-QA accuracy (%) on larger dense and MoE backbones.

RetentionMethodQwen3.5-27BQwen3.5-35B-A3B
100%Uncompressed71.2369.20
2%VidCom253.4151.87
2%FlashVID57.6255.25
2%GSTEP60.5358.32
2%AVTC61.7360.91
5%VidCom258.2257.38
5%FlashVID62.4660.54
5%GSTEP63.8061.74
5%AVTC65.7563.83
10%VidCom261.8261.06
10%FlashVID66.1564.60
10%GSTEP66.0665.07
10%AVTC67.5466.32

AVTC achieves the highest mean accuracy among the compared methods in all six model–budget settings. Its gains over the strongest compared baseline range from 1.20 to 1.95 points on Qwen3.5-27B and from 1.25 to 2.59 points on Qwen3.5-35B-A3B, demonstrating that the gains extend to both larger dense and MoE architectures.

Beyond multiple-choice QA, we evaluate temporal grounding on Charades-STA and free-form video captioning on VATEX with both backbones. Charades-STA mIoU measures temporal localization quality, while VATEX CIDEr evaluates generated descriptions against reference captions.

Table R6. Temporal grounding and video captioning. Higher is better; VATEX CIDEr is reported ×100.

RetentionMethod27B: Charades mIoU27B: VATEX CIDEr35B-A3B: Charades mIoU35B-A3B: VATEX CIDEr
100%Uncompressed62.0064.9561.6863.46
2%VidCom231.5818.0430.4416.39
2%FlashVID36.8745.9135.9141.72
2%GSTEP41.6252.4536.9847.81
2%AVTC42.9554.7439.3550.66
5%VidCom239.0734.1039.4136.37
5%FlashVID45.3057.5847.7452.11
5%GSTEP49.5859.8646.7956.15
5%AVTC52.1560.6749.4857.44
10%VidCom246.9347.7448.3450.01
10%FlashVID54.7263.0053.9659.19
10%GSTEP56.0062.7152.7760.66
10%AVTC56.8864.4455.2860.90

AVTC achieves the highest mIoU and CIDEr in all six model–budget settings among the compared compression methods. On Charades-STA, its gains over the strongest baseline range from 0.88 to 2.57 mIoU points on Qwen3.5-27B and from 1.32 to 2.37 points on Qwen3.5-35B-A3B. On VATEX, the CIDEr gains at 2% retention are 2.29 and 2.85 points, respectively. At 10% retention, AVTC achieves 64.44 CIDEr on the dense backbone, compared with 64.95 for the uncompressed model. These results demonstrate that the benefits extend to temporal localization and descriptive generation, beyond answer selection.

Reviewer_xHBdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AVTC, a training-free visual token compression framework for Video-LLMs. The method introduces a set-aware selection strategy that greedily selects visual tokens based on their contribution to reducing the importance-weighted reconstruction error of the original visual features. It further incorporates selective token merging, attention calibration, and text-guided compression within the LLM. Experiments on three Video-LLMs and four benchmarks demonstrate strong performance under aggressive token compression, together with improvements in inference efficiency.

Critical strengths
  1. The paper is generally well organized.
  2. The authors conduct comparisons across multiple Video-LLMs, benchmarks, and compression ratios. The ablation studies and efficiency analysis also help demonstrate the effectiveness of the proposed framework.
Critical weaknesses
  1. The proposed set-aware selection is closely related to existing subspace reconstruction and column subset selection approaches, including SPARE[1] and ResPrune[2]. Moreover, several components, such as token merging and attention calibration, are adapted from existing methods. The authors should more clearly explain the methodological differences and key technical contributions beyond these established approaches.
  2. The paper uses layer-average token retention ratios for comparison. However, different methods compress tokens at different stages, potentially resulting in different computational costs and information availability across layers. Reporting the actual token counts and per-layer retention distributions would make the comparisons more convincing.
  3. AVTC uses diagonal self-attention scores as a proxy for visual token importance. However, high self-attention does not necessarily indicate relevance to downstream video understanding.
  4. Attention mechanisms may suffer from long-range attenuation in lengthy videos. Can AVTC reliably preserve important visual evidence across long temporal sequences? Further visual analysis would strengthen the paper.

Ref: [1] Moving Beyond Diversity: Visual Token Pruning as Subspace Reconstruction for Efficient VLMs. [2] ResPrune: Text-Conditioned Subspace Reconstruction for Visual Token Pruning in Large Vision-Language Models.

Questions for authors

Please see the Critical Weaknesses section.

4 comments
AuthorsOpenReview ↗

Response to Weakness 1 (Key Technical Contributions beyond Existing Approaches)

Thank you for the comment.

The key distinction between AVTC and other subspace reconstruction methods, such as SPARE and ResPrune, lies in how the benefit of selecting an anchor token is evaluated. SPARE[1] and ResPrune[2] score each candidate using its own projection residual. This may favor a candidate that mainly adds its own unique feature over one that captures information missing from many other important tokens. Under a fixed budget, selecting the former spends a token slot mainly on reconstructing that candidate itself, rather than reducing the reconstruction errors of many other important tokens. This can leave useful visual cues insufficiently represented in the selected set. AVTC addresses this limitation by selecting the candidate that most reduces the total importance-weighted reconstruction error of the original token set, thereby improving feature coverage under a limited token budget.

SPARE and ResPrune are developed and evaluated for image token compression. We could not locate publicly available official implementations, which limits faithful reproduction and adaptation of their complete pipelines to video. We therefore compare their residual-based selection principle within our video compression framework. Specifically, we replace AVTC’s selection criterion with a Self Residual baseline while keeping all remaining components unchanged. As shown in Table 4, this replacement reduces average accuracy from 57.78 to 57.05 on LLaVA-OneVision-7B at 5% layer-average retention, a drop of 0.73 percentage points. This controlled comparison isolates the benefit of AVTC’s selection criterion. Beyond this ablation, the complete AVTC framework achieves state-of-the-art average accuracy among the evaluated methods across all three backbones.

[1] Jaeyeon Lee, Shunjie Wen, and Dong-Wan Choi. Moving beyond diversity: Visual token pruning as subspace reconstruction for efficient VLMs. arXiv preprint arXiv:2606.18681, 2026. URL https://arxiv.org/abs/2606.18681.

[2] Xu Li, Yi Zheng, Yuxuan Liang, Zhe Liu, Xiaolei Chen, Haotian Chen, Rui Zhu, and Xiangyang Xue. ResPrune: Text-conditioned subspace reconstruction for visual token pruning in large vision-language models. arXiv preprint arXiv:2603.21105, 2026. URL https://arxiv.org/abs/2603.21105.

AuthorsOpenReview ↗

Response to Weakness 2 (Token Budget Alignment Method)

Thank you for the suggestion.

Matching the average number of visual tokens across LLM layers is an established evaluation practice adopted by multiple token-compression methods, including FlashVID [1], TwigVLM [2], and PACT [3]. We follow this practice to compare methods that compress tokens at different stages.

To make these comparisons more transparent, we additionally report the actual layer-wise visual token counts, layer-average counts, realized retention ratios, average accuracy, and inference latency on LLaVA-OneVision-7B under the nominal 10% layer-average retention setting. Vanilla provides the full-token reference. Fractional token counts denote averages across evaluated samples. Average accuracy is computed over four benchmarks, and all latencies are in milliseconds. Prefill latency includes compression overhead.

MethodVisual tokens by LLM layerLayer-average countActual average retentionAvg. accuracy ↑Prefill latency ↓TTFT ↓
VanillaLayers 1–28: 62726272.00100%58.43578.6961.1
UniCompLayers 1–28: 629.25629.2510.033%58.03203.2585.3
GSTEPLayers 1–28: 627627.009.997%56.63201.0579.6
FlashVIDLayers 1–20: 814.875; 21–28: 245652.0510.396%57.72109.4488.1
HoliTomLayers 1–18: 778.875; 19–28: 389639.6310.198%57.97172.1550.6
AVTCLayers 1–20: 784; 21–28: 236627.4310.004%58.5587.7469.6

These results explicitly show the token counts at each compression stage and their associated accuracy–efficiency trade-offs. AVTC achieves the highest average accuracy among the compared compression methods, together with the lowest measured compression-inclusive prefill latency and TTFT.

[1] Ziyang Fan et al. “FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token Merging.” ICLR, 2026.

[2] Zhenwei Shao et al. “Growing a Twig to Accelerate Large Vision-Language Models.” ICCV, 2025.

[3] Mohamed Dhouib, Davide Buscaldi, Sonia Vanier, and Aymen Shabou. “PACT: Pruning and Clustering-Based Token Reduction for Faster Visual Language Models.” CVPR, 2025.

AuthorsOpenReview ↗

Response to Weakness 3 (Diagonal Attention as an Importance Proxy)

Thank you for raising this point.

AVTC uses diagonal self-attention from the vision encoder as a lightweight, query-independent proxy for visual importance within each frame. Although this estimate may not coincide with relevance to a specific question, it provides an inexpensive signal for prioritizing visual content. It requires neither an additional encoder forward pass nor a trainable importance estimator or dedicated [CLS] token. AVTC combines this importance estimate with reconstruction-based feature coverage. The scores weight the reconstruction errors of the original tokens, while selection depends on each candidate’s contribution to reducing the total weighted error.

Table 4 provides empirical support for this choice. Replacing diagonal attention with Incoming Attention reduces average accuracy from 57.78 to 57.12 on LLaVA-OneVision-7B at 5% retention. Removing importance weighting from the selection objective while retaining importance-weighted merging yields 57.47. The first comparison supports diagonal attention over the evaluated alternative within our framework. The second isolates the benefit of importance weighting in the reconstruction objective.

AuthorsOpenReview ↗

Response to Weakness 4 (Preserving Temporally Distant Visual Evidence)

Thank you for highlighting this concern.

AVTC’s initial anchor selection is driven by visual feature reconstruction gains rather than LLM attention scores. Reconstruction is evaluated within each temporal segment under a shared token budget. Consequently, early visual evidence can contribute to anchor selection through its reconstruction benefit, without requiring strong attention from a temporally distant question.

The newly added Figure 4 in Appendix A.5 provides qualitative evidence from a 15-minute LongVideoBench video. The question asks which of five characters appears first. We visualize AVTC’s selected anchors across all 32 sampled frames and highlight two early evidence frames at 145.9 s and 204.3 s, both within the first quarter of the video. These frames receive 12 and 19 anchors, respectively. At 5% layer-average retention, AVTC answers correctly, whereas all six compared compression baselines (FlashVID, GSTEP, UniComp, HoliTom, VisionZip, and FastVID) answer incorrectly. This example illustrates successful preservation and use of early visual evidence for temporal-order reasoning under aggressive compression.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.