acceptodds
Under review as a conference paper at ICLR 2027

HEAT: Hierarchical Elastic Adaptive Token Sparsification via Depth Scaling

Abstract

Excessively long visual token sequences introduce substantial computational and memory overhead during the prefill stage of large vision-language models (LVLMs). Existing training-free visual token reduction methods alleviate this burden by pruning visual tokens at predefined LLM layers. However, fixed-depth pruning implicitly assumes that the same reduction depths are appropriate for all inputs, resulting in a mismatch between pruning depth and visual processing demand. To address this issue, we propose Hierarchical Elastic Adaptive Token Sparsification (HEAT), a training-free framework that scales pruning depth according to the evolving relevance state of each input. Specifically, HEAT introduces a cross-layer relevance monitor to quantify visual relevance concentration under the current token-retention budget, facilitating input-specific identification of appropriate pruning depths. Building on this, HEAT further develops a depth-adaptive threshold scheduler to adjust the pruning criterion according to both reduction ratio and model depth, supporting progressive visual token reduction at appropriate depths. Experiments across five LVLM backbones and a broad range of image, document, and video benchmarks demonstrate the effectiveness and generalizability of HEAT. With 66.7% of visual tokens removed, HEAT preserves 99.4% of full-token performance, reduces visual KV-cache memory by 66%, and accelerates LLM prefill by up to . Code is available at https://anonymous.4open.science/r/HEAT-C08E/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision Language Models, Visual Token Compression, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sLdXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HEAT, a training-free visual token pruning method that adaptively decide which layers visual tokens are pruned, instead of using a fixed layer. The target reduction ratio is split into M equal multiplicative stages. At a set of predefined monitored layers, similarity scores are computed between visual and text tokens to measure how much text-relevant mass the top-K tokens cover. Then this mass is compared against a threshold to decide whether to prune at the current layer or defer to the next stage. Tokens to keep are then selected using standard text-to-image attention during pruning. Any unfinished stages are merged and executed at a fixed fallback layer, so every input ends with the same final number of visual tokens. The method is evaluated on image, document, video and captioning benchmarks with Qwen2.5-VL-7B, InternVL2.5-8B and LLaVA-1.5-7B/13B models.

Critical strengths
  • The motivation makes sense. Pruning every input at the same fixed layer is a strong assumption, and letting the pruning depth depend on the input is more reasonable than a fixed schedule.
  • The proposed method achieves good results on the tested benchmarks and outperforms the compared baselines across most settings and budgets.
Critical weaknesses
  • The comparison with baselines might not be fair. All methods are matched on the final number of retained tokens, not on the actual compute. Since HEAT often defers pruning to deeper layers (Fig. 4 shows most MME/MMStar/POPE samples first prune at L12–L14), many more tokens pass through many more layers than in methods like FastV or DivPrune, which prune early or before the LLM. A fair comparison should match the total token-layer budget (or FLOPs). For example, with 500 tokens, a 33% target and 10 layers, the budget should be 500×33%×10 ≈ 1670 token-layers, however the pruning is scheduled. In Table 2, HEAT still uses more than half of the full prefill FLOPs at a "66.7% reduction". Also, the efficiency comparison only includes V2Drop and only on MME and DocVQA, where DocVQA is the most favorable case since most samples prune at L6. In addition, the tables label the setting "Average Retain 33.3% Tokens", which is misleading because the budget is actually the terminal token count. Finally, against a fixed progressive schedule with similar depth (12→14→16 in Table 4a), the gain is quite small (63.0 vs 63.0 MMStar, 2318.4 vs 2305.3 MME), which makes it unclear how much of the improvement comes from adaptivity rather than just pruning later.

  • The method is not as adaptive as described. Even though it's called adaptive depth scaling, the monitored layers (L6–L14), the fallback layer (L16), the number of stages (M=3) and the per-stage retention ratio are all predefined. Each check can only remove a fixed fraction of tokens or nothing at all. I had hoped to see a method that can prune at any layer and by any amount, as long as the overall budget is met. On top of that, the temperature, the threshold endpoints for each budget, and the decay schedule are all hand-tuned, and LLaVA-1.5-7B even needs its own separate schedule (Table 5). It's not clear how these values were selected or whether they were tuned on the evaluation benchmarks.

  • The experiments use older models. The main experiments are on Qwen2.5-VL-7B and InternVL2.5-8B, and the appendix adds LLaVA-1.5, which is pretty outdated now. InternVL3.5 only appears in the appendix and is only compared against FastV. Newer models like the Qwen3-VL and Qwen3.5 series are available, but none are evaluated. I understand prior works used these models, but most of the baselines are open-sourced, and the authors could adapt them to newer models. For a 2027 submission, this weakens the generality claims.

Questions for authors
  1. Can you report results for all methods under a matched total compute budget (token-layers or FLOPs), or at least report FLOPs/latency for all baselines across all benchmarks rather than only V2Drop on MME and DocVQA?
  2. How do the baselines perform if their pruning layers are moved to depths comparable to HEAT's (e.g., FastV/PyramidDrop pruning at L12–L16)?
  3. How were the monitored layers, fallback layer, temperature and threshold endpoints chosen? Were they tuned on the evaluation benchmarks? How sensitive is the method to the choice of monitored layers?
  4. Can the method be extended to prune at arbitrary layers and by arbitrary amounts under a total budget constraint?
  5. How does HEAT work with batched inference where different samples prune at different layers? Have you tried integrating it into vLLM or SGLang, and what are the throughput and memory results on larger models?
2 comments
AuthorsOpenReview ↗

Discussion is highly welcome

Dear Reviewer sLdX,
Thanks for your insightful comments and suggestions to strengthen our paper. In our response, we addressed each of your questions with detailed explanations and new experimental results. As the discussion phase closes on November 18, we would like to check whether all your concerns have been well addressed.
For any questions about our paper, we are happy to provide any further details. Thank you for your time!
Best,
Authors of 422

Reviewer_sLdXOpenReview ↗

Response to authors

Thank you for providing the additional results. I appreciate the clarification of the terms used in the paper and the results for matched FLOPs and matched layer average retention. Although the performance gap with baselines largely shrinked, HEAT still achieved some gains for both models. Thus my biggest concern is largely addressed, and I think this point is also shared by other reviewers too. I also appreciate the additional results with newer Qwen models, different adaptive pruning schedules and the vllm inference latency measures, I think those results make the overall paper stronger. Therefore I'm increasing my overall score to 3.

Reviewer_g3CSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HEAT, a training-free framework for input-adaptive visual token pruning. To address the mismatch between fixed pruning depths and input-specific processing requirements, HEAT decomposes the target final token budget into multiple stages with equal retention ratios. A cross-layer relevance monitor (CRM) measures text-conditioned relevance coverage under each stage's token budget, while a depth-adaptive threshold scheduler (DTS) uses thresholds that decrease with depth to determine when to prune. Subsequent pruning decisions are reassessed after each reduction. The evaluation spans five models and image, document, and video understanding tasks.

Critical strengths

At matched final visual token retention ratios, HEAT demonstrates competitive performance across multiple tasks on five LLM models from three families. Table 2 also reports algorithmic overhead below 3 ms in the evaluated setting.

Critical weaknesses
  1. This paper offers limited innovation compared to existing adaptive token pruning work. AutoPrune has pointed out the mismatch between fixed pruning arrangements and different input processing requirements, and proposed an input-dependent cross-layer retention strategy [1]. In addition, input-adaptive selection of pruning layers has also been studied [2,3,4]. HEAT's incremental innovation lies in decomposing the final compression budget into multiple stages with the same retention ratio, and combining budget-related CRM indicators and depth-varying thresholds to re-determine the pruning timing stage by stage. However, the novelty of this proportional budget decomposition itself is limited, and existing experiments have not fully isolated the additional benefits of multi-stage repeated adaptation compared to simpler input-dependent scheduling strategies.

  2. The performance advantage at the same computational cost has not yet been fully validated. The main experiment primarily compares the final visual token retention rate, but pruning depth affects the average computational cost per layer, and the same retention rate does not necessarily mean the same computational budget. Therefore, the accuracy advantage in the main table cannot fully exclude the impact of computational cost differences. Furthermore, HEAT fixes the pruning rate at each stage and only adaptively selects pruning layers; it is unclear whether a fixed proportional allocation limits the performance-computational cost tradeoff. Comparing this design choice at the same cost with strategies that jointly adjust pruning layers and stage pruning rates will help evaluate this design choice.

[1] Wang H, Xu Y, Xu Z, et al. Each Complexity Deserves a Pruning Policy[J]. Advances in Neural Information Processing Systems, 2026, 38: 118209-118228.

[2] Choi J, Lee S, Kim J, et al. DocPrune: Efficient Document Question Answering via Background, Question, and Comprehension-aware Token Pruning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2026: 3543–3552.

[3] Li D, Yang Z, Zhang X, et al. Todre: Effective visual token pruning via token diversity and task relevance[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026: 635-653.

[4] Lee Y, Min K, Kim Y. ERASE: Eliminating Redundant Visual Tokens via Adaptive Two-Stage Token Pruning[J]. arXiv preprint arXiv:2605.09982, 2026.

Questions for authors
  1. Compare HEAT with the main baselines at matched layer-averaged number of retained visual tokens. Under the same computational budget, assess equal-ratio stage allocation against unequal allocations or policies that jointly adjust pruning layers and stage-wise retention ratios.
  2. Specify the backbone used in Table 2 and report efficiency results for both Qwen2.5-VL-7B and InternVL2.5-8B, the two primary models evaluated in the main experiments.
  3. Clarify HEAT's technical advances over existing input-adaptive token pruning methods, particularly AutoPrune and prior work on input-adaptive layer selection. Controlled comparisons with simpler input-dependent scheduling policies, using the same token selector and stage budgets at comparable computational cost, would help isolate the additional benefit of repeated adaptation across stages.
  4. Explain why the same absolute monitoring layers (L6, L8, L10, L12, and L14) and fallback layer (L16) are used across backbones with different numbers of LLM layers.
1 comment
AuthorsOpenReview ↗

Discussion is highly welcome

Dear Reviewer g3CS,
Thanks for your insightful comments and suggestions to strengthen our paper. In our response, we addressed each of your questions with detailed explanations and new experimental results. As the discussion phase closes on November 18, we would like to check whether all your concerns have been well addressed.
For any questions about our paper, we are happy to provide any further details. Thank you for your time!
Best,
Authors of 422

Reviewer_eqVyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes HEAT, a training-free framework for visual token pruning in large vision-language models. Given fixed stage-wise token budgets, HEAT adapts pruning depths to each input using text–vision relevance concentration and a depth-dependent threshold schedule. Experiments across five backbones and image, document, video, and captioning tasks demonstrate competitive accuracy retention, with reported LLM-prefill speedups of up to 1.59×.

Critical strengths
  1. Useful focus on input-dependent pruning depth. The shuffled-depth control provides evidence that matching pruning depths to individual inputs matters beyond simply varying the depth distribution.
  2. Broad empirical coverage. Evaluation across multiple architectures and task types supports the method’s applicability. The reported efficiency measurements include monitoring and pruning overhead.
Critical weaknesses
  1. Limited methodological novelty and reliance on heuristics. The method combines progressive pruning with a cosine-similarity-based concentration score and a manually specified threshold schedule. The stage count, candidate layers, temperature, and budget-dependent thresholds are heuristic choices, with a separate configuration for LLaVA-1.5-7B. The conceptual advance over existing adaptive and progressive pruning methods appears incremental.
  2. The accuracy–efficiency advantage is not fully established. Most comparisons match the final token budget, although pruning later can consume substantially more computation. Table 2 provides useful efficiency results against V2Drop, but broader comparisons at matched latency or FLOPs are needed to establish whether HEAT offers a better accuracy–efficiency trade-off than well-tuned fixed schedules.
Questions for authors
  1. At matched measured prefill latency or FLOPs, how does HEAT compare with tuned fixed-depth and fixed progressive schedules?
  2. How were the threshold endpoints and other scheduling parameters selected? Were any evaluation benchmarks used for tuning, and how well does one unchanged configuration transfer?
  3. CRM measures coverage using cosine-similarity scores, whereas pruning uses attention scores. Does high CRM coverage reliably predict low pruning-induced error for the actual attention-selected subset?
1 comment
AuthorsOpenReview ↗

Discussion is highly welcome

Dear Reviewer eqVy,
Thanks for your insightful comments and suggestions to strengthen our paper. In our response, we addressed each of your questions with detailed explanations and new experimental results. As the discussion phase closes on November 18, we would like to check whether all your concerns have been well addressed.
For any questions about our paper, we are happy to provide any further details. Thank you for your time!
Best,
Authors of 422

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.