- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes HEAT, a training-free visual token pruning method that adaptively decide which layers visual tokens are pruned, instead of using a fixed layer. The target reduction ratio is split into M equal multiplicative stages. At a set of predefined monitored layers, similarity scores are computed between visual and text tokens to measure how much text-relevant mass the top-K tokens cover. Then this mass is compared against a threshold to decide whether to prune at the current layer or defer to the next stage. Tokens to keep are then selected using standard text-to-image attention during pruning. Any unfinished stages are merged and executed at a fixed fallback layer, so every input ends with the same final number of visual tokens. The method is evaluated on image, document, video and captioning benchmarks with Qwen2.5-VL-7B, InternVL2.5-8B and LLaVA-1.5-7B/13B models.
Critical strengths
- The motivation makes sense. Pruning every input at the same fixed layer is a strong assumption, and letting the pruning depth depend on the input is more reasonable than a fixed schedule.
- The proposed method achieves good results on the tested benchmarks and outperforms the compared baselines across most settings and budgets.
Critical weaknesses
-
The comparison with baselines might not be fair. All methods are matched on the final number of retained tokens, not on the actual compute. Since HEAT often defers pruning to deeper layers (Fig. 4 shows most MME/MMStar/POPE samples first prune at L12–L14), many more tokens pass through many more layers than in methods like FastV or DivPrune, which prune early or before the LLM. A fair comparison should match the total token-layer budget (or FLOPs). For example, with 500 tokens, a 33% target and 10 layers, the budget should be 500×33%×10 ≈ 1670 token-layers, however the pruning is scheduled. In Table 2, HEAT still uses more than half of the full prefill FLOPs at a "66.7% reduction". Also, the efficiency comparison only includes V2Drop and only on MME and DocVQA, where DocVQA is the most favorable case since most samples prune at L6. In addition, the tables label the setting "Average Retain 33.3% Tokens", which is misleading because the budget is actually the terminal token count. Finally, against a fixed progressive schedule with similar depth (12→14→16 in Table 4a), the gain is quite small (63.0 vs 63.0 MMStar, 2318.4 vs 2305.3 MME), which makes it unclear how much of the improvement comes from adaptivity rather than just pruning later.
-
The method is not as adaptive as described. Even though it's called adaptive depth scaling, the monitored layers (L6–L14), the fallback layer (L16), the number of stages (M=3) and the per-stage retention ratio are all predefined. Each check can only remove a fixed fraction of tokens or nothing at all. I had hoped to see a method that can prune at any layer and by any amount, as long as the overall budget is met. On top of that, the temperature, the threshold endpoints for each budget, and the decay schedule are all hand-tuned, and LLaVA-1.5-7B even needs its own separate schedule (Table 5). It's not clear how these values were selected or whether they were tuned on the evaluation benchmarks.
-
The experiments use older models. The main experiments are on Qwen2.5-VL-7B and InternVL2.5-8B, and the appendix adds LLaVA-1.5, which is pretty outdated now. InternVL3.5 only appears in the appendix and is only compared against FastV. Newer models like the Qwen3-VL and Qwen3.5 series are available, but none are evaluated. I understand prior works used these models, but most of the baselines are open-sourced, and the authors could adapt them to newer models. For a 2027 submission, this weakens the generality claims.
Questions for authors
- Can you report results for all methods under a matched total compute budget (token-layers or FLOPs), or at least report FLOPs/latency for all baselines across all benchmarks rather than only V2Drop on MME and DocVQA?
- How do the baselines perform if their pruning layers are moved to depths comparable to HEAT's (e.g., FastV/PyramidDrop pruning at L12–L16)?
- How were the monitored layers, fallback layer, temperature and threshold endpoints chosen? Were they tuned on the evaluation benchmarks? How sensitive is the method to the choice of monitored layers?
- Can the method be extended to prune at arbitrary layers and by arbitrary amounts under a total budget constraint?
- How does HEAT work with batched inference where different samples prune at different layers? Have you tried integrating it into vLLM or SGLang, and what are the throughput and memory results on larger models?
2 comments
Response to authors
Thank you for providing the additional results. I appreciate the clarification of the terms used in the paper and the results for matched FLOPs and matched layer average retention. Although the performance gap with baselines largely shrinked, HEAT still achieved some gains for both models. Thus my biggest concern is largely addressed, and I think this point is also shared by other reviewers too. I also appreciate the additional results with newer Qwen models, different adaptive pruning schedules and the vllm inference latency measures, I think those results make the overall paper stronger. Therefore I'm increasing my overall score to 3.
Discussion is highly welcome
Dear Reviewer sLdX,
Thanks for your insightful comments and suggestions to strengthen our paper. In our response, we addressed each of your questions with detailed explanations and new experimental results. As the discussion phase closes on November 18, we would like to check whether all your concerns have been well addressed.
For any questions about our paper, we are happy to provide any further details. Thank you for your time!
Best,
Authors of 422