acceptodds
Under review as a conference paper at ICLR 2027

DIPrune: Task-Aware Token Pruning for Efficient Multimodal Language Models

Abstract

Recent training-free pruning approaches for Multimodal Large Language Models (MLLMs) effectively cut computational overhead by exploiting visual redundancy or text-vision attention. However, they frequently suffer from semantic degradation due to their task-agnostic design or unreliable attention estimates. Based on our empirical analysis, we have found that this issue arises because salient tokens in shallow layers persistently suppress emerging semantic ones through numerical inertia, leading to premature discarding of signals crucial for deep reasoning. To address the aforementioned issue, from the task-oriented aspects, we first reformulate training-free pruning as a minimization of the distortion in the final task loss and derive a tractable, token-wise upper bound to serve as a surrogate objective. Specifically, this formulation inherently reveals a previously neglected inter-layer term that accounts for gradients across layers. Accordingly, for the implementation, we propose DIPrune, a rank-based framework that employs a dual importance scoring mechanism to jointly optimize intra-layer static feature saliency and inter-layer dynamic semantic evolution. Extensive experiments on LLaVA and Qwen-VL demonstrate that DIPrune consistently achieves state-of-the-art results.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Efficient Multimodal Large Language Model, Visual Token Pruning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_E5YoOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies visual token pruning when attention-based rankings change across language-model layers. DIPrune combines a token's current text-to-vision attention rank with its rank improvement over adjacent layers, aiming to retain both currently salient and emerging task-relevant tokens. The method performs a single pruning step inside the language model and uses a separate attention calculation compatible with efficient attention kernels. Experiments on LLaVA and Qwen-VL variants report strong accuracy retention at reduced token counts, with additional evaluations of pruning position, score components, latency, and broader visual tasks.

Critical strengths

Substantial supplementary evaluation. The paper includes Qwen3 dense/MoE models, generative and grounding tasks, component ablations, and cross-architecture timing. This is broader than the main LLaVA tables alone suggest.

Critical weaknesses
  1. The main token-budget comparison is not fully auditable. DIPrune's “64 tokens” is the count after one-shot pruning at L10; earlier layers still process the full sequence. In contrast, PyramidDrop's original Table 2 labels its budgets as average tokens. The manuscript does not provide the evaluated PDrop/SparseVLM layer schedules or a common FLOPs table. The layer-matched VisionZip/HoloV controls are helpful, but do not resolve the whole main-table comparison.
  2. The claimed suppression mechanism is stronger than the empirical evidence. Figures 2–3 establish persistence and changes in attention rank, but identify later-important tokens using later attention itself. A token can remain highly attended because it is useful, so persistence alone does not establish redundancy or suppression of answer-critical evidence. Appendix B.2 supports the score's usefulness, but does not independently test this causal explanation. The paper should distinguish the observed ranking pattern from the mechanism claimed to produce it.
Questions for authors
  1. Please give the actual per-layer visual-token counts and total prefill FLOPs for every Table 1 method at the “64 tokens” setting. Were PDrop/SparseVLM budgets converted from layer averages to post-pruning counts, or reimplemented with another schedule?
  2. Do existing experiments independently test the claimed suppression of answer-relevant tokens? Otherwise, can this be presented as an observed ranking pattern motivating the heuristic, rather than an established causal mechanism?
Reviewer_Hra7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents DIPrune, a training-free visual token pruning framework for multimodal large language models (MLLMs). The authors identify a phenomenon termed Early Inertial Bias, whereby visual tokens receiving high attention in shallow layers tend to maintain their importance, potentially obscuring tokens that become semantically relevant in deeper layers. To address this issue, the paper formulates pruning as minimizing final-task loss distortion and derives an upper bound involving feature magnitude and gradient sensitivity. Motivated by this decomposition, DIPrune combines two practical importance scores: a Semantic Alignment Score based on text-to-vision attention ranks and a Semantic Evolution Score based on rank displacement between adjacent layers. Experiments across LLaVA and Qwen-VL models demonstrate competitive performance under aggressive pruning, with additional evaluations covering video understanding, visual grounding, generative tasks, and inference efficiency.

Critical strengths
  1. DIPrune demonstrates consistent improvements across multiple MLLM backbones and compression ratios. In particular, on LLaVA-v1.5-7B with only 64 retained visual tokens, it preserves 97.7% of the unpruned average performance, compared with 93.7% for HoloV. The gains on LLaVA-NeXT and the Qwen-VL series further support the practical usefulness and architectural generality of the method. The inclusion of layer-aligned comparisons also strengthens the evaluation.
  2. The combination of attention-based alignment and rank-based evolution requires no model retraining and introduces limited reported computational overhead. The component ablations and offline gradient-mass analysis in Appendix B.2 provide useful evidence that the two importance signals capture complementary information. This makes the method potentially practical for deployment.
Critical weaknesses
  1. A particularly relevant omission is TransPrune: Token Transition Pruning for Efficient Large Vision-Language Model (Li et al., CVPR 2026). TransPrune already combines instruction-guided attention with token-transition information for training-free visual token pruning. Although DIPrune differs by modeling attention-rank displacement rather than representation magnitude and direction changes, the overall motivation and dual-signal design are closely related. The absence of a discussion and direct comparison with TransPrune makes it difficult to assess the incremental novelty and empirical significance of DIPrune relative to a particularly relevant existing approach.
  2. Although the paper derives an upper bound on pruning-induced task-loss distortion, the connection between this theoretical formulation and the implemented importance scores remains insufficiently justified. Specifically, Eq. (4) decomposes the bound into terms involving token gradient energy and feature magnitude. However, the practical method replaces these quantities with attention-based alignment ranks and cross-layer rank displacement. It is unclear under what assumptions these rank-based scores provide faithful approximations to the quantities appearing in the theoretical bound.
  3. I acknowledge that Appendix Table 12 provides encouraging empirical evidence for the proposed proxies, particularly regarding gradient-energy retention. Nevertheless, these results do not fully establish the theoretical relationship or justify the stronger claim that the proposed scoring mechanism minimizes the derived error bound.
Questions for authors
  1. TransPrune (CVPR 2026) already combines instruction-guided attention with token-transition information for training-free visual token pruning. Could the authors clarify the fundamental differences between DIPrune and TransPrune, particularly the advantages of attention-rank displacement over representation-based transition measures? Could the authors also provide a direct comparison on common benchmarks under comparable computational budgets (e.g., FLOPs or measured inference latency), accounting for their different pruning schedules?
  2. Could the authors further justify how attention-rank displacement approximates the gradient-energy term in Eq. (4)? While Appendix Table 12 provides evidence based on retained gradient mass, additional token-level correlation analysis or comparisons against simpler cross-layer attention statistics would help clarify the specific contribution of rank displacement.
  3. Could the authors clarify how the final-layer key tokens and low-importance tokens in Figure 3 are identified? More importantly, is there evidence that persistently high-attention tokens are actually less relevant to the final task than emerging tokens?
Reviewer_w7j9OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes DIPrune, a training-free visual token pruning method that addresses Early Inertial Bias in multimodal large language models. It combines task-aware semantic alignment and inter-layer attention evolution to preserve important visual tokens. Extensive experiments across various MLLMs demonstrate strong performance under aggressive token pruning with low computational overhead.

Critical strengths

DIPrune consistently outperforms competitive pruning baselines across multiple architectures and benchmarks, particularly under aggressive token compression. These results demonstrate the effectiveness and generalizability of the proposed method.

Critical weaknesses

W1.

While the paper identifies Inertial Tokens through persistent attention rankings, further analysis of their actual contribution to downstream tasks would strengthen the motivation. In particular, quantifying how effectively DIPrune removes task-irrelevant Inertial Tokens while preserving Emerging Tokens would help validate the proposed mechanism.

Questions for authors

see weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.