acceptodds
Under review as a conference paper at ICLR 2027

DAPrune: Dual-Adaptive Pruning via Adaptive Ratio and Layer Selection for Large Vision Language Models

Abstract

Visual tokens in large vision language models (LVLMs) increase rapidly with image resolution and video inputs, resulting in substantial computational overhead. To reduce this cost, a pruning policy is needed to reduce the number of tokens involved in the attention computation. However, existing token pruning methods mostly adopt fixed pruning policies for all inputs, which are not sufficiently efficient and cannot adapt to input difficulty. To address this limitation, we propose DAPrune, a training-based dual-adaptive pruning framework that jointly learns where to prune and how many visual tokens to retain for each input. Specifically, DAPrune introduces two lightweight modules: a routing module for pruning layer selection and a ratio module for adaptive token retention. With the base LVLM frozen, these modules can adapt efficiently within 3 hours on a single H100 GPU for LLaVA-1.5. Empirically, DAPrune preserves over 98.5% of the baseline performance under high FLOPs reduction on LLaVA-1.5, LLaVA-NeXT, and Qwen-2.5-VL, consistently outperforming prior methods under comparable settings. Furthermore, theoretical analysis suggests that, under the same average budget, our adaptive policy achieves lower representation error than fixed pruning strategies.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision language models, dual-adaptive pruning, lightweight

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_F4YAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DAPrune, a dynamic token pruning method that selects which tokens to prune or store for each input. It uses lightweight modules to predict pruning decisions, trained with an auxiliary loss while freezing the LVLM backbone. Experiments on LLaVA-1.5, LLaVA-NeXT, and Qwen-2.5-VL show that the trade-off between performance and efficiency is well-balanced.

Critical strengths
  1. DAPrune achieves promising results across various scenarios by optimizing the balance between performance and computational cost. For instance, with LLaVA-1.5, it reduces FLOPs by 64.9% while retaining 98.7% of the baseline performance. Further experimental results using LLaVA-NeXT and Qwen2.5-VL also substantiate the method's effectiveness.

  2. Section 3 and Figure 1 highlight the limitations of manually configured pruning methods and demonstrate the need for dynamic pruning approaches. The method addresses this challenge by optimizing pruning positions and the number of retained tokens at each decoder stage based on the input. This approach offers an effective way of reducing redundant tokens while preserving model performance.

Critical weaknesses
  1. The research motivation and novelty are somewhat limited. In lines 48–50, the authors claim that existing methods typically rely on pruning configurations preset for all inputs, such as uniform pruning rates or fixed pruning layers. However, existing works have already addressed the heterogeneity of input data and variations in token redundancy across layers. For instance, SparseVLM (ICML 25) and AutoPrune (NeurIPS 25) incorporate adaptive token retention mechanisms, while PDrop (CVPR 25) explicitly leverages redundancy characteristics across layers. Therefore, the authors should more clearly articulate the distinctions between DAPrune and these existing methods, and provide a more compelling justification for the necessity and novelty of simultaneously adapting both the pruning layers and retention rates.

  2. Although DAPrune aims to reduce reliance on manual and pre-defined pruning configurations, it still requires several hyperparameters, e.g., number of pruning stages, candidate pruning layers, target retention ratios, and loss weights. While this paper examines the impact of sparsity weights and retention budgets, the sensitivity to the number of pruning stages and the selection of candidate layers has not been thoroughly investigated. It remains unclear how these configurations are determined and to what extent the reported performance gains depend on how well these parameters are developed. More importantly, reliance on multiple manually set hyperparameters appears to contradict the paper's primary objective, i.e., overcoming fixed, manual pruning schemes.

  3. Comparisons with existing methods are insufficient and potentially unfair. Table 3 primarily compares DAPrune with existing methods under training-free conditions, yet DAPrune actually utilizes training data to train its routing and retention modules. Representative training-based methods cited in related literature, such as ATP-LLaVA (CVPR 25), are absent from these comparisons. This weakens the evidence supporting the claim that the proposed method outperforms existing approaches.

Questions for authors
  1. Given that the depth of the decoder varies across model architectures (e.g., 32 layers for LLaVA-1.5-7B and 28 layers for Qwen2.5-VL-7B), do DAPrune hyperparameters (such as the number of pruning stages and target candidate layers) need to be manually reset or tuned for each model? If so, the complexity and cost associated with deploying DAPrune on new architectures would increase dramatically, potentially limiting its practicality and scalability in real-world applications.

  2. Recent studies, such as LLaVA-Mini, have demonstrated that efficient multi-modal processing can be achieved using a single visual token. Since DAPrune and LLaVA-Mini are both training-based approaches, is it possible to compare the two to clearly demonstrate the advantages of the proposed method? Furthermore, the results from LLaVA-Mini raise concerns regarding whether general multimodal benchmarks are sufficient for token pruning evaluations (as only a single visual token is enough for correct answering). To further validate the effectiveness of the proposed method, could DAPrune be evaluated using benchmarks that require fine-grained visual perception, such as V* Bench or HR-Bench (4K/8K)?

  3. Can DAPrune be extended to other practical use cases, such as LVLMs other than 7B and video-based reasoning tasks, to further validate its robustness and generalizability?

  4. Could the authors provide some failure cases or specific challenging scenarios where DAPrune’s performance degrades? Such an analysis would help readers gain a deeper understanding of the method's limitations, grasp the conditions under which it performs well, and offer insights for further improvements and future directions.

Reviewer_XDomOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DAPrune, a visual token pruning framework that adaptively selects pruning layers and token retention ratios for each input. Two lightweight controllers are trained while the LVLM backbone remains frozen, using a task loss together with routing, budget, and sparsity regularization. Experiments evaluate the performance–computation trade-off on LLaVA-1.5, LLaVA-NeXT, and Qwen2.5-VL. The paper also provides a theoretical analysis comparing adaptive and fixed pruning policies under average-budget constraints.

Critical strengths

1. Joint adaptation with modest training overhead.

Jointly adapting pruning depth and token count is well motivated, since both decisions affect information preservation and computational cost. The framework trains lightweight controllers while keeping the backbone frozen. On LLaVA-1.5, the paper reports approximately 0.53% additional trainable parameters and about three hours of training on a single H100, making the approach relatively inexpensive to train.

2. Encouraging results across three LVLM backbones.

The evaluation covers LLaVA-1.5, LLaVA-NeXT, and Qwen2.5-VL-7B, including both visual question answering and referring expression comprehension. In Table 1, DAPrune retains 98.7% of baseline performance with a 64.9% reduction in visual-token FLOPs, compared with 98.0% and 63.9% for AutoPrune. Table 2 shows a similar advantage on LLaVA-NeXT at comparable computation. On Qwen2.5-VL-7B, Table 3 reports 98.4% and 96.3% performance retention at average token retention rates of 75% and 50%, slightly above VScan’s 98.3% and 96.1%. Although the Qwen gains are small, they extend the empirical support to another backbone and task setting. Table 4 also supports combining the two adaptive components at the reported operating points.

Critical weaknesses

1. It is unclear how the task loss trains the pruning controllers.

I could not reconstruct the task-gradient path to the ratio predictor. Section 4.2 converts its output to ns=⌊rsn0⌋n_s=\lfloor r_s n_0\rfloor and retains the top nsn_s tokens, yet line 263 states that the module can be trained by backpropagation because rsr_s is continuous. If this hard selection is also used during training, the task loss is locally constant in rsr_s. The budget and sparsity losses can update the predictor, but they do not explain how it learns which inputs need more tokens. Is a soft mask or another gradient estimator used during training? For the router, Section 4.1 also needs to explain how the Gumbel-softmax weights enter the computation after hard layer selection. A short training algorithm showing these gradient paths would help establish how the proposed policy is learned.

2. The theoretical interpretation needs to be narrower.

Theorems 1 and 2 compare the distortion quantity DD, under assumptions that place the adaptive decisions between the fixed and oracle decisions. The paper does not establish that the learned controllers satisfy these assumptions. Moreover, a smaller upper bound on representation error does not imply a smaller actual error. I therefore read these results as a conditional explanation of adaptive allocation, rather than an established error advantage for DAPrune. Section 5.4 and the conclusion should make this distinction clearer. There is also a correctable issue in Lemma 1: line 927 omits the renormalization of the surviving attention weights after token deletion.

Questions for authors

See Critical Weaknesses

Reviewer_CmqJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DAPrune, a sample-adaptive training-based pruning strategy that jointly predicts where to prune and how many visual tokens to retain. A router module selects the pruning layer within each predefined stage, and a ratio module predicts the retention ratio. DAPrune preserves over 98.5% of the baseline performance under high FLOPs reduction on LLaVA-1.5, LLaVA-NeXT, and Qwen-2.5-VL.

Critical strengths
  1. The proposed DAPrune framework is well motivated, as input-adaptive pruning policies remain underexplored in this field. Through two learnable modules, it jointly adapts pruning locations and token retention ratios to balance performance and computational cost. The ablation studies also support the effectiveness of both modules.

  2. Extensive experiments are conducted on multiple VLM architectures and benchmarks, including LLaVA-v1.5, LLaVA-NeXT, and Qwen2.5-VL.

Critical weaknesses
  1. The pruning method relies on several empirical design choices. As discussed in the paper, a pruning policy should flexibly determine pruning locations and token retention ratios for each input. However, DAPrune still requires predefined stage boundaries, candidate pruning layers, and target retention ratios for each stage, which limits its adaptivity. Moreover, it's unclear why LLaVA uses three pruning stages whereas Qwen uses only one. An ablation comparing a single stage, three stages, and more stages would help assess the sensitivity to this choice.

  2. The efficiency analysis is limited. While FLOPs are reported, the paper does not provide an end-to-end latency comparison against baselines or report the overhead introduced by the two learnable modules. Although the paper claims compatibility with FlashAttention, repeatedly accessing attention scores for pruning may introduce additional latency, particularly for long visual token sequences. Actual runtime measurements are needed to assess the practical efficiency gains.

  3. Since DAPrune requires additional training, experimental comparisons with other training-based pruning methods, sunch as TwigVLM, are necessary to demonstrate its advantages. Moreover, the selected baselines are relatively old. Incorporating recent SOTA token pruning methods would provide a more comprehensive and convincing evaluation.

Questions for authors

I wonder how gradients from the task loss propagate through the discrete pruning decisions to optimize the two learnable modules. A more detailed explanation of the backward pass would help clarify the training procedure.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.