acceptodds
Under review as a conference paper at ICLR 2027

RDPrune: Learning Visual Token Pruning via Rate-Distortion Optimization for Large Vision-Language Models

Abstract

Large vision-language models (LVLMs) incur substantial inference overhead due to long visual token sequences. Existing visual token pruning methods typically operate under predefined budgets, overlooking that inputs of different difficulty may require different amounts of visual information. We formulate visual token pruning as lossy compression, where the retained token ratio defines the rate and the behavioral discrepancy induced by pruning defines the distortion. Joint rate-distortion optimization over the data distribution enables adaptive allocation of a shared token budget across inputs according to their difficulty. We instantiate this formulation with RDPrune, a lightweight coarse-to-fine framework consisting of an early selector after vision encoder and a mid-layer selector inside the LLM. Both selectors learn binary keep-or-drop decisions to progressively compress the visual tokens. Crucially, we explicitly model pruning-induced distortion at three complementary levels: task distortion via cross-entropy, prediction distortion via output distribution KL divergence, and representation distortion via hidden states discrepancy. This directly quantifies how token removal affects downstream model behavior, rather than relying solely on pre-pruning token importance proxies. Across seven benchmarks and three LVLM backbones, RDPrune retains 97.7% of the original performance with only 11.1% of the visual tokens, achieving 2.8 inference speedup.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual Token Compression, Rate-Distortion Optimization

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_uaVzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RDPrune, a visual token pruning method that employs trainable visual token selectors. The authors introduce a novel perspective on token pruning by reformulating visual token reduction and selection as a problem of minimizing pruning-induced distortion during MLLM inference. They characterize this distortion through task fidelity, output distribution consistency, and hidden-state representation consistency, and use Spearman correlation analysis to demonstrate the effectiveness of these three measures in assessing the degradation of downstream inference performance caused by token pruning. Experiments on the LLaVA-v1.5-7B and LLaVA-NeXT-7B backbones further demonstrate the methodological soundness and effectiveness of RDPrune.

However, the paper still has shortcomings in its methodological descriptions and ablation studies. Several implementation details remain insufficiently specified, and the supplementary materials do not provide further clarification. The authors should therefore provide additional details and explanations to address these issues.

Critical strengths
  1. The experimental results in Figure 3 demonstrate the effectiveness of the three distortion measures—task fidelity, output distribution consistency, and hidden-state representation consistency—in assessing whether visual token pruning degrades downstream performance. These results provide empirical evidence that all three measures offer informative signals for evaluating the impact of token pruning.
  2. RDPrune is a training-based framework with a comprehensive training objective, L=D+λR\mathcal{L}=\mathcal{D}+\lambda\mathcal{R}, which combines the distortion term D\mathcal{D}, comprising the three components above, with the rate term R\mathcal{R}. This joint rate-distortion optimization objective enables the visual token selectors to learn a balance between preserving MLLM performance and reducing the number of visual tokens. This well-designed loss formulation constitutes a notable strength of the proposed method.
Critical weaknesses

This paper introduces a novel perspective on assessing visual token pruning. However, it has several weaknesses and limitations, including the following:

  1. Figure 2 presents an overview of the RDPrune framework and its processing pipeline, but the MLP module shown in the diagram is insufficiently described. The current description does not specify whether the module includes nonlinear activation functions or how many layers it contains.
  2. Figure 2 includes a “Text Encoder” that converts user instructions into textual token representations. However, the paper does not clarify whether this refers to an additional text encoder introduced by the authors or the input embedding layer of the backbone LLM.
  3. The authors state that globally aggregated visual context is incorporated when predicting whether to retain or discard each visual token. However, the aggregation operation is not specified. It remains unclear whether this context is obtained through average pooling, weighted pooling, or another aggregation operator.
  4. The paper states that visual tokens derived from images and textual tokens derived from user instructions are projected into a 512-dimensional latent space within the token selector, where cross-attention is computed. However, it does not clarify whether the early selector takes the raw outputs of the ViT or the visual representations obtained after the backbone model’s multimodal projector.
Questions for authors
  1. Section 4.4, “Ablation Studies,” reports several ablation results. However, given that RDPrune employs a two-stage visual token pruning framework, the ablation analysis appears incomplete. I recommend adding an experiment that uses visual-to-text cross-modal attention scores as the sole criterion for visual token selection, while keeping the pruning stages and layer locations identical to those of RDPrune. Comparing this attention-only strategy with RDPrune would help assess the benefits of the proposed pruning mechanism. Without this comparison, the current ablation analysis seems insufficient.
  2. Tables 1 and 2 report results on the LLaVA-v1.5-7B and LLaVA-NeXT-7B backbones. However, among the competing methods included in these tables, only TwigVLM and LearnPruner are training-based visual token reduction methods under the taxonomy introduced in the Related Work section. The remaining methods, including FastV, DivPrune, and DART, are training-free. Consequently, the comparative evaluation against other training-based approaches is limited. Furthermore, both competing methods in Table 3 are training-free, whereas RDPrune requires training. Comparisons with additional training-based methods would strengthen the experimental evaluation.
1 comment

Response to Reviewer uaVz

We thank the reviewer for the constructive feedback and suggestions. We provide clarifications and additional comparisons below.

W1. Scoring MLP architecture

The scoring MLP has three fully connected layers with two GELU activations. A separate RMSNorm-Linear-GELU projection transforms token features, which are split into local and global components. Masked average pooling aggregates the global components; the resulting context is concatenated with local features for scoring. We will clarify this in the revision.

W2. "Text Encoder" in Figure 2

"Text Encoder" refers to the backbone LLM's existing tokenizer and embedding layer (e.g., Vicuna's in LLaVA-v1.5). RDPrune introduces no additional text encoder. We will revise the figure label accordingly.

W3. Global visual context aggregation

The scorer explicitly uses masked average pooling, not weighted pooling: after the input projection, global components are averaged over valid visual tokens and concatenated with each token's local features.

Additionally, at Stage-II, global visual context is implicitly incorporated through the backbone LLM's causal self-attention. In LLaVA, instruction tokens follow visual tokens, allowing textual representations to aggregate information from preceding visual tokens. The selector further leverages these context-aware textual features through cross-attention to estimate visual token importance.

W4. Early-selector input representations

The Stage-I selector uses isual representations after LLaVA's multimodal projector, not raw ViT outputs. We will state this explicitly in the revised manuscript.

Q1. Attention-based pruning ablation

We agree that an attention-based baseline is informative. However, backbone LLM attention scores are unavailable before LLM ingestion, where Stage-I prunes tokens. To preserve the pruning locations and isolate the selection criterion, we therefore fix RDPrune at Stage-I and compare RDPrune versus attention-score-based selection at Stage-II, with the same final budget of 64 tokens. The visual-to-text cross-modal attention scores are computed as:

\sum_{q\in\mathcal{Q}} A^{(l)}_{q,i}.$$ where $s_i$ denotes the importance score of the visual token $i$, $A^{(l)}_{q,i}$ represents the attention weight from question token $q$ to visual token $i$ at pruning layer $l$. **Table 1. Stage-II selection ablation (LLaVA-v1.5-7B, 64 tokens).** | Stage-II selection | GQA | SQA-IMG | TextVQA | POPE | MME | VQAv2 | MMB-EN | Rel. (%) | |---|---:|---:|---:|---:|---:|---:|---:|---:| | RDPrune | 61.0 | 68.4 | 57.3 | 85.9 | 1818 | 76.9 | 63.8 | 98.5 | | Attention-based | 59.4 | 68.6 | 56.1 | 84.2 | 1770 | 74.3 | 63.0 | 96.5 | With Stage-I unchanged, RDPrune performs better on six of seven benchmarks and improves average relative performance by 2.0 percentage points. This comparison isolates the advantage of its learned Stage-II criterion rather than constituting a fully attention-only two-stage baseline. ### Q2. Training-based method comparisons We agree that more training-based comparisons are needed. We add StepPrune, MVP-Token, LearnPrunerV2, and TwigVLM++, alongside LearnPruner and TwigVLM, at reported average budgets of approximately 128 and 64 visual tokens. As shown in **Table 2**, RDPrune achieves relative high performance among the listed methods. **Table 2. Training-based comparisons on LLaVA-v1.5-7B.** | Method | GQA | SQA-IMG | TextVQA | POPE | MME | VQAv2 | MMB-EN | Rel. (%) | |---|---:|---:|---:|---:|---:|---:|---:|---:| | Vanilla (576 tokens) | 61.9 | 69.5 | 58.2 | 85.9 | 1862 | 78.5 | 64.7 | 100.0 | | **128 tokens (77.8% reduction)** | | | | | | | | | | StepPrune | 57.3 | 69.8 | 55.3 | 85.2 | 1749 | 75.7 | 63.2 | 96.3 | | MVP-Token | 59.3 | 68.5 | 57.2 | 85.6 | 1760 | 76.4 | -- | 97.4 | | LearnPruner | 60.3 | 68.5 | 57.3 | 86.7 | 1820 | 77.3 | 63.8 | 98.6 | | LearnPrunerV2 | 61.0 | 68.4 | 57.7 | 86.2 | 1843 | 77.7 | 64.4 | 99.1 | | TwigVLM | 60.6 | 69.5 | 57.8 | 86.6 | 1818 | 77.9 | 63.5 | 99.2 | | TwigVLM++ | 60.8 | 69.5 | 58.0 | 87.0 | 1856 | 77.9 | 63.7 | 99.7 | | RDPrune | 61.5 | 68.4 | 57.6 | 85.8 | 1824 | 78.0 | 63.9 | 99.0 | | **64 tokens (88.9% reduction)** | | | | | | | | | | StepPrune | 56.4 | 69.3 | 54.7 | 83.1 | 1698 | 72.9 | 61.8 | 94.3 | | MVP-Token | 56.8 | 68.8 | 56.3 | 82.1 | 1709 | 74.1 | -- | 94.9 | | LearnPruner | 58.9 | 68.3 | 56.6 | 86.8 | 1750 | 76.0 | 62.6 | 97.1 | | LearnPrunerV2 | 58.9 | 68.3 | 56.6 | 86.1 | 1830 | 76.3 | 63.3 | 97.7 | | TwigVLM | 58.8 | 70.0 | 55.8 | 82.7 | 1760 | 75.6 | 60.4 | 96.5 | | TwigVLM++ | 59.7 | 69.5 | 56.7 | 86.6 | 1801 | 76.8 | 63.2 | 98.2 | | RDPrune | 61.0 | 68.4 | 57.3 | 85.9 | 1818 | 76.9 | 63.8 | 98.5 | *Notes:* Baseline scores are obtained from the respective publications and our local experimental results.
Reviewer_qVHCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper formulates visual token pruning in LVLMs as lossy compression and optimizes a rate-distortion objective. Two lightweight query-conditioned selectors (cross-attention + MLP, with Gumbel-Softmax and straight-through binary masks) are inserted after the vision encoder (Stage I) and after LLM layer 12 (Stage II). The rate is the layer-averaged fraction of retained visual tokens. Distortion combines three terms between the pruned and unpruned frozen model: the task cross-entropy, the KL divergence between output distributions, and the MSE between final hidden states. Only the selectors are trained, on 20K GQA samples for one epoch (under an hour on four A100s); at inference, pruned tokens are physically removed.

On LLaVA-1.5-7B, RDPrune retains 98.6/98.2/97.6% of full performance at "averaged" 128/64/32 tokens, above training-free (VisionZip, DivPrune, CDPruner) and training-based (TwigVLM, LearnPruner) baselines, and its performance drops only 1 point from 128 to 32 tokens. It also reports results on LLaVA-NeXT-7B and Qwen3-VL-8B, prefill speedups of 2.0×/3.7×, and ablations on the distortion terms, the Stage II layer, and the two-stage design.

Critical strengths

S1. A principled, behavior-level training objective that is cheap to optimize. Instead of pre-pruning proxies, RDPrune supervises the selectors with the deviation of the pruned model from the unpruned one, at the task, output-distribution, and hidden-state levels. Figure 3 motivates this: KL and hidden-state MSE correlate strongly with accuracy loss across methods and budgets. Table 5 shows that each term helps, rising from 92.8% (CE only) to 97.6% (all three), with hidden-state MSE strongest alone. Training touches only the selectors, uses 20K samples, and takes under an hour, so the method is practical to deploy.

S2. Strong and flat low-budget performance with systematic design ablations. Under the paper's protocol, RDPrune loses only 1 point when going from 128 to 32 tokens on LLaVA-1.5-7B (98.6% → 97.6%), while CDPruner falls from 97.1% to 93.1%. It matches or beats LearnPruner on LLaVA-NeXT-7B. The Stage II layer sweep (93.7 → 98.1 from layer 4 to 20) and the λ\lambda sweep give a clear view of the accuracy–compute trade-off, and the paper candidly discusses its own failure cases and the training requirement.

Critical weaknesses

W1. The comparisons may not be at matched token budgets. Token counts are averaged over layers (Eq. 9): Stage I keeps more tokens in layers 1–12 and Stage II cuts them afterwards, whereas baselines keep N tokens from the first layer. In the qualitative figure Stage I keeps 249 tokens before the final 32; with Stage II at layer 12 of 32, that would average about 113 tokens, not 32. Two reported numbers point the same way. At "64 tokens" the KV cache is 146.8MB, 38% of the full 382MB, far more than 64 visual tokens would need. And in Table 6, a Stage-I-only variant matched in token count and compute drops to 77.9%, well below single-stage baselines at the same nominal budget. If baselines are compared at the final rather than the averaged count, the headline low-budget advantage (e.g., 97.6% vs 94.8% at 32 tokens) is likely overstated. Matched-KV-cache and matched-FLOPs comparisons are needed.

W2. Limited novelty, in-distribution training, and an unevaluated adaptive-rate claim. The ingredients (DynamicViT-style Gumbel masks with masked attention, KL and feature distillation, an L1-type rate term) are standard. RUTA (Zou et al., 2026) already frames visual token allocation as rate-distortion and is not compared. The selectors are trained on GQA and evaluated on GQA, where RDPrune barely degrades (61.0 vs 61.9 at 64 tokens), while most competitors are training-free. Finally, the main claimed benefit, that difficult inputs keep more tokens, is not tested: Stage II uses fixed top-K at evaluation, and no per-sample token distribution or difficulty analysis is shown. Results on Qwen3-VL-8B compare only against FastV and VisionZip and use a 500-sample VQAv2 subset.

Questions for authors
  1. Per-stage budgets. For each row of Tables 1–3, how many visual tokens does Stage I keep (mean and distribution) and how many after Stage II? How do baselines compare when matched to RDPrune's actual KV cache or prefill FLOPs, or when baselines are allowed the same two-stage schedule (e.g., VisionZip at input + FastV at layer 12)?
  2. Table 6 consistency. Why is the full RDPrune 93.3% in Table 6 but 97.6% in Tables 5 and 7? Which benchmarks and budget does Table 6 use?
  3. Adaptive rate. With both stages left adaptive, what is the per-sample token distribution, and does it correlate with question difficulty or with the unpruned model's loss? How does adaptive Stage II compare with fixed top-K at the same average budget?
  4. Training data and baselines. What are the results when training on data disjoint from all evaluation benchmarks (e.g., LLaVA-665K without GQA)? Were TwigVLM and LearnPruner trained on the same data, and how does RDPrune compare with RUTA?
5 comments

Response to Reviewer qVHC

We sincerely thank the reviewer for the careful evaluation and constructive feedback. The comments have helped us identify important issues regarding budget fairness, computational efficiency, experimental consistency, and adaptive token allocation. We have conducted additional evaluations and revisited our experimental protocol. We address the concerns below.

W1. Matched Budgets and Computational Cost

We thank the reviewer for identifying the inconsistency between final retained token counts and layer-averaged token budgets. We agree that comparing our final Stage-II token count against the input-level budgets of single-stage methods does not ensure a fair comparison.

1. Revised evaluation under layer-averaged token budgets.

We now adopt a consistent budget definition based on the number of visual tokens processed across all LLM layers:

Nˉ=(12NI+20NII)/32.\bar N = (12N_{\mathrm I}+20N_{\mathrm {II}})/32.

Here, NIN_{\mathrm I} and NIIN_{\mathrm {II}} denote the token counts retained after Stage I and Stage II, respectively.

To evaluate our existing model without additional training, we use the checkpoint trained with λ=8\lambda=8 and impose a fixed retention ratio of approximately NI:NII=3:1N_{\mathrm I}:N_{\mathrm {II}}=3:1.

The resulting performance is summarized below.

Nˉ\bar NNIN_{\mathrm I}NIIN_{\mathrm {II}}VQAv2GQASQATextVQAPOPEMMEMMB-ENMMB-CNRel. Acc.
576--78.561.969.558.285.9186264.758.3100.0
1282197377.560.768.557.585.7183363.355.698.2
641083675.659.068.455.883.8180962.153.496.0
32541873.855.168.653.576.7166359.650.191.3

These results indicate that RDPrune can maintain a reasonable token-performance trade-off without retraining, although further retraining is still needed to optimize its performance under the layer-averaged token budget. We acknowledge that our previous low-budget comparisons overstated the advantage associated with the final token count. We will revise the corresponding comparisons and conclusions to reflect the corrected budget definition.

2. Updated computational efficiency measurements.

We additionally re-evaluated the computational and memory efficiency of RDPrune on LLaVA-v1.5-7B and LLaVA-NeXT-7B.

ModelTokenTFLOPsKV cache (MiB)Peak memory (GiB)Prefill Time (ms)Total Time (ms)
LLaVA-v1.5-7B57610.6382.014.255.978.1
+ RDPrune642.162.013.124.541.3
LLaVA-NeXT-7B288036.31209.016.3226.8238.1
+ RDPrune3207.0190.113.451.774.3

For LLaVA-v1.5-7B, RDPrune reduces prefill FLOPs by 80.9% and measured KV-cache memory by 83.8%, with 2.28$\times$ faster prefill and 1.89$\times$ faster total inference.

For LLaVA-NeXT-7B, prefill FLOPs and KV-cache memory decrease by 80.7% and 84.3%, respectively, with 4.39$\times$ faster prefill and 3.20$\times$ faster total inference.

W2. Novelty, Training Data, and Adaptive Token Allocation

We appreciate the reviewer's concerns regarding the novelty of the formulation, training-data overlap, adaptive inference, and evaluation coverage.

1. Distinction from RUTA.

While RUTA focuses on reducing visual tokens before LLM processing, RDPrune jointly optimizes visual token pruning at two complementary stages: before LLM ingestion (Stage-I) and within the LLM (Stage-II). Our motivation is that visual information requirements evolve across LLM layers, and optimizing only the input token count does not explicitly account for the different levels of visual information required during subsequent processing. Accordingly, RDPrune defines the rate based on the layer-averaged number of retained visual tokens, enabling joint optimization of token usage across the two stages.

Furthermore, unlike RUTA's task-utility-based formulation, RDPrune explicitly models functional distortion at three complementary levels: task-level cross-entropy, output-level distribution alignment, and hidden-state representation alignment. These objectives encourage the pruned model to preserve not only task performance but also the predictive behavior and internal representations of the original model.

We acknowledge that the individual loss components are established techniques. Our contribution lies in establishing the connection between these complementary metrics and downstream model performance, thereby modeling the distortion. By jointly optimizing the modeled distortion and visual token retention rate under a unified rate-distortion framework, RDPrune achieves an effective balance between token efficiency and performance preservation.

2. Training-data protocols and additional mixed-source evaluation.

We acknowledge that training RDPrune on 20K GQA samples may introduce potential in-distribution advantages on GQA. However, this training protocol is not unique to RDPrune. TwigVLM and LearnPruner trains their modules on LLaVA-665K. Notably, LLaVA-665K contains training data from benchmarks such as GQA and TextVQA, which are also used for evaluation in both works. Therefore, using benchmark-related training sources is a common practice among competitive training-based pruning methods.

To evaluate cross-dataset generalization, we constructed a filtered subset of Visual-CoT (VisCoT) by excluding GQA and other evaluation benchmark-related data sources, and used it to train RDPrune. Details are shown in Q4.

3. Sample-adaptive token allocation.

Our use of fixed token budgets during evaluation was primarily intended to ensure fair comparisons with existing pruning methods, rather than to abandon sample-adaptive pruning. RDPrune inherently supports sample-adaptive token allocation at both pruning stages. As detailed in Q3, we provide additional experimental results demonstrating its adaptive pruning behavior, including sample-level token distributions and performance under varying compression levels.

4. Qwen3-VL-8B evaluation limitations.

We thank the reviewer for highlighting the limitations of our Qwen3-VL-8B evaluation. We clarify that these experiments are intended to demonstrate the applicability of RDPrune across different LVLM architectures, rather than to establish comprehensive superiority over existing pruning methods on Qwen3-VL-8B. Although the current evaluation includes a limited number of baselines and a 500-sample VQAv2 subset, the results provide preliminary evidence that RDPrune can be effectively applied to Qwen3-VL-8B while maintaining a reasonable performance-compression trade-off. We acknowledge that more extensive evaluations with additional baselines and larger test sets are needed to establish stronger comparative conclusions.

Q1. Per-stage Budgets and Matched Schedules

We thank the reviewer for requesting clarification on the per-stage token budgets and computational fairness.

1. Per-stage token statistics. In the original experiments (Tables 1-3), Stage-I retains an average of 317 visual tokens before LLM processing, while Stage-II applies fixed Top-K pruning at layer 12 of the 32-layer LLM, retaining 128, 64, or 32 tokens. Accordingly, the actual layer-averaged token count is calculated as:

Final TokensStage-I (Avg. ±\pm Std.)Stage-IILayer-Avg. Tokens
128317 ±\pm 53.39128198.88
64317 ±\pm 53.3964158.88
32317 ±\pm 53.3932138.88

We acknowledge that the original comparisons used final retained token counts, which did not fully account for the additional computation incurred before Stage-II.

2. Revised budget-matched evaluation. To address this issue, we have re-evaluated RDPrune under the consistent layer-averaged token budget definition adopted by competing methods. As reported in W1, the revised configurations achieve relative accuracies of approximately 98.4%, 96.2%, and 91.5% at nominal average budgets of 128, 64, and 32 tokens, respectively, without retraining the model. The result is shown in W1

Q2. Consistency of the Stage-Ablation

We sincerely thank the reviewer for identifying the inconsistency between the relative accuracy reported in Table 6 (93.3%) and that reported in Tables 5 and 7 (97.6%).Initially, we attributed this performance degradation to the fixed Stage-I token budget. However, after carefully repeating the experiments and examining the evaluation configurations, we discovered that an incorrect checkpoint had inadvertently been used for the stage-wise ablation in Table 6. We sincerely apologize for this oversight and the confusion it may have caused.

We have identified the source of the discrepancy and correct the corresponding results as below.

MethodStage-IStage-IIAvg TokensRelAcc (%)TFLOPsPrefill Time (ms)
RDPrune2566413698.23.423.8
w/o Stage-II256-25698.95.032.5
w/o Stage-I-6425698.85.128.8
w/o Stage-II*136-13696.93.423.4

We appreciate the reviewer's careful examination, which helped us identify and address this error.

Comments on the paper

3 comments

Response to Reviewer udzt

We thank the reviewer for the constructive feedback and suggestions. We provide clarifications and additional experimental evidence below.

W1: Clarifying the novelty of RDPrune and its instance-adaptive pruning mechanism.

1. Distinction from RUTA and prior adaptive pruning methods.

While RUTA focuses on reducing visual tokens before LLM processing, RDPrune jointly optimizes visual token pruning at two complementary stages: before LLM ingestion (Stage-I) and within the LLM (Stage-II). Our motivation is that visual information requirements evolve across LLM layers, and optimizing only the input token count does not explicitly account for the different levels of visual information required during subsequent processing. Accordingly, RDPrune defines the rate based on the layer-averaged number of retained visual tokens, enabling joint optimization of token usage across the two stages.

Furthermore, unlike RUTA's task-utility-based formulation, RDPrune explicitly models functional distortion at three complementary levels: task-level cross-entropy, output-level distribution alignment, and hidden-state representation alignment. These objectives encourage the pruned model to preserve not only task performance but also the predictive behavior and internal representations of the original model.

We acknowledge that the individual loss components are established techniques. Our contribution lies in establishing the connection between these complementary metrics and downstream model performance, thereby modeling the distortion. By jointly optimizing the modeled distortion and visual token retention rate under a unified rate-distortion framework, RDPrune achieves an effective balance between token efficiency and performance preservation.

2. Clarification of instance-adaptive pruning and evaluation protocol.

In our original evaluation, Stage-I performed adaptive pruning using learned binary decisions, whereas Stage-II employed fixed top-K selection to facilitate comparisons with existing methods under comparable token budgets.

We thank the reviewer for identifying two limitations in our original evaluation: (1) the fixed top-K constraint did not fully reflect RDPrune's learned adaptive pruning capability, and (2) comparing methods based on their final retained token counts did not adequately account for the different pruning stages, potentially leading to inconsistent comparisons of token consumption.

To address these issues, we have adopted two complementary evaluation protocols:

(1) Fixed-budget evaluation: We impose fixed token budgets at both Stage-I and Stage-II, using the layer-averaged number of retained visual tokens as a unified metric to enable more consistent comparisons with existing pruning methods.

(2) Fully adaptive evaluation: We remove all fixed token constraints and directly apply the learned binary keep/drop decisions at both stages. We report the mean and standard deviation of retained tokens across samples to characterize instance-adaptive allocation. By varying the rate coefficient λ\lambda during training, we obtain models with different trade-offs between token retention and task performance.

The resulting rate-performance curves demonstrate that RDPrune outperforms most competing pruning methods across a wide range of token budgets, highlighting the effectiveness of our rate-distortion optimization. Detailed adaptive token statistics and benchmark results are provided in Q1 and Q2. We will revise the manuscript accordingly to clearly distinguish these two evaluation protocols.

Q1: Fully Adaptive Inference and Per-Sample Token Allocation

Following the clarification in W1, we report the results under fully adaptive inference across different rate-distortion coefficients. We adopt the layer-averaged number of retained visual tokens as the unified measure of token consumption as:

Nˉ=12NI+(32−12)NII32.\bar{N}=\frac{12N_{\mathrm{I}}+(32-12)N_{\mathrm{II}}}{32}.

which LLaVA-v1.5-7B has 32 decoder layers, and Stage-I prunes tokens before LLM processing and Stage-II operates after the 12th decoder layer.

The following table reports the mean, standard deviation, and variance of retained tokens across samples, together with the relative performance across eight benchmarks.

λ\lambdaNIN_{\mathrm{I}} (Avg. ±\pm Std.)NIIN_{\mathrm{II}} (Avg. ±\pm Std.)Nˉ\bar{N} (Avg. ±\pm Std.)Rel. Acc. (%)
1.0294.59 ±\pm 46.46134.76 ±\pm 34.27194.70 ±\pm 35.7698.98
2.0193.19 ±\pm 34.2182.89 ±\pm 21.77124.25 ±\pm 24.4898.17
4.0133.84 ±\pm 28.0249.71 ±\pm 15.4381.26 ±\pm 18.4797.25
8.097.46 ±\pm 24.0636.04 ±\pm 12.3059.08 ±\pm 15.3995.74
16.071.05 ±\pm 19.7921.21 ±\pm 7.2439.90 ±\pm 10.6593.18
32.044.33 ±\pm 12.4414.35 ±\pm 5.0725.59 ±\pm 6.7290.15

Rel. Acc. denotes the average performance relative to the unpruned LLaVA-v1.5-7B across eight benchmarks.

W2 & Q2: Computational Fairness and Contribution Analysis

We thank the reviewer for raising these important concerns regarding computational fairness and the individual contributions of RDPrune.

1. Unified evaluation of computational efficiency.

We agree that matching only the final retained token count may not accurately reflect the computational cost of multi-stage pruning. To address this concern, we adopt the layer-averaged number of retained visual tokens as a unified evaluation metric. Although layer-averaged tokens do not exactly equal FLOPs or latency, they provide a transparent, architecture-aware measure of token efficiency.

2. Additional experiments under controlled token budgets.

We further evaluate RDPrune under different layer-averaged token budgets. Without retraining, we impose fixed retention constraints on Stage I (layer 0) and Stage II (layer 12) during inference, using the same trained selectors.

Nˉ\bar{N}NIN_{\mathrm{I}}NIIN_{\mathrm{II}}VQAv2GQASQA-IMGTextVQAPOPEMMEMMB-ENMMB-CNRel. Acc. (%)
Vanilla (576)--78.561.969.558.285.9186264.758.3100.00
1282197377.560.768.557.585.7183363.355.698.4
641083675.659.068.455.883.8180962.153.496.2
32541873.855.168.653.576.7166359.650.191.5

3. Contribution analysis. Regarding the contributions of the two-stage architecture and distortion modeling, Tables 5 and 6 in the original submission already provide stage-wise and loss-component ablations. These experiments demonstrate the complementary benefits of the two pruning stages and examine the contributions of task supervision, logit alignment, and hidden-state alignment.

Furthermore, rate regularization and distortion minimization serve distinct purposes: the former penalizes excessive token retention, while the latter encourages preservation of task-relevant information. Their joint optimization establishes an explicit trade-off between compression and predictive fidelity.

Reviewer_udztOpenReview ↗

comments

Thank you for the detailed response and additional experiments. The new results address some of my concerns and further strengthen the paper. However, I still have two concerns:

  1. Although layer-averaged token counts provide a more reasonable comparison, they do not directly reflect the actual computational costs. The authors should provide further comparisons with SOTA training-based methods under similar FLOPs or inference latency.
  2. The existing ablation studies examine the contributions of different stages and loss components. However, the specific contribution of rate-distortion optimization remains insufficiently demonstrated.

I maintain my rating of 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.