- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes RDPrune, a visual token pruning method that employs trainable visual token selectors. The authors introduce a novel perspective on token pruning by reformulating visual token reduction and selection as a problem of minimizing pruning-induced distortion during MLLM inference. They characterize this distortion through task fidelity, output distribution consistency, and hidden-state representation consistency, and use Spearman correlation analysis to demonstrate the effectiveness of these three measures in assessing the degradation of downstream inference performance caused by token pruning. Experiments on the LLaVA-v1.5-7B and LLaVA-NeXT-7B backbones further demonstrate the methodological soundness and effectiveness of RDPrune.
However, the paper still has shortcomings in its methodological descriptions and ablation studies. Several implementation details remain insufficiently specified, and the supplementary materials do not provide further clarification. The authors should therefore provide additional details and explanations to address these issues.
Critical strengths
- The experimental results in Figure 3 demonstrate the effectiveness of the three distortion measures—task fidelity, output distribution consistency, and hidden-state representation consistency—in assessing whether visual token pruning degrades downstream performance. These results provide empirical evidence that all three measures offer informative signals for evaluating the impact of token pruning.
- RDPrune is a training-based framework with a comprehensive training objective, , which combines the distortion term , comprising the three components above, with the rate term . This joint rate-distortion optimization objective enables the visual token selectors to learn a balance between preserving MLLM performance and reducing the number of visual tokens. This well-designed loss formulation constitutes a notable strength of the proposed method.
Critical weaknesses
This paper introduces a novel perspective on assessing visual token pruning. However, it has several weaknesses and limitations, including the following:
- Figure 2 presents an overview of the RDPrune framework and its processing pipeline, but the MLP module shown in the diagram is insufficiently described. The current description does not specify whether the module includes nonlinear activation functions or how many layers it contains.
- Figure 2 includes a “Text Encoder” that converts user instructions into textual token representations. However, the paper does not clarify whether this refers to an additional text encoder introduced by the authors or the input embedding layer of the backbone LLM.
- The authors state that globally aggregated visual context is incorporated when predicting whether to retain or discard each visual token. However, the aggregation operation is not specified. It remains unclear whether this context is obtained through average pooling, weighted pooling, or another aggregation operator.
- The paper states that visual tokens derived from images and textual tokens derived from user instructions are projected into a 512-dimensional latent space within the token selector, where cross-attention is computed. However, it does not clarify whether the early selector takes the raw outputs of the ViT or the visual representations obtained after the backbone model’s multimodal projector.
Questions for authors
- Section 4.4, “Ablation Studies,” reports several ablation results. However, given that RDPrune employs a two-stage visual token pruning framework, the ablation analysis appears incomplete. I recommend adding an experiment that uses visual-to-text cross-modal attention scores as the sole criterion for visual token selection, while keeping the pruning stages and layer locations identical to those of RDPrune. Comparing this attention-only strategy with RDPrune would help assess the benefits of the proposed pruning mechanism. Without this comparison, the current ablation analysis seems insufficient.
- Tables 1 and 2 report results on the LLaVA-v1.5-7B and LLaVA-NeXT-7B backbones. However, among the competing methods included in these tables, only TwigVLM and LearnPruner are training-based visual token reduction methods under the taxonomy introduced in the Related Work section. The remaining methods, including FastV, DivPrune, and DART, are training-free. Consequently, the comparative evaluation against other training-based approaches is limited. Furthermore, both competing methods in Table 3 are training-free, whereas RDPrune requires training. Comparisons with additional training-based methods would strengthen the experimental evaluation.
Response to Reviewer uaVz
We thank the reviewer for the constructive feedback and suggestions. We provide clarifications and additional comparisons below.
W1. Scoring MLP architecture
The scoring MLP has three fully connected layers with two GELU activations. A separate RMSNorm-Linear-GELU projection transforms token features, which are split into local and global components. Masked average pooling aggregates the global components; the resulting context is concatenated with local features for scoring. We will clarify this in the revision.
W2. "Text Encoder" in Figure 2
"Text Encoder" refers to the backbone LLM's existing tokenizer and embedding layer (e.g., Vicuna's in LLaVA-v1.5). RDPrune introduces no additional text encoder. We will revise the figure label accordingly.
W3. Global visual context aggregation
The scorer explicitly uses masked average pooling, not weighted pooling: after the input projection, global components are averaged over valid visual tokens and concatenated with each token's local features.
Additionally, at Stage-II, global visual context is implicitly incorporated through the backbone LLM's causal self-attention. In LLaVA, instruction tokens follow visual tokens, allowing textual representations to aggregate information from preceding visual tokens. The selector further leverages these context-aware textual features through cross-attention to estimate visual token importance.
W4. Early-selector input representations
The Stage-I selector uses isual representations after LLaVA's multimodal projector, not raw ViT outputs. We will state this explicitly in the revised manuscript.
Q1. Attention-based pruning ablation
We agree that an attention-based baseline is informative. However, backbone LLM attention scores are unavailable before LLM ingestion, where Stage-I prunes tokens. To preserve the pruning locations and isolate the selection criterion, we therefore fix RDPrune at Stage-I and compare RDPrune versus attention-score-based selection at Stage-II, with the same final budget of 64 tokens. The visual-to-text cross-modal attention scores are computed as:
\sum_{q\in\mathcal{Q}} A^{(l)}_{q,i}.$$ where $s_i$ denotes the importance score of the visual token $i$, $A^{(l)}_{q,i}$ represents the attention weight from question token $q$ to visual token $i$ at pruning layer $l$. **Table 1. Stage-II selection ablation (LLaVA-v1.5-7B, 64 tokens).** | Stage-II selection | GQA | SQA-IMG | TextVQA | POPE | MME | VQAv2 | MMB-EN | Rel. (%) | |---|---:|---:|---:|---:|---:|---:|---:|---:| | RDPrune | 61.0 | 68.4 | 57.3 | 85.9 | 1818 | 76.9 | 63.8 | 98.5 | | Attention-based | 59.4 | 68.6 | 56.1 | 84.2 | 1770 | 74.3 | 63.0 | 96.5 | With Stage-I unchanged, RDPrune performs better on six of seven benchmarks and improves average relative performance by 2.0 percentage points. This comparison isolates the advantage of its learned Stage-II criterion rather than constituting a fully attention-only two-stage baseline. ### Q2. Training-based method comparisons We agree that more training-based comparisons are needed. We add StepPrune, MVP-Token, LearnPrunerV2, and TwigVLM++, alongside LearnPruner and TwigVLM, at reported average budgets of approximately 128 and 64 visual tokens. As shown in **Table 2**, RDPrune achieves relative high performance among the listed methods. **Table 2. Training-based comparisons on LLaVA-v1.5-7B.** | Method | GQA | SQA-IMG | TextVQA | POPE | MME | VQAv2 | MMB-EN | Rel. (%) | |---|---:|---:|---:|---:|---:|---:|---:|---:| | Vanilla (576 tokens) | 61.9 | 69.5 | 58.2 | 85.9 | 1862 | 78.5 | 64.7 | 100.0 | | **128 tokens (77.8% reduction)** | | | | | | | | | | StepPrune | 57.3 | 69.8 | 55.3 | 85.2 | 1749 | 75.7 | 63.2 | 96.3 | | MVP-Token | 59.3 | 68.5 | 57.2 | 85.6 | 1760 | 76.4 | -- | 97.4 | | LearnPruner | 60.3 | 68.5 | 57.3 | 86.7 | 1820 | 77.3 | 63.8 | 98.6 | | LearnPrunerV2 | 61.0 | 68.4 | 57.7 | 86.2 | 1843 | 77.7 | 64.4 | 99.1 | | TwigVLM | 60.6 | 69.5 | 57.8 | 86.6 | 1818 | 77.9 | 63.5 | 99.2 | | TwigVLM++ | 60.8 | 69.5 | 58.0 | 87.0 | 1856 | 77.9 | 63.7 | 99.7 | | RDPrune | 61.5 | 68.4 | 57.6 | 85.8 | 1824 | 78.0 | 63.9 | 99.0 | | **64 tokens (88.9% reduction)** | | | | | | | | | | StepPrune | 56.4 | 69.3 | 54.7 | 83.1 | 1698 | 72.9 | 61.8 | 94.3 | | MVP-Token | 56.8 | 68.8 | 56.3 | 82.1 | 1709 | 74.1 | -- | 94.9 | | LearnPruner | 58.9 | 68.3 | 56.6 | 86.8 | 1750 | 76.0 | 62.6 | 97.1 | | LearnPrunerV2 | 58.9 | 68.3 | 56.6 | 86.1 | 1830 | 76.3 | 63.3 | 97.7 | | TwigVLM | 58.8 | 70.0 | 55.8 | 82.7 | 1760 | 75.6 | 60.4 | 96.5 | | TwigVLM++ | 59.7 | 69.5 | 56.7 | 86.6 | 1801 | 76.8 | 63.2 | 98.2 | | RDPrune | 61.0 | 68.4 | 57.3 | 85.9 | 1818 | 76.9 | 63.8 | 98.5 | *Notes:* Baseline scores are obtained from the respective publications and our local experimental results.