- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes CentroidPrune, a training-free visual token pruning method that decomposes attention output changes into two factors and selects tokens through centroid matching and a batch-wise greedy algorithm. Experiments show good performance across multiple models and benchmarks, especially at high pruning ratios. However, its compatibility with FlashAttention, actual efficiency gains, comparisons with recent pruning methods, and the relationship between attention output reconstruction error and final model performance require further investigation.
Critical strengths
S1. The motivation is clear, and the method has some novelty.
The paper formulates token selection as a set-level optimization problem based on attention output changes. The theoretical analysis provides a reasonable justification for the proposed approach.
S2. The experimental results are promising, especially at high pruning ratios.
The method is evaluated on three VLM architectures, retaining 98.42% of average relative performance on LLaVA-1.5-7B with 64 visual tokens and approximately 98.4% on LLaVA-NeXT-7B with 160 visual tokens (5.4% visual tokens retain). These results support the effectiveness of the method under high pruning ratios.
Critical weaknesses
W1. Weak Link Between Theoretical Analysis and Model Performance
The theoretical analysis assumes fixed Attention Weights and Value Vectors in the current layer, whereas pruning affects subsequent Transformer layers. Therefore, minimizing the Centroid Reconstruction Error does not necessarily guarantee better final performance. The authors should analyze its correlation with downstream performance and output distribution differences.
W2. FlashAttention Compatibility and Practical Efficiency
Could the authors clarify whether CentroidPrune is compatible with FlashAttention? If FlashAttention must be disabled, could this offset the speedup from token pruning? It would be helpful to provide implementation details and compare inference latency and memory usage with the unpruned model using FlashAttention.
W3. Insufficient Comparison with Recent Methods and VLM Architectures
The authors should compare CentroidPrune with recent methods, including TransPrune, HAWK, and Hi-Lo Prune (CVPR 2026), and evaluate it across diverse benchmarks and VLM architectures, including Qwen-VL, InternVL, and DeepSeek-VL.
W4. Insufficient Ablation Studies
The ablation study mainly examines the number of Greedy Selection iterations, but lacks a direct comparison with Attention Top-k under the same pruning settings. The effects of pruning layers, initialization methods, and one-stage versus two-stage pruning also remain unclear, making it difficult to isolate the contribution of the Centroid Matching Objective.
W5. Limited Evaluation on Fine-Grained Visual Tasks
Preserving the weighted attention-output centroid may not retain critical visual details for tasks such as OCR, small-object recognition, and spatial reasoning. The evaluation should include task-specific benchmarks, visual analyses, and failure cases to assess robustness under high pruning ratios.
Questions for authors
Q1. FlashAttention Compatibility and Practical Efficiency
Does CentroidPrune support FlashAttention? The paper should clarify its compatibility and compare end-to-end inference latency and GPU memory usage against the unpruned model with FlashAttention enabled under the same settings.
Q2. Comparison with Recent Methods
The evaluation should include comparisons with recent methods, such as TransPrune, HAWK, and Hi-Lo Prune, under the same model architectures and token budgets. Experiments across diverse VLM architectures, including Qwen-VL, InternVL, and DeepSeek-VL, and a broader range of benchmarks are also needed to demonstrate the method’s performance and generalizability.
Q3. Relationship Between the Theoretical Objective and Global Model Performance
Equation (5) measures local attention output changes within the current layer, but its relationship to global output errors and downstream task performance after pruning remains unclear. Further analysis of the correlation between Centroid Reconstruction Error, downstream performance, and output-distribution KL divergence is needed, along with layer-wise representation or output-distribution analyses to validate its predictive capability.
Q4. Ablation Studies and Greedy Selection Quality
The ablation study should directly compare CentroidPrune with Attention Top-k under the same pruning layers and token budgets, and investigate the effects of pruning locations, initialization strategies, and one-stage versus two-stage pruning. In addition, reporting the objective values under Equation (6) and comparing batch-wise greedy selection with sequential token updates, local swaps, or other selection strategies would help assess the quality of the greedy approximation.
Q5. Robustness on Fine-Grained Visual Tasks
Preserving the weighted attention-output centroid may not retain critical visual details for tasks such as OCR, small-object recognition, and spatial reasoning. Task-specific evaluations, visual analyses, and failure cases under high pruning ratios are needed to assess the method’s robustness on these tasks.
5 comments
Response to Reviewer PwgH (2/3)
W2 & Q1
CentroidPrune is compatible with FlashAttention. The selection needs only the attention row of the last query at each cut layer, that is, one row per head. We compute this row directly from the query of the last position and the keys of that layer, so FlashAttention-2 runs in every layer, including the cut layers, and the full attention map is never materialized. Table R4 compares the unpruned model and CentroidPrune with FlashAttention disabled and enabled on the same GPU and settings (Qwen2.5-VL-7B, 20% budget).
Table R4. Latency and peak memory with and without FlashAttention-2
| Attention | Method | End-to-end | First token | Peak memory |
|---|---|---|---|---|
| Disabled | Unpruned | 644.8 ms | 557.2 ms | 19.81 GB |
| Disabled | CentroidPrune | 618.9 ms | 523.9 ms | 18.24 GB |
| Enabled | Unpruned | 171.1 ms | 89.4 ms | 15.93 GB |
| Enabled | CentroidPrune | 130.2 ms | 53.0 ms | 14.85 GB |
With FlashAttention enabled, CentroidPrune reduces the time to the first token by 41% and the end-to-end latency by 24% relative to the unpruned model with FlashAttention, and lowers the peak memory by 1.1 GB. The speedup from pruning is therefore not offset by FlashAttention; it is larger than without FlashAttention (6% and 4%).
W3 & Q2
Following the suggestion, we compared CentroidPrune with TransPrune, HAWK and Hi-Lo Prune, and extended the evaluation to Qwen3-VL-8B, InternVL3-8B and DeepSeek-VL-7B. Results of all pruning methods below were obtained in our codebase. Avg. is the mean relative score over GQA, MMB, MME and POPE with respect to the unpruned model.
Table R5. Qwen2.5-VL-7B at the 20% budget
| Method | Tokens/layer | FLOPs | GQA | MMB | MME | POPE | Avg. |
|---|---|---|---|---|---|---|---|
| Unpruned | 100% | 100% | 60.84 | 84.10 | 2310.0 | 86.30 | 100.00 |
| HAWK | 20.0% | 48.0% | 57.90 | 76.80 | 2195.0 | 82.39 | 94.24 |
| TransPrune | 21.7% | 48.7% | 56.30 | 77.66 | 2164.3 | 86.68 | 94.75 |
| Hi-Lo Prune | 28.6% | 53.5% | 58.14 | 79.07 | 2255.7 | 83.92 | 96.12 |
| CentroidPrune | 20.0% | 47.3% | 59.12 | 80.93 | 2296.6 | 85.47 | 97.97 |
CentroidPrune achieves the highest average with the lowest FLOPs, using no more visual tokens per layer than any other method. TransPrune is higher only on POPE (by 1.21) while keeping 21.7% of the tokens per layer.
Table R6. Other architectures at the 20% budget (64 of 576 tokens for DeepSeek-VL-7B)
| Model | Method | GQA | MMB | MME | POPE | Avg. |
|---|---|---|---|---|---|---|
| Qwen3-VL-8B | Unpruned | 61.61 | 84.28 | 2363.4 | 88.23 | 100.00 |
| ZOO-Prune | 59.34 | 78.09 | 2008.1 | 87.02 | 93.14 | |
| TransPrune | 50.45 | 73.97 | 1809.8 | 74.83 | 82.76 | |
| Hi-Lo Prune | 58.06 | 77.58 | 2069.4 | 85.64 | 92.73 | |
| CentroidPrune | 59.23 | 80.15 | 2206.0 | 87.27 | 95.87 | |
| InternVL3-8B | Unpruned | 62.45 | 85.22 | 2381.5 | 90.74 | 100.00 |
| ZOO-Prune | 60.37 | 78.44 | 2035.5 | 89.63 | 93.24 | |
| TransPrune | 59.26 | 81.62 | 2242.8 | 89.03 | 95.74 | |
| CentroidPrune | 61.59 | 83.33 | 2281.0 | 90.02 | 97.85 | |
| DeepSeek-VL-7B | Unpruned | 61.29 | 72.94 | 1781.9 | 86.07 | 100.00 |
| ZOO-Prune | 57.76 | 67.96 | 1520.7 | 83.73 | 92.51 | |
| TransPrune | 52.66 | 63.78 | 1394.5 | 79.60 | 86.03 | |
| CentroidPrune | 58.13 | 70.10 | 1674.2 | 84.74 | 95.84 |
CentroidPrune achieves the best average on all three architectures, 2.1 to 3.3 points above the strongest baseline. The only per-benchmark exception is GQA on Qwen3-VL, where ZOO-Prune is higher by 0.11. Within the rebuttal period, we evaluated HAWK and Hi-Lo Prune on the Qwen models, and we will extend these comparisons in the revision. Results on OCR, document and fine-grained benchmarks are given in our response to W5.
Response to Reviewer PwgH (3/3)
W4 & Q4
When only the selection rule is replaced and the cut positions and budgets are kept fixed, CentroidPrune is the best in 42 of 44 benchmark-budget cells, and its margin over attention top-k grows as the budget shrinks (Table R2; full results in our response to Reviewer CqdN). This isolates the contribution of the centroid-matching objective.
Table R7 reports the other ablations on LLaVA-1.5-7B (same benchmarks as Table R2) with the layer-averaged budget fixed. Removing one of the two in-LLM cuts gives a one-stage variant with the cut at layer 8 or at layer 16.
Table R7. Ablations (average relative performance, %)
| Variant | 192 | 128 | 64 |
|---|---|---|---|
| Two stages, layers 8 and 16 (default) | 98.75 | 98.23 | 96.33 |
| One stage, layer 8 only | 98.81 | 98.20 | 96.07 |
| One stage, layer 16 only | 98.56 | 97.34 | 95.40 |
| Random initialization | 98.68 | 98.01 | 96.17 |
| Diversity-based initialization | 98.81 | 98.11 | 96.13 |
| Sequential greedy (one token per step) | 98.56 | 97.94 | 96.11 |
Keeping only the layer-8 cut changes the average by at most 0.26 points, whereas keeping only the layer-16 cut lowers it by up to 0.93 points, so an early instruction-conditioned cut matters more than a second one. The second cut still gives a small gain at the smallest budget (0.26 points at 64), so we keep the two-stage default of the submission; the method is otherwise robust to this choice. The initialization changes the average by at most 0.22 points, and Table 5 of the submission shows that the number of rounds T changes it by at most 0.70 points.
Table R8. Objective of Eq. (6), summed over the layer-8 and layer-16 cuts
| Selection | GQA 192 | GQA 128 | GQA 64 | TextVQA 192 | TextVQA 128 | TextVQA 64 |
|---|---|---|---|---|---|---|
| Attention top-k | 0.291 | 0.412 | 0.525 | 0.262 | 0.417 | 0.497 |
| Random | 0.427 | 0.606 | 0.549 | 0.453 | 0.652 | 0.688 |
| Diversity | 0.354 | 0.405 | 0.461 | 0.338 | 0.384 | 0.474 |
| Ours, sequential | 0.204 | 0.224 | 0.283 | 0.191 | 0.204 | 0.259 |
| Ours, batched (T = 8) | 0.200 | 0.229 | 0.291 | 0.191 | 0.212 | 0.262 |
Batched selection reaches nearly the same objective as sequential selection (within 0.006 at each cut) and is not worse in accuracy (Table R7), while using a fixed number of rounds. Both are well below the other rules.
W5 & Q5
We evaluated Qwen2.5-VL-7B on document and OCR understanding (DocVQA, InfoVQA), diagrams (AI2D), high-resolution perception of small objects (HR-Bench 4K) and spatial reasoning (CV-Bench). All baselines use their official code. Avg. is the mean relative score over the five tasks.
Table R9. Fine-grained benchmarks on Qwen2.5-VL-7B
| Budget | Method | DocVQA | InfoVQA | AI2D | HR-Bench 4K | CV-Bench | Avg. |
|---|---|---|---|---|---|---|---|
| 100% | Unpruned | 95.96 | 82.43 | 82.71 | 66.62 | 80.03 | 100.00 |
| 20% | HAWK | 87.08 | 63.87 | 77.66 | 65.88 | 75.29 | 91.02 |
| VisionZip | 80.16 | 53.55 | 77.72 | 65.75 | 77.50 | 87.60 | |
| DivPrune | 78.58 | 53.05 | 77.72 | 63.88 | 74.70 | 85.89 | |
| Hi-Lo Prune | 75.83 | 46.45 | 78.14 | 64.62 | 76.89 | 84.58 | |
| CentroidPrune | 86.71 | 63.94 | 80.18 | 66.12 | 77.16 | 92.11 | |
| 10% | HAWK | 70.56 | 48.95 | 71.60 | 65.12 | 66.99 | 80.19 |
| VisionZip | 56.50 | 39.59 | 71.37 | 62.25 | 72.14 | 75.36 | |
| DivPrune | 62.67 | 40.27 | 71.57 | 66.50 | 70.62 | 77.75 | |
| Hi-Lo Prune | 60.87 | 38.20 | 73.22 | 61.25 | 73.79 | 76.49 | |
| CentroidPrune | 74.97 | 50.59 | 75.71 | 64.75 | 73.67 | 84.06 |
CentroidPrune has the highest average at both budgets, 1.1 points above HAWK at 20% and 3.9 points at 10%. At 20%, it is the best on three of the five tasks and within 0.4 points of the best on the other two. At 10%, it is the best on DocVQA, InfoVQA and AI2D, while DivPrune and HAWK are higher on HR-Bench 4K and Hi-Lo Prune is higher on CV-Bench by 0.12. Hi-Lo Prune also uses more FLOPs (53.5% and 47.6% of the unpruned model at 20% and 10%, versus 47.3% and 41.2% for CentroidPrune). On TextVQA with LLaVA-NeXT at 160 tokens and an identical schedule, CentroidPrune reaches 56.18, compared with 48.69 for attention top-k, 50.75 for diversity and 37.35 for random selection. We attribute this to the instruction-conditioned target: tokens that the question attends to strongly carry a large weight in , so dropping them yields a large centroid error. For every method, the largest drops occur on DocVQA and InfoVQA at 10%, where the answer depends on many small text regions spread over the page; CentroidPrune loses 21.0 and 31.8 points there, less than any baseline. We will add visualizations of the selected tokens, including such failure cases, to the appendix.
We will incorporate these comparisons and analyses into the revision, with complete tables in the appendix. We thank the reviewer for the detailed suggestions, which broadened our evaluation.
Response to Reviewer PwgH (1/3)
We thank the reviewer for the detailed and constructive review. Below we respond to the weaknesses (W) and questions (Q).
All new experiments use the compute-matched schedule described in our response to Reviewer CqdN, in which the average number of visual tokens per layer equals the nominal budget. A first cut is selected from layer-0 signals (the keys, values and attention row of the last query, about 1 ms), the forward pass then runs on the selected tokens, and the cuts at layers 8 and 16 follow. The selection objective is unchanged.
W1 & Q3
We agree that Eq. (5) is exact only at the pruning layer, with the attention logits and values of that layer fixed, and that minimizing it does not by itself guarantee better final performance. We will revise Section 3.2 to describe Eq. (5) as an exact local identity and Eq. (6) as a local surrogate, and support its relation to the final output with the analyses below on LLaVA-1.5-7B.
At a fixed schedule and budget, the per-example centroid error, summed over the cuts at layers 8 and 16, is positively correlated with the KL divergence between the next-token distributions of the unpruned and pruned models in every setting (Table R1). KL in turn reflects downstream behavior: the fraction of answers identical to the unpruned model falls from 90.5 to 98.5% in the lowest KL quintile to 48.0 to 75.5% in the highest.
Table R1. Spearman correlation between the per-example centroid error and the next-token KL, LLaVA-1.5-7B
With identical cut positions and per-stage budgets, CentroidPrune attains the lowest centroid error (Table R8) as well as the lowest KL and the highest accuracy at every budget (Table R2).
Table R2. Average relative accuracy (%) over GQA, MMB, MME, POPE, SQA and TextVQA, and mean next-token KL, under an identical schedule
For the layer-wise analysis, we applied the logit lens to the last position at every layer and measured the KL to the unpruned model on 1,000 examples. Table R3 shows budget 64; the other budgets follow the same pattern.
Table R3. Logit-lens KL to the unpruned model at selected layers, and top-1 agreement at the output (budget 64)
Up to layer 8, where only the instruction-agnostic layer-0 cut has been applied, the rules are close and CentroidPrune is not consistently better. From layer 12 onward, after the first instruction-conditioned cut, CentroidPrune has the lowest KL at every layer, budget and dataset, and its KL at the output is 41 to 53% of that of the best alternative. The advantage thus originates at the centroid-matching cuts and persists to the output.