acceptodds
Under review as a conference paper at ICLR 2027

CentroidPrune: Centroid Matching for Visual Token Pruning in Vision-Language Models

Abstract

Vision-Language Models (VLMs) incur substantial inference cost because a single image is converted into a large number of visual tokens. Token pruning addresses this by removing less informative visual tokens. Existing methods rank tokens with an importance measure such as attention score or token similarity, and retain the top-ranked ones. However, these measures do not reflect how removing tokens changes the model's output, and existing methods degrade sharply as the pruning ratio increases. In this work, we analyze the effect of pruning on the attention output and show that the induced change decomposes into two factors: the removed attention mass and the directional deviation of the removed tokens. Attention-based pruning minimizes the former while leaving the latter uncontrolled, and therefore does not minimize the change in the attention output. Motivated by this analysis, we propose CentroidPrune, a training-free visual token pruning method. We formulate token pruning as a subset selection problem that directly minimizes the distance between the original attention output and the output after pruning and renormalization. Extensive experiments on various VLMs and multimodal benchmarks show that CentroidPrune consistently outperforms prior methods, retaining 98.4% of the unpruned performance while discarding up to 94.4% of the visual tokens.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Models, Visual Token Pruning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PwgHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CentroidPrune, a training-free visual token pruning method that decomposes attention output changes into two factors and selects tokens through centroid matching and a batch-wise greedy algorithm. Experiments show good performance across multiple models and benchmarks, especially at high pruning ratios. However, its compatibility with FlashAttention, actual efficiency gains, comparisons with recent pruning methods, and the relationship between attention output reconstruction error and final model performance require further investigation.

Critical strengths

S1. The motivation is clear, and the method has some novelty.

The paper formulates token selection as a set-level optimization problem based on attention output changes. The theoretical analysis provides a reasonable justification for the proposed approach.

S2. The experimental results are promising, especially at high pruning ratios.

The method is evaluated on three VLM architectures, retaining 98.42% of average relative performance on LLaVA-1.5-7B with 64 visual tokens and approximately 98.4% on LLaVA-NeXT-7B with 160 visual tokens (5.4% visual tokens retain). These results support the effectiveness of the method under high pruning ratios.

Critical weaknesses

W1. Weak Link Between Theoretical Analysis and Model Performance

The theoretical analysis assumes fixed Attention Weights and Value Vectors in the current layer, whereas pruning affects subsequent Transformer layers. Therefore, minimizing the Centroid Reconstruction Error does not necessarily guarantee better final performance. The authors should analyze its correlation with downstream performance and output distribution differences.

W2. FlashAttention Compatibility and Practical Efficiency

Could the authors clarify whether CentroidPrune is compatible with FlashAttention? If FlashAttention must be disabled, could this offset the speedup from token pruning? It would be helpful to provide implementation details and compare inference latency and memory usage with the unpruned model using FlashAttention.

W3. Insufficient Comparison with Recent Methods and VLM Architectures

The authors should compare CentroidPrune with recent methods, including TransPrune, HAWK, and Hi-Lo Prune (CVPR 2026), and evaluate it across diverse benchmarks and VLM architectures, including Qwen-VL, InternVL, and DeepSeek-VL.

W4. Insufficient Ablation Studies

The ablation study mainly examines the number of Greedy Selection iterations, but lacks a direct comparison with Attention Top-k under the same pruning settings. The effects of pruning layers, initialization methods, and one-stage versus two-stage pruning also remain unclear, making it difficult to isolate the contribution of the Centroid Matching Objective.

W5. Limited Evaluation on Fine-Grained Visual Tasks

Preserving the weighted attention-output centroid may not retain critical visual details for tasks such as OCR, small-object recognition, and spatial reasoning. The evaluation should include task-specific benchmarks, visual analyses, and failure cases to assess robustness under high pruning ratios.

Questions for authors

Q1. FlashAttention Compatibility and Practical Efficiency

Does CentroidPrune support FlashAttention? The paper should clarify its compatibility and compare end-to-end inference latency and GPU memory usage against the unpruned model with FlashAttention enabled under the same settings.

Q2. Comparison with Recent Methods

The evaluation should include comparisons with recent methods, such as TransPrune, HAWK, and Hi-Lo Prune, under the same model architectures and token budgets. Experiments across diverse VLM architectures, including Qwen-VL, InternVL, and DeepSeek-VL, and a broader range of benchmarks are also needed to demonstrate the method’s performance and generalizability.

Q3. Relationship Between the Theoretical Objective and Global Model Performance

Equation (5) measures local attention output changes within the current layer, but its relationship to global output errors and downstream task performance after pruning remains unclear. Further analysis of the correlation between Centroid Reconstruction Error, downstream performance, and output-distribution KL divergence is needed, along with layer-wise representation or output-distribution analyses to validate its predictive capability.

Q4. Ablation Studies and Greedy Selection Quality

The ablation study should directly compare CentroidPrune with Attention Top-k under the same pruning layers and token budgets, and investigate the effects of pruning locations, initialization strategies, and one-stage versus two-stage pruning. In addition, reporting the objective values under Equation (6) and comparing batch-wise greedy selection with sequential token updates, local swaps, or other selection strategies would help assess the quality of the greedy approximation.

Q5. Robustness on Fine-Grained Visual Tasks

Preserving the weighted attention-output centroid may not retain critical visual details for tasks such as OCR, small-object recognition, and spatial reasoning. Task-specific evaluations, visual analyses, and failure cases under high pruning ratios are needed to assess the method’s robustness on these tasks.

5 comments
AuthorsOpenReview ↗

Response to Reviewer PwgH (1/3)

We thank the reviewer for the detailed and constructive review. Below we respond to the weaknesses (W) and questions (Q).

All new experiments use the compute-matched schedule described in our response to Reviewer CqdN, in which the average number of visual tokens per layer equals the nominal budget. A first cut is selected from layer-0 signals (the keys, values and attention row of the last query, about 1 ms), the forward pass then runs on the selected tokens, and the cuts at layers 8 and 16 follow. The selection objective is unchanged.

W1 & Q3

We agree that Eq. (5) is exact only at the pruning layer, with the attention logits and values of that layer fixed, and that minimizing it does not by itself guarantee better final performance. We will revise Section 3.2 to describe Eq. (5) as an exact local identity and Eq. (6) as a local surrogate, and support its relation to the final output with the analyses below on LLaVA-1.5-7B.

At a fixed schedule and budget, the per-example centroid error, summed over the cuts at layers 8 and 16, is positively correlated with the KL divergence between the next-token distributions of the unpruned and pruned models in every setting (Table R1). KL in turn reflects downstream behavior: the fraction of answers identical to the unpruned model falls from 90.5 to 98.5% in the lowest KL quintile to 48.0 to 75.5% in the highest.

Table R1. Spearman correlation between the per-example centroid error and the next-token KL, LLaVA-1.5-7B

GQA 192GQA 128GQA 64TextVQA 192TextVQA 128TextVQA 64
Spearman0.490.430.400.420.350.33

With identical cut positions and per-stage budgets, CentroidPrune attains the lowest centroid error (Table R8) as well as the lowest KL and the highest accuracy at every budget (Table R2).

Table R2. Average relative accuracy (%) over GQA, MMB, MME, POPE, SQA and TextVQA, and mean next-token KL, under an identical schedule

RuleAcc. 192Acc. 128Acc. 64KL GQA 64KL TextVQA 64
Attention top-k96.3393.3686.710.5230.266
Random93.9789.4481.300.3940.505
Diversity96.5594.9691.490.2900.241
CentroidPrune98.7598.2396.330.1230.104

For the layer-wise analysis, we applied the logit lens to the last position at every layer and measured the KL to the unpruned model on 1,000 examples. Table R3 shows budget 64; the other budgets follow the same pattern.

Table R3. Logit-lens KL to the unpruned model at selected layers, and top-1 agreement at the output (budget 64)

RuleGQA L8L16L24L31Top-1TextVQA L8L16L24L31Top-1
Attention top-k0.1210.3221.4270.50462.5%0.1450.2190.8610.29580.9%
Random0.1240.2491.0200.37167.8%0.1720.3431.3210.48673.3%
Diversity0.1080.1980.8640.29871.6%0.1490.1760.6820.23983.1%
CentroidPrune0.1630.1460.4020.12182.1%0.1690.1420.3300.10688.6%

Up to layer 8, where only the instruction-agnostic layer-0 cut has been applied, the rules are close and CentroidPrune is not consistently better. From layer 12 onward, after the first instruction-conditioned cut, CentroidPrune has the lowest KL at every layer, budget and dataset, and its KL at the output is 41 to 53% of that of the best alternative. The advantage thus originates at the centroid-matching cuts and persists to the output.

AuthorsOpenReview ↗

Response to Reviewer PwgH (2/3)

W2 & Q1

CentroidPrune is compatible with FlashAttention. The selection needs only the attention row of the last query at each cut layer, that is, one row per head. We compute this row directly from the query of the last position and the keys of that layer, so FlashAttention-2 runs in every layer, including the cut layers, and the full attention map is never materialized. Table R4 compares the unpruned model and CentroidPrune with FlashAttention disabled and enabled on the same GPU and settings (Qwen2.5-VL-7B, 20% budget).

Table R4. Latency and peak memory with and without FlashAttention-2

AttentionMethodEnd-to-endFirst tokenPeak memory
DisabledUnpruned644.8 ms557.2 ms19.81 GB
DisabledCentroidPrune618.9 ms523.9 ms18.24 GB
EnabledUnpruned171.1 ms89.4 ms15.93 GB
EnabledCentroidPrune130.2 ms53.0 ms14.85 GB

With FlashAttention enabled, CentroidPrune reduces the time to the first token by 41% and the end-to-end latency by 24% relative to the unpruned model with FlashAttention, and lowers the peak memory by 1.1 GB. The speedup from pruning is therefore not offset by FlashAttention; it is larger than without FlashAttention (6% and 4%).

W3 & Q2

Following the suggestion, we compared CentroidPrune with TransPrune, HAWK and Hi-Lo Prune, and extended the evaluation to Qwen3-VL-8B, InternVL3-8B and DeepSeek-VL-7B. Results of all pruning methods below were obtained in our codebase. Avg. is the mean relative score over GQA, MMB, MME and POPE with respect to the unpruned model.

Table R5. Qwen2.5-VL-7B at the 20% budget

MethodTokens/layerFLOPsGQAMMBMMEPOPEAvg.
Unpruned100%100%60.8484.102310.086.30100.00
HAWK20.0%48.0%57.9076.802195.082.3994.24
TransPrune21.7%48.7%56.3077.662164.386.6894.75
Hi-Lo Prune28.6%53.5%58.1479.072255.783.9296.12
CentroidPrune20.0%47.3%59.1280.932296.685.4797.97

CentroidPrune achieves the highest average with the lowest FLOPs, using no more visual tokens per layer than any other method. TransPrune is higher only on POPE (by 1.21) while keeping 21.7% of the tokens per layer.

Table R6. Other architectures at the 20% budget (64 of 576 tokens for DeepSeek-VL-7B)

ModelMethodGQAMMBMMEPOPEAvg.
Qwen3-VL-8BUnpruned61.6184.282363.488.23100.00
ZOO-Prune59.3478.092008.187.0293.14
TransPrune50.4573.971809.874.8382.76
Hi-Lo Prune58.0677.582069.485.6492.73
CentroidPrune59.2380.152206.087.2795.87
InternVL3-8BUnpruned62.4585.222381.590.74100.00
ZOO-Prune60.3778.442035.589.6393.24
TransPrune59.2681.622242.889.0395.74
CentroidPrune61.5983.332281.090.0297.85
DeepSeek-VL-7BUnpruned61.2972.941781.986.07100.00
ZOO-Prune57.7667.961520.783.7392.51
TransPrune52.6663.781394.579.6086.03
CentroidPrune58.1370.101674.284.7495.84

CentroidPrune achieves the best average on all three architectures, 2.1 to 3.3 points above the strongest baseline. The only per-benchmark exception is GQA on Qwen3-VL, where ZOO-Prune is higher by 0.11. Within the rebuttal period, we evaluated HAWK and Hi-Lo Prune on the Qwen models, and we will extend these comparisons in the revision. Results on OCR, document and fine-grained benchmarks are given in our response to W5.

AuthorsOpenReview ↗

Response to Reviewer PwgH (3/3)

W4 & Q4

When only the selection rule is replaced and the cut positions and budgets are kept fixed, CentroidPrune is the best in 42 of 44 benchmark-budget cells, and its margin over attention top-k grows as the budget shrinks (Table R2; full results in our response to Reviewer CqdN). This isolates the contribution of the centroid-matching objective.

Table R7 reports the other ablations on LLaVA-1.5-7B (same benchmarks as Table R2) with the layer-averaged budget fixed. Removing one of the two in-LLM cuts gives a one-stage variant with the cut at layer 8 or at layer 16.

Table R7. Ablations (average relative performance, %)

Variant19212864
Two stages, layers 8 and 16 (default)98.7598.2396.33
One stage, layer 8 only98.8198.2096.07
One stage, layer 16 only98.5697.3495.40
Random initialization98.6898.0196.17
Diversity-based initialization98.8198.1196.13
Sequential greedy (one token per step)98.5697.9496.11

Keeping only the layer-8 cut changes the average by at most 0.26 points, whereas keeping only the layer-16 cut lowers it by up to 0.93 points, so an early instruction-conditioned cut matters more than a second one. The second cut still gives a small gain at the smallest budget (0.26 points at 64), so we keep the two-stage default of the submission; the method is otherwise robust to this choice. The initialization changes the average by at most 0.22 points, and Table 5 of the submission shows that the number of rounds T changes it by at most 0.70 points.

Table R8. Objective of Eq. (6), summed over the layer-8 and layer-16 cuts

SelectionGQA 192GQA 128GQA 64TextVQA 192TextVQA 128TextVQA 64
Attention top-k0.2910.4120.5250.2620.4170.497
Random0.4270.6060.5490.4530.6520.688
Diversity0.3540.4050.4610.3380.3840.474
Ours, sequential0.2040.2240.2830.1910.2040.259
Ours, batched (T = 8)0.2000.2290.2910.1910.2120.262

Batched selection reaches nearly the same objective as sequential selection (within 0.006 at each cut) and is not worse in accuracy (Table R7), while using a fixed number of rounds. Both are well below the other rules.

W5 & Q5

We evaluated Qwen2.5-VL-7B on document and OCR understanding (DocVQA, InfoVQA), diagrams (AI2D), high-resolution perception of small objects (HR-Bench 4K) and spatial reasoning (CV-Bench). All baselines use their official code. Avg. is the mean relative score over the five tasks.

Table R9. Fine-grained benchmarks on Qwen2.5-VL-7B

BudgetMethodDocVQAInfoVQAAI2DHR-Bench 4KCV-BenchAvg.
100%Unpruned95.9682.4382.7166.6280.03100.00
20%HAWK87.0863.8777.6665.8875.2991.02
VisionZip80.1653.5577.7265.7577.5087.60
DivPrune78.5853.0577.7263.8874.7085.89
Hi-Lo Prune75.8346.4578.1464.6276.8984.58
CentroidPrune86.7163.9480.1866.1277.1692.11
10%HAWK70.5648.9571.6065.1266.9980.19
VisionZip56.5039.5971.3762.2572.1475.36
DivPrune62.6740.2771.5766.5070.6277.75
Hi-Lo Prune60.8738.2073.2261.2573.7976.49
CentroidPrune74.9750.5975.7164.7573.6784.06

CentroidPrune has the highest average at both budgets, 1.1 points above HAWK at 20% and 3.9 points at 10%. At 20%, it is the best on three of the five tasks and within 0.4 points of the best on the other two. At 10%, it is the best on DocVQA, InfoVQA and AI2D, while DivPrune and HAWK are higher on HR-Bench 4K and Hi-Lo Prune is higher on CV-Bench by 0.12. Hi-Lo Prune also uses more FLOPs (53.5% and 47.6% of the unpruned model at 20% and 10%, versus 47.3% and 41.2% for CentroidPrune). On TextVQA with LLaVA-NeXT at 160 tokens and an identical schedule, CentroidPrune reaches 56.18, compared with 48.69 for attention top-k, 50.75 for diversity and 37.35 for random selection. We attribute this to the instruction-conditioned target: tokens that the question attends to strongly carry a large weight in oqo_q, so dropping them yields a large centroid error. For every method, the largest drops occur on DocVQA and InfoVQA at 10%, where the answer depends on many small text regions spread over the page; CentroidPrune loses 21.0 and 31.8 points there, less than any baseline. We will add visualizations of the selected tokens, including such failure cases, to the appendix.

We will incorporate these comparisons and analyses into the revision, with complete tables in the appendix. We thank the reviewer for the detailed suggestions, which broadened our evaluation.

Reviewer_CqdNOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CentroidPrune formulates visual token selection as preserving attention outputs after pruning and renormalization. For fixed attention logits and value vectors, the paper derives an identity relating the output change to both the removed attention mass and the displacement of the removed tokens' weighted value centroid. This motivates a batched greedy selector that minimizes a normalized reconstruction error across attention heads using the last query position. The training-free method prunes tokens at two decoder layers. Experiments on LLaVA-1.5, LLaVA-NeXT, and Qwen2.5-VL assess benchmark accuracy, next-token distribution divergence, latency, and memory, reporting strong performance retention at small final token budgets.

Critical strengths
  1. The mass-and-direction decomposition is clear and mathematically useful. Equation (5) correctly captures the effect of renormalization for fixed attention logits and values, explaining why attention magnitude alone need not minimize reconstruction error. This gives the proposed selection criterion a more direct local target than independent importance ranking.

  2. The reported empirical improvements are substantial, especially at small final budgets and on a second architecture. At 10% final retention on Qwen2.5-VL, Table 2 reports 98.9% average retention versus 90.8% for ZOO-Prune. The next-token KL results on GQA and TextVQA provide an additional behavioral measure beyond benchmark accuracy, and Fig. 3 reports actual system costs rather than token counts alone.

Critical weaknesses
  1. The comparisons do not isolate the centroid criterion from the relatively late two-stage pruning schedule. Appendix A.2 preserves all visual tokens through the first pruning stage at layer 8, retains half until layer 16, and only then reaches the advertised budget. Equal final token counts therefore provide substantially different computation and contextualization opportunities from pre-LLM or early-pruning baselines. Fig. 3 is helpful, but does not replace a controlled comparison of centroid selection with attention top-K and diversity selection at the same layers and intermediate budgets. Without that control, it is difficult to determine how much of the large accuracy gap is due to the new objective.

  2. The exact local identity is not an exact account of the implemented model perturbation. Selection uses the last query's attention at a layer whose forward computation has already occurred, and removes tokens for subsequent layers. It does not directly preserve future queries, their changing keys/values, or the multi-head output after projection. The KL comparison is encouraging, but is also confounded by different pruning schedules. The paper needs evidence that reducing this particular local centroid error predicts downstream fidelity under a fixed schedule; otherwise the main conceptual advantage remains a plausible surrogate rather than an established explanation.

Questions for authors
  1. Under identical layer-8/layer-16 pruning and the same intermediate/final budgets, how do attention top-K, a diversity selector, and CentroidPrune compare in accuracy, KL divergence, and end-to-end latency? This is the most important missing control for my assessment.

  2. At a fixed schedule and budget, does per-example centroid reconstruction error correlate with next-token KL or answer preservation? Does using multiple relevant query positions change that relationship compared with the last query alone?

  3. Please specify whether pruning occurs before or after the attention/residual update and exactly which cached states are removed at each layer. How are attention weights obtained under the chosen attention backend, and is that overhead included in Fig. 3? This clarification is necessary to connect Equation (3) to the executed model.

4 comments
AuthorsOpenReview ↗

Response to Reviewer CqdN (1/2)

We thank the reviewer for the detailed and constructive review. Below we respond to the weaknesses (W) and questions (Q).

W1 & Q1

We agree that equal final token counts do not mean equal compute. With the schedule in Appendix A.2, LLaVA-1.5 processes on average 312, 280 and 248 visual tokens per layer for the final budgets of 192, 128 and 64. We therefore added a compute-matched schedule and ran the requested control under it.

The compute-matched schedule computes the selection signal at layer 0 (only the layer-0 keys and values and the attention row of the last query), selects tokens with Eq. (6), restarts the forward pass from layer 0 with them, and cuts again at layers 8 and 16. The per-stage budgets follow one rule, 1.5k, 1.25k and 0.625k (1.5k, 1.14k and 0.57k for the 28-layer LLM of Qwen2.5-VL), so the average number of visual tokens per layer equals the nominal budget kk and no layer sees more than 1.5k1.5k visual tokens. Since the last position has not yet attended to the instruction at layer 0, the first cut is not instruction-specific; the instruction-conditioned cuts at layers 8 and 16 follow. The layer-0 signal costs 1.0 ms (0.45% of prefill FLOPs) on LLaVA-1.5 and 1.2 ms (0.36%) on LLaVA-NeXT.

Table R1. Prefill FLOPs relative to the unpruned model (%), LLaVA-1.5-7B, selection cost included

Method19212864
ToMe, HiRED, VisionZip, DivPrune41.7-42.032.1-32.422.6-23.0
FastV41.932.423.0
PyramidDrop43.333.623.9
SparseVLM43.934.825.6
ZOO-Prune*76.166.657.1
CentroidPrune (compute-matched)42.533.023.4

*Includes its zeroth-order sensitivity estimation, which re-runs the projector on all visual tokens for every perturbation.

Our FLOPs are at most 0.9 points above those of the pre-LLM methods and below those of PyramidDrop and SparseVLM.

Table R2. Average relative performance (%). ZOO-Prune, DivPrune and VisionZip are from Tables 1 to 3, each in its own configuration. Top-k, Random and Diversity use our compute-matched schedule with only the selection rule replaced at every cut. Averages are over GQA, MMB, MME, POPE, SQA and TextVQA (GQA, MMB, MME and POPE for Qwen2.5-VL).

SettingOursZOO-PruneDivPruneVisionZipTop-kRandomDiversity
LLaVA-1.5, 19298.7598.3397.7597.6396.3393.9796.55
LLaVA-1.5, 12898.2397.1496.7396.1893.3689.4494.96
LLaVA-1.5, 6496.3395.1594.4392.8286.7181.3091.49
LLaVA-NeXT, 64097.7697.3595.7497.5494.8691.5394.96
LLaVA-NeXT, 32096.2696.1893.6194.1187.7480.6690.36
LLaVA-NeXT, 16095.0694.4591.0189.8382.6470.9888.61
Qwen2.5-VL, 20%97.9796.3396.0195.5791.2389.2092.85
Qwen2.5-VL, 10%94.9790.7690.5388.6784.1976.1886.73

At matched compute, CentroidPrune is the best at every budget, with margins of 0.1 to 4.2 points over the baselines. These margins are smaller than with the original schedule, whose results remain valid as a higher-compute operating point, and we will update the abstract, the introduction and Figure 1 (FLOPs on the x-axis). Since the baselines run in their own configurations, this comparison does not depend on our schedule.

Under the identical schedule, CentroidPrune is the best in 42 of the 44 benchmark-budget cells; the exceptions are POPE at 128 on LLaVA-1.5 and SQA at 160 on LLaVA-NeXT, where diversity is higher by 0.03 and 0.99. As all rules share the same cuts and budgets, these differences come from the selection objective. The gap over top-k grows as the budget shrinks, from 2.4 to 9.6 points on LLaVA-1.5, from 2.9 to 12.4 on LLaVA-NeXT and from 6.7 to 10.8 on Qwen2.5-VL. Per-benchmark results will be added to the appendix.

Table R3. Mean KL(unpruned ∥ pruned)\mathrm{KL}(\text{unpruned}\,\|\,\text{pruned}) of the first answer token under the identical schedule, LLaVA-1.5-7B

RuleGQA 192TextVQA 192GQA 128TextVQA 128GQA 64TextVQA 64
Ours0.04270.03930.07190.06080.12270.1042
Top-k0.15120.07290.24550.14010.52250.2655
Random0.13380.23530.22130.34450.39380.5050
Diversity0.09100.08210.14980.13530.29000.2413

CentroidPrune has the lowest mean and median KL in every setting. These values are higher than in Table 4 of the submission because the compute-matched schedule processes fewer tokens. End-to-end latency relative to the unpruned model on LLaVA-NeXT at 320 is 69% for CentroidPrune, 65% for top-k, 66% for random and 83% for diversity, so our selection adds about 4 points over top-k and is faster than diversity selection.

AuthorsOpenReview ↗

Response to Reviewer CqdN (2/2)

W2 & Q2

We agree that Eq. (5) is exact only at the pruning layer, given the attention logits and values of that layer, and that it does not model how the queries, keys and values of later layers change. We will revise Section 3.2 to describe Eq. (5) as an exact local identity and Eq. (6) as a local surrogate, and support its relation to downstream behavior with the analysis below.

For the multi-head output after projection, let Δ(h)\Delta^{(h)} be the change of head hh and J(S)=∑h∥Δ(h)∥2/∥oq(h)∥2J(S)=\sum_h \|\Delta^{(h)}\|^2/\|o_q^{(h)}\|^2 be the objective in Eq. (6). Then

∥WO [Δ(1);… ;Δ(H)]∥≤∥WO∥2 max⁡h∥oq(h)∥ J(S),\big\|W_O\,[\Delta^{(1)};\dots;\Delta^{(H)}]\big\| \le \|W_O\|_2\,\max_h \|o_q^{(h)}\|\,\sqrt{J(S)},

and the constant does not depend on SS, so minimizing Eq. (6) minimizes an upper bound on the change of the attention output after the output projection.

At a fixed schedule and budget, the per-example centroid error (summed over the cuts at layers 8 and 16) is positively correlated with the next-token KL in every setting, with Spearman correlations between 0.33 and 0.49 (Table R4). The correlation is moderate, which is expected because KL also depends on how difficult each question is, but it is consistent across budgets and datasets. KL in turn reflects answer preservation (Table R5): examples whose answer changed have 2.5 to 4.4 times higher mean KL than those whose answer was kept. Across rules (Tables R2 and R3), CentroidPrune, the rule that directly minimizes this error, gives the lowest KL and the highest accuracy at every budget.

Table R4. Spearman correlation between the per-example centroid error and the next-token KL, LLaVA-1.5-7B

Query positionsGQA 192GQA 128GQA 64TextVQA 192TextVQA 128TextVQA 64
Last query0.490.430.400.420.350.33
All instruction tokens0.460.440.430.310.320.35

Table R5. Fraction of answers identical to the unpruned model in each KL quintile, CentroidPrune on LLaVA-1.5-7B

SettingLowest 20%2nd3rd4thHighest 20%
GQA 19298.5%98.0%93.0%84.0%75.5%
GQA 12898.0%97.0%90.0%84.0%63.5%
GQA 6498.0%91.0%87.5%82.5%48.0%
TextVQA 19292.5%93.5%92.0%83.5%72.5%
TextVQA 12891.0%93.0%87.0%79.0%60.0%
TextVQA 6490.5%82.0%88.0%73.0%55.0%

Using all instruction tokens as queries changes the correlation by at most 0.03, except on TextVQA at 192, where it drops from 0.42 to 0.31. We keep the last query because it is the position that produces the first answer token, and under causal attention it already attends to the whole instruction. The other positions also include function words and chat-template tokens whose attention to visual tokens is not specific to the question, which can dilute the signal, and each additional position needs its own attention row.

Q3

Pruning happens after the cut layer ℓ\ell has been fully computed on the current sequence, including attention, the MLP and both residual updates. The selection uses the attention row of the last query and the value vectors computed inside this layer, which together give oqo_q in Eq. (1) without extra computation. The hidden states of the unselected visual tokens are then removed from the output of layer ℓ\ell, and layers ℓ+1\ell+1 onward run on the shorter sequence in the same forward pass. Eq. (3) is therefore the renormalized output of layer ℓ\ell that serves as a local target, not the output of later layers, as discussed in W2.

Each layer stores keys and values only for the tokens present when that layer was computed, and these caches are reused during decoding. With the original schedule, the layers up to the first cut store all visual tokens, the layers between the two cuts store the retained half, and the layers after the second cut store only the final subset. With the compute-matched schedule, the layer-0 signal is used only for selection and then discarded; the forward pass restarts from layer 0 with the selected tokens, so every layer stores only these tokens, and fewer after layers 8 and 16.

In the submission, attention probabilities were requested only at the cut layers, and only the row of the last query was used. We now compute this row directly from the query of the last position and the keys of that layer (one matrix-vector product per head), so FlashAttention runs in every layer, including the cut layers; the resulting latency and memory are reported in our response to Reviewer PwgH (W2 & Q1). All reported latencies are end-to-end and include the cost of the selection, as well as the layer-0 signal of the compute-matched schedule.

We will add all of these results to the revised paper, with full per-benchmark tables in the appendix. We sincerely appreciate the constructive feedback, which helped us strengthen the paper.

Dear Reviewer CqdN, Thank you again for your detailed review. As the discussion period is coming to an end, we would like to check whether our response has addressed your concerns. Following your suggestions, we:

  • W1/Q1: ran the control you asked for. With the cut layers and per-stage budgets fixed and only the selection rule swapped, CentroidPrune is the best in 42 of 44 benchmark-budget cells and has the lowest KL in every setting. Its gap over attention top-k grows from 2.4 to 9.6 pp on LLaVA-1.5 as the budget shrinks.
  • W1: added a compute-matched schedule in which the average number of visual tokens per layer equals the budget. CentroidPrune remains the best at every budget, by 0.1 to 4.2 pp.
  • W2/Q2: showed that, under a fixed schedule, the per-example centroid error correlates with the next-token KL in every setting (Spearman 0.33 to 0.49). Answers that change have 2.5 to 4.4 times higher KL.
  • Q3: clarified when pruning happens, which caches are kept, and how the attention row is obtained under FlashAttention.

If any of these points remains unresolved, please let us know. We will answer before the deadline. If our response has resolved your concerns, we would appreciate it if you could take it into account in your final assessment. Best regards, The Authors

The authors have addressed most of my concerns with additional controlled experiments and implementation clarifications. In particular, the compute-matched comparisons provide stronger evidence that the improvements come from the centroid-based selection criterion rather than the pruning schedule. The additional correlation analysis also supports the usefulness of the proposed objective, although its theoretical connection to downstream fidelity remains somewhat limited. Overall, the rebuttal has substantially strengthened the paper, and I am inclined to raise my rating.

Reviewer_CKySOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Training-free visual token pruning inside the LLM decoder. Shows the attention-output change from pruning splits exactly into (removed attention mass) × (direction deviation); attention top-k only controls the first. Picks the token subset whose renormalized attention centroid best matches the original output (last query position), via a cheap greedy procedure. Applied in two stages at intermediate decoder layers. Retains ~98% of full performance at ~90% pruning on LLaVA-1.5 / LLaVA-NeXT / Qwen2.5-VL.

Critical strengths

1. Clean analysis that directly gives the method.

  • Exact decomposition, no approximation; explains the failure of attention top-k (high-attention tokens cluster).
  • Fidelity results (KL to the unpruned model) back up the mechanism.

2. Strong, stable results at aggressive budgets.

  • Degradation stays small where baselines drop sharply.
  • Training-free, no calibration data, no extra forward pass, negligible selection cost.
Critical weaknesses

1. Comparisons aren't at matched compute.

  • Prunes late, inside the LLM, in two stages. Early layers still process all / half of the visual tokens, so the effective per-layer token count is well above the nominal budget.
  • Most baselines prune before the LLM and really run at the nominal budget throughout.
  • So the headline claim (beats baselines given far more tokens) is overstated. The paper's own efficiency plot shows little saving on LLaVA-1.5.
  • Need: average tokens per layer / FLOPs, and comparison at matched FLOPs or latency.

2. Criterion vs. schedule not disentangled.

  • Both the selection rule and the late two-stage schedule change at once.
  • Missing: same schedule with attention top-k / random / diversity-based selection. Fidelity results have the same confound.
  • Pruning layers and stage ratio not ablated.

3. Evaluation scope and baselines.

  • Baseline provenance mixed (some look copied from prior papers, some re-run), although the paper claims a single codebase.
  • No recent query-aware baselines; Qwen2.5-VL evaluated on few benchmarks, none OCR/document-heavy.
  • 7B models and single-image tasks only.
Questions for authors
  1. Average tokens per layer / FLOPs for all methods, and results at matched compute?
  2. Same schedule with other selection rules (attention top-k, random, diversity): how much gain remains?
  3. Sensitivity to pruning layers and stage ratio; does one schedule transfer across backbones?
  4. Which baseline numbers were re-run vs. copied?
  5. Results on OCR / document tasks with Qwen2.5-VL?
4 comments
AuthorsOpenReview ↗

Response to Reviewer CKyS (1/3)

We thank the reviewer for the detailed and constructive review. Below we respond to the weaknesses (W) and questions (Q).

W1 & Q1

We agree that the headline comparison was not at matched compute. With the schedule in Appendix A.2, LLaVA-1.5 processes on average 312, 280 and 248 visual tokens per layer for the final budgets of 192, 128 and 64, so the claim in Section 1 that CentroidPrune with 64 tokens surpasses every baseline evaluated with three times that budget is overstated. We will revise the abstract, the introduction and Figure 1 (with FLOPs on the horizontal axis) accordingly. Regarding Figure 3, the 576 visual tokens of LLaVA-1.5 make the prefill short, so the end-to-end latency is largely determined by decoding and the saving is small for every method.

To compare at matched compute, we added a compute-matched schedule (details in our response to Reviewer CqdN). We select a first cut from layer-0 signals (the keys, values and attention row of the last query, about 1 ms or 0.45% of the prefill FLOPs), run the forward pass on the selected tokens, and cut again at layers 8 and 16. The per-stage budgets follow one rule (1.5k, 1.25k and 0.625k on 32-layer LLMs), so the average number of visual tokens per layer equals the nominal budget kk and no layer processes more than 1.5k1.5k visual tokens. The selection objective is unchanged.

Table R1. Average visual tokens per layer / prefill FLOPs relative to the unpruned model, LLaVA-1.5-7B (selection cost included)

Method19212864
ToMe192.0 / 41.8%128.0 / 32.2%64.0 / 22.7%
FastV191.6 / 41.9%127.9 / 32.4%64.1 / 23.0%
HiRED192.0 / 41.7%128.0 / 32.1%64.0 / 22.6%
VisionZip192.0 / 41.7%128.0 / 32.2%64.0 / 22.7%
DivPrune192.0 / 42.0%128.0 / 32.4%64.0 / 23.0%
PyramidDrop192.3 / 43.3%128.0 / 33.6%63.9 / 23.9%
SparseVLM206.5 / 43.9%145.4 / 34.8%82.2 / 25.6%
ZOO-Prune*192.0 / 76.1%128.0 / 66.6%64.0 / 57.1%
CentroidPrune (compute-matched)192.0 / 42.5%128.0 / 33.0%64.0 / 23.4%

*Includes its zeroth-order sensitivity estimation, which re-runs the projector on all visual tokens for every perturbation.

The FLOPs of the compute-matched CentroidPrune are at most 0.9 percentage points above those of the pre-LLM methods and below those of PyramidDrop and SparseVLM.

Table R2. Average relative performance (%) at matched compute. Baselines are from Tables 1 to 3 of the submission, each in its own configuration. Averages are over GQA, MMB, MME, POPE, SQA and TextVQA (GQA, MMB, MME and POPE for Qwen2.5-VL).

SettingCentroidPruneZOO-PruneDivPruneVisionZip
LLaVA-1.5, 19298.7598.3397.7597.63
LLaVA-1.5, 12898.2397.1496.7396.18
LLaVA-1.5, 6496.3395.1594.4392.82
LLaVA-NeXT, 64097.7697.3595.7497.54
LLaVA-NeXT, 32096.2696.1893.6194.11
LLaVA-NeXT, 16095.0694.4591.0189.83
Qwen2.5-VL, 20%97.9796.3396.0195.57
Qwen2.5-VL, 10%94.9790.7690.5388.67

At matched compute, CentroidPrune remains the best at every budget on all three backbones, with margins of 0.1 to 4.2 points. These margins are smaller than with the original schedule, whose results we will present as a higher-compute operating point. For latency, the end-to-end time on LLaVA-NeXT at 320 is 69% of the unpruned model, and on Qwen2.5-VL at 20% with FlashAttention, CentroidPrune reduces the time to the first token by 41% and the end-to-end latency by 24% (our response to Reviewer PwgH, W2 & Q1).

AuthorsOpenReview ↗

Response to Reviewer CKyS (2/3)

W2, Q2 & Q3

With the compute-matched schedule fixed, we replaced only the selection rule at every cut with attention top-k, random selection or diversity selection (max-min, as in DivPrune), keeping the cut positions and per-stage budgets identical.

Table R3. Average relative performance (%) with an identical schedule (same benchmarks as Table R2)

SettingCentroidPruneAttention top-kRandomDiversity
LLaVA-1.5, 19298.7596.3393.9796.55
LLaVA-1.5, 12898.2393.3689.4494.96
LLaVA-1.5, 6496.3386.7181.3091.49
LLaVA-NeXT, 64097.7694.8691.5394.96
LLaVA-NeXT, 32096.2687.7480.6690.36
LLaVA-NeXT, 16095.0682.6470.9888.61
Qwen2.5-VL, 20%97.9791.2389.2092.85
Qwen2.5-VL, 10%94.9784.1976.1886.73

CentroidPrune is the best in 42 of the 44 benchmark-budget cells. The gain that remains under the identical schedule is 2.2 to 8.2 points over the strongest alternative rule (diversity) and 2.4 to 12.4 points over attention top-k, and it grows as the budget shrinks. The fidelity comparison under the same schedule shows the same ordering: CentroidPrune has the lowest KL in every setting (Table R4).

Table R4. Mean KL(unpruned ∥ pruned)\mathrm{KL}(\text{unpruned}\,\|\,\text{pruned}) of the first answer token under the identical schedule, LLaVA-1.5-7B

RuleGQA 192TextVQA 192GQA 128TextVQA 128GQA 64TextVQA 64
Attention top-k0.15120.07290.24550.14010.52250.2655
Random0.13380.23530.22130.34450.39380.5050
Diversity0.09100.08210.14980.13530.29000.2413
CentroidPrune0.04270.03930.07190.06080.12270.1042

For the pruning layers and the stage ratio, Table R5 varies the in-LLM cuts on LLaVA-1.5-7B with the layer-averaged budget fixed. Removing one cut gives a one-stage variant, which is also the extreme case of the stage ratio, since that cut keeps all tokens and the other stage absorbs the budget.

Table R5. Sensitivity to the in-LLM cuts (average relative performance, %)

Variant19212864
Two stages, layers 8 and 16 (default)98.7598.2396.33
One stage, layer 8 only98.8198.2096.07
One stage, layer 16 only98.5697.3495.40

Keeping only the layer-8 cut changes the average by at most 0.26 points, whereas keeping only the layer-16 cut lowers it by up to 0.93 points, so the method is robust to the cut configuration as long as an early instruction-conditioned cut is kept. Regarding transfer across backbones, the same schedule, with the same cut layers and per-stage rule adjusted only to keep the layer average equal to the budget, is used without per-model tuning on all six backbones (LLaVA-1.5, LLaVA-NeXT, Qwen2.5-VL, Qwen3-VL, InternVL3 and DeepSeek-VL), and CentroidPrune obtains the best average on each (Tables R2 and R6).

AuthorsOpenReview ↗

Response to Reviewer CKyS (3/3)

W3, Q4 & Q5

We thank the reviewer for pointing out the inconsistency in baseline provenance, and we will correct the statement in Appendix A.2. For baselines without public code, we reported the numbers from their papers. For baselines with public code, we re-ran them in our codebase and, when our results closely matched the published ones, reported the published numbers. In the revision, we will instead report our reproduced numbers for every baseline with public code and mark the numbers taken from the original papers. All new results in this response, including the unpruned models in Tables R6 and R7, were obtained in a single codebase.

For recent query-aware baselines, we compared with TransPrune, HAWK and Hi-Lo Prune and extended the evaluation to Qwen3-VL-8B, InternVL3-8B and DeepSeek-VL-7B (full tables in our response to Reviewer PwgH, W3 & Q2).

Table R6. Average relative performance (%) over GQA, MMB, MME and POPE at the 20% budget (64 of 576 tokens for DeepSeek-VL-7B); – denotes not evaluated

ModelCentroidPruneZOO-PruneTransPruneHAWKHi-Lo Prune
Qwen2.5-VL-7B97.97–94.7594.2496.12
Qwen3-VL-8B95.8793.1482.76–92.73
InternVL3-8B97.8593.2495.74––
DeepSeek-VL-7B95.8492.5186.03––

On Qwen2.5-VL, TransPrune and Hi-Lo Prune keep 21.7% and 28.6% of the tokens per layer on average, compared with 20.0% for CentroidPrune.

For OCR and document tasks, we evaluated Qwen2.5-VL-7B on DocVQA and InfoVQA, together with AI2D, HR-Bench 4K and CV-Bench (Table R7). All baselines use their official code.

Table R7. Fine-grained benchmarks on Qwen2.5-VL-7B

BudgetMethodDocVQAInfoVQAAI2DHR-Bench 4KCV-BenchAvg.
100%Unpruned95.9682.4382.7166.6280.03100.00
20%HAWK87.0863.8777.6665.8875.2991.02
VisionZip80.1653.5577.7265.7577.5087.60
DivPrune78.5853.0577.7263.8874.7085.89
Hi-Lo Prune75.8346.4578.1464.6276.8984.58
CentroidPrune86.7163.9480.1866.1277.1692.11
10%HAWK70.5648.9571.6065.1266.9980.19
VisionZip56.5039.5971.3762.2572.1475.36
DivPrune62.6740.2771.5766.5070.6277.75
Hi-Lo Prune60.8738.2073.2261.2573.7976.49
CentroidPrune74.9750.5975.7164.7573.6784.06

CentroidPrune has the highest average at both budgets and is the best on DocVQA and InfoVQA at 10%, where all methods drop the most. Qwen2.5-VL is now evaluated on nine benchmarks.

The added architectures broaden the evaluation beyond the LLaVA family, but they are still 7B to 8B models, and all benchmarks use single images. We could not evaluate larger models or multi-image and video inputs within the rebuttal period, and we will state this as a limitation of the current study.

We will reflect these results and the corrected claims in the revised paper, with full tables in the appendix. We appreciate the reviewer's careful reading, which helped us make the comparisons fair and transparent.

Dear Reviewer CKyS, Thank you again for your careful review and concrete questions. As the discussion period is coming to an end, we would like to check whether our response has addressed your concerns. Following your suggestions, we:

  • W1/Q1: reported the average visual tokens per layer and the prefill FLOPs of all methods, and added a compute-matched schedule whose FLOPs are within 0.9 pp of the pre-LLM methods. CentroidPrune stays the best at every budget on all three backbones, by 0.1 to 4.2 pp. We also revised the claims in the abstract and introduction.
  • W2/Q2/Q3: kept the schedule fixed and swapped only the selection rule. CentroidPrune is the best in 42 of 44 cells, 2.2 to 8.2 pp above the strongest alternative. Removing one cut changes the average by at most 0.93 pp, and the same schedule is used without tuning on all six backbones.
  • W3/Q4: explained which baseline numbers were re-run and which were copied, and corrected the statement in Appendix A.2.
  • W3/Q5: added TransPrune, HAWK and Hi-Lo Prune, three more architectures, and DocVQA, InfoVQA, AI2D, HR-Bench 4K and CV-Bench on Qwen2.5-VL. CentroidPrune has the best average in each case. We also state the 7B to 8B, single-image scope as a limitation.

If any of these points remains unresolved, please let us know. We will answer before the deadline. If our response has resolved your concerns, we would appreciate it if you could take it into account in your final assessment. Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.