acceptodds
Withdrawn from ICLR 2027

SHAPrune: Shapley-Supervised Visual-Token Pruning for Frozen Vision–Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We hypothesize that a frozen vision–language model's own removal effects identify the visual tokens it needs better than the attention it assigns them. To test this, we compute offline the Shapley values of the 64 3×3 visual-token blocks of LLaVA-1.5-7B on its own first answer token for 1,400 VQAv2 pairs, and amortize them into 256 weights, one per head of the first eight LLM layers, over each token's text-to-image attention. After layer 8 the scores select tokens by top-k (SHAPrune) or serve as the relevance of a determinantal point process (DPP; SHAPrune-D); inference evaluates no coalition. With depth, features and top-k fixed, the readout keeps 95.9/84.6% of POPE/GQA answers at 64 tokens, against 83.2/70.0% for FastV's score and 91.4/78.3% for the best label-free readout, and a registered fresh-image test confirms the gap to FastV's score (85.3 against 71.4%). As DPP relevance it keeps more answers than CLIP relevance and than none on 10,000 fresh images (registered; 88.01 against 86.58 and 85.45%; accuracy against CLIP relevance: no difference detected). Its 256 head-level weights keep more answers than a per-layer weighting trained on the same labels (registered), and cheaper removal-effect labels, down to single ablations, train heads from which no difference is detected. On 22 of 24 benchmark cells no difference from CLIP relevance at the same depth is detected, and at equal kept tokens a substantial part of SHAPrune-D's margin over earlier diversity-based pruners comes from pruning after layer 8. The model's own removal effects are thus a better pruning signal than its attention scores.

Primary area:
foundation or frontier models, including LLMs
Keywords:
visual token pruning, vision-language models, Shapley values, amortized explanation, efficient inference, evaluation metrics

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2xm5OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SHAPrune, a visual token pruning method for LLaVA-1.5-7B. It computes block-level Shapley values of 3×3 visual-token blocks with respect to the model’s own first answer token, then amortizes these removal-effect labels into a 256-parameter readout over early-layer text-to-image attention. At inference, tokens are scored after layer 8 and kept either by top-k (SHAPrune) or by using the scores as the relevance of a conditional DPP (SHAPRune-D). On POPE, GQA, MME, and TextVQA, the supervised readout preserves more unpruned answers than FastV and several training-free attention readouts at matched depth, and SHAPRune-D is competitive with CDPruner@8 and pure DPP@8. The paper also reports that other removal-effect labels tie with Shapley labels, that placement after layer 8 explains much of the margin, and that gains are weaker on TextVQA and Qwen2.5-VL.

Critical strengths
  1. Placement analysis is independently useful. The paper systematically shows that moving diversity-based pruners (pure DPP, CDPruner) from before the LLM to after layer 8 yields large recall gains, often accounting for most of SHAPrune’s margin. This finding is consequential for VLM pruning regardless of whether removal effects are the better signal.

  2. Reusable empirical artifacts and protocol. It releases block Shapley labels, trained heads, per-question outputs, and a detailed evaluation protocol across four benchmarks, three budgets, and multiple baselines. These enable future controlled comparisons with matched-cost learned selectors, which the current paper does not provide.

Critical weaknesses
  1. The hypothesis is old and untested. “Removal beats attention” is established (Jain & Wallace, 2019; Covert et al., 2021) and already used by FastSHAP/ViT-Shapley/AT2. Experiments compare supervised labels only with zero-cost training-free readouts, never a learned selector at matched cost, so they show supervision helps—not that removal effects are the better signal. §5.4’s ties further confirm the core claim is never isolated.

  2. The fixed, non-overlapping 3×3 partition defines the players, but Shapley values are partition-dependent: supervision therefore measures which geometric windows matter, not which visual entities. Objects spanning blocks are split; object-background blocks share one identity. Block size was chosen for estimator reliability (split-half +0.52 vs +0.11), not semantics. This makes selection coarser than scoring (block averaging discards within-block ranking; k=64 keeps 7 blocks plus one token). TextVQA, the consistent loss, is exactly the fine-grained, small-entity case predicted—yet never analyzed (App. E).

  3. As a method that requires offline training (~59 GPU-hours of labels plus a supervised readout), all six published baselines are training-free methods from 2025 or earlier; no contemporary learned selector is compared at matched cost, so the claimed SOTA cannot be verified on fair terms. Architecture coverage is limited to two models: on Qwen2.5-VL-7B the label split-half reliability collapses from +0.52 (LLaVA) to +0.31. The core mechanism may not transfer across architectures. As a method that already incurs offline cost, its efficiency ceiling may be far below trained pre-LLM compressors (LLaVA-Mini ICLR25, VisPCO ACL26), a regime the paper entirely avoids comparing against.

Questions for authors

Please see weaknesses.

1 comment

We thank the reviewer for the detailed review. Recall means the answer recall of the paper, brackets are 99% intervals of the difference in points, and the new POPE results count its 5,170 distinct image–question pairs.

1. "The hypothesis is old and untested."

We do not claim the idea that removal effects explain a model better than attention; §2, §3 and App. N cite FastSHAP, ViT-Shapley, AT2, Covert et al. and Jain & Wallace. What we add is a controlled test of this idea as a pruning signal for a frozen VLM, and an account of where the gains come from. Pruning after layer 8 explains a large part of the margin over diversity-based pruners, and the revision will list this as a contribution.

On a learned selector at the same cost: the submission has controls with the same head and cost. Trained on labels shuffled across pairs, the head only ties random selection; trained on the average label map, it keeps 86.0/75.4% of POPE/GQA answers. To test a real label that is not a removal effect, we now trained the same head on later-layer attention (layers 8–31, all heads, averaged over blocks), with the same pairs and training. Only the kind of label differs.

64 tokens, after layer 8POPE recallPOPE accuracyGQA recallGQA accuracy
head trained on later-layer attention90.1785.3871.2453.47
head trained on removal effects (SHAPrune)96.3289.2584.5860.14

The recall gaps are +6.15 ([+5.04, +7.28]) and +13.34 ([+11.89, +14.78]). The accuracy gaps are +3.87 and +6.67, with both intervals above zero. Recall is also higher at 128 and 192 tokens. This is one alternative label and one head seed.

On the ties in §5.4: inside the DPP we also find no difference between single-ablation and Shapley labels (recall 88.01% for Shapley against 87.94% on 10,000 fresh images, [−0.50, +0.63]; CLIP relevance gives 86.58%). The claim we can support is therefore that training the head on how much the answer changes when image tokens are removed gives a better pruning signal than attention, whether the change is measured with Shapley values or with single ablations. The revision will state it this way.

2. The fixed 3×3 partition and TextVQA

The blocks are a grid chosen for reliable labels (App. E), not objects. We ran two checks.

  • We moved the block borders at selection time, averaging the scores over the grid shifted by 0, 1 and 2 tokens. At 64 tokens recall is 96.65 against 96.32% on POPE ([−0.12, +0.81]) and 85.11 against 84.58% on GQA ([−0.18, +1.20]). We find no difference at any budget.
  • On the label side, smaller players did not help. Labels on 2×2 blocks are less reliable (split-half +0.229 against +0.519), and the head trained on them keeps fewer answers (88.8 against 94.1%, development subset; App. E).

For TextVQA, we split the questions into three equal groups by the area of the OCR boxes that contain the answer (found for 2,631 of 5,000 questions). In these data the 3×3 blocks do not lose more on small text.

  • The two cells that SHAPrune loses in §6 are against FastV's official code at layer 8, which differs from our pipeline in more than the score. When FastV's score is applied once through our pipeline, SHAPrune ties it at 192 and 128 tokens and keeps more answers at 64 (App. G). Against this version, the margin does not depend on the size of the answer region: the smallest group minus the largest is −0.9 points at 64 tokens ([−5.8, +3.9]). The same is true for SHAPrune-D against CDPruner@8.
  • Against FastV after layer 2, the margin is larger on small regions: the smallest group minus the largest is +6.3 points ([+1.5, +11.3]).

3. Training-free baselines, state of the art, backbones and trained compressors

We did not run the published learned selectors. Comparing with them would change the training data, the depth and the features together with the label, whereas the experiment in point 1 changes the label only.

On state of the art: together with the bold entries of Table 3, the "71 of 72" count in §1 can be read as such a claim. We do not make it. The revision moves the count out of §1 and removes the bold.

On the two backbones: the Qwen labels are noisier (+0.312 against +0.515, App. L). Even so, inside the DPP the Qwen head keeps more GQA answers than pure DPP@8 at 64 tokens (81.94 against 79.92%, [+0.71, +3.30]). On POPE we find no difference in recall, while accuracy is higher (90.87 against 89.86%). The signal carries over in part (one head seed).

On LLaVA-Mini and VisPCO: we claim no efficiency advantage over pruning or compressing before the LLM (§6). LLaVA-Mini trains a new model including the LLM, so recall against the frozen model's own answers cannot be computed for it. VisPCO learns how many tokens to keep at each layer for existing in-LLM pruners. It does not choose the importance signal and could be combined with ours. We will add both to the related work.

Reviewer_zyeMOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies visual-token pruning for frozen vision–language models using supervision derived from the model’s own removal effects. On LLaVA-1.5-7B, the authors estimate block-level Shapley values for the log-probability of the first informative token of the unpruned answer. These labels supervise a linear readout with 256 weights over text-to-image attention from the first eight LLM layers. After these layers, tokens are selected either by block-averaged top-k scores (SHAPRUNE) or by using the scores as the relevance of a DPP (SHAPRUNE-D). The experiments support a substantial improvement over direct attention-based scores at the same pruning depth. A subsequent 10,000-image VQAv2 test also supports a modest answer-recall improvement over same-depth CLIP-relevance and relevance-free DPP baselines. However, the evidence does not establish that Shapley weighting is necessary, that accuracy improves over CDPruner@8, or that the method consistently offers a better compute–quality tradeoff.

Critical strengths

A well-supported finding with a lightweight learned component. Controlled comparisons show that removal-effect supervision improves answer preservation over the tested direct attention scores at the same pruning depth and token budget. Fresh-image evaluation supports this finding, while the per-layer control shows that learning individual head weights adds value beyond choosing which layers to emphasize. The approach achieves this with a small linear readout while keeping the underlying vision–language model frozen.

Informative controls that clarify where pruning gains come from. The study separates the effects of supervision, diversity, and pruning placement through same-depth controls and matched-compute comparisons. SHAPRUNE-D also improves answer preservation over CLIP-based and relevance-free DPP selection on the larger fresh-image evaluation. The alternative-label and out-of-distribution experiments provide useful evidence about the limits of these benefits.

Critical weaknesses

The added value of Shapley-specific supervision remains unclear. Much cheaper single-ablation labels train a readout with no detected difference in the main POPE recall comparison. This supports removal-effect supervision more directly than the Shapley-specific formulation, limiting the justification for the more expensive labels. The alternative-label controls use top-k selection, leaving unresolved whether Shapley-trained relevance provides an additional benefit over single-ablation-trained relevance within the same DPP selector.

The demonstrated practical advantage over strong diversity baselines is limited. Although SHAPRUNE-D improves recall on the larger fresh-image evaluation, most main-benchmark comparisons against CDPruner@8 detect no difference, and VizWiz does not establish superiority. Cross-backbone evidence is also incomplete: the top-k readout trails pure DPP in some same-depth Qwen comparisons, while SHAPRUNE-D is not evaluated. Together with the matched-compute and optimized-latency results, these findings leave a consistently better compute–quality tradeoff unestablished.

Questions for authors

Targeted validation: Could the single-ablation-trained readout be evaluated as DPP relevance under the same-depth 64-token protocol, ideally on an existing fresh-image evaluation set? This would clarify whether the observed benefit over CLIP relevance requires the substantially more expensive Shapley labels. I do not assume the two supervision schemes are equivalent based on the existing non-significant top-k comparisons.

Clarification and targeted validation: Why was SHAPRUNE-D not evaluated on Qwen2.5-VL-7B? Since both a trained readout and pure DPP are already evaluated, their combination would help distinguish limitations of top-k selection from limitations of the learned relevance signal. Such a comparison would materially inform the cross-backbone assessment; if it is infeasible, please explain the constraint.

1 comment

We thank the reviewer for the careful reading. Both experiments you asked for are done and reported below. Recall means the answer recall of the paper, and brackets are 99% intervals of the difference, in points.

Q1. Single-ablation readout as the DPP relevance

We followed your protocol: same depth, 64 tokens, the existing 10,000-image fresh set. Every row uses the same DPP and only the relevance changes.

Relevance used in the DPPRecall (%)Accuracy (%)
none (pure DPP@8)85.4569.05
CLIP (CDPruner@8)86.5870.46
learned from single-ablation labels (new)87.9470.71
learned from Shapley labels (SHAPrune-D)88.0170.47

Single-ablation relevance keeps more answers than CLIP relevance (+1.36, [+0.61, +2.12]), and we find no difference between Shapley and single-ablation relevance ([−0.50, +0.63]).

You were right that the earlier top-k tie did not tell us how close the two are. This test does: any gap between the two heads is below about 0.6 points. The gain over CLIP relevance therefore does not require the more expensive Shapley labels. Single-ablation labels need 15.5 times fewer forward passes, and 3.73 GPU-hours instead of 58.76. The method itself stays the same and simply becomes cheaper to train.

In the revision we will present the contribution as removal-effect supervision. By this we mean that the head is trained to predict how much the model's answer changes when a block of image tokens is removed, whether that change is measured with Shapley values or with single ablations.

Q2. SHAPrune-D on Qwen2.5-VL-7B

It was missing from the submission because we ran out of time before the deadline. There was no technical constraint. We have now run it after layer 8 on the full POPE and GQA sets at all three budgets, with one Qwen head. Results at 64 tokens:

Selection after layer 8POPE recallPOPE accuracyGQA recallGQA accuracy
pure DPP@895.4589.8679.9257.56
learned head, top-k95.3090.3379.0057.31
learned head inside the DPP (SHAPrune-D, new)95.7690.8781.9458.38

On GQA, the learned head inside the DPP keeps more answers than pure DPP@8 (81.94 against 79.92%, [+0.71, +3.30]). On POPE we find no difference in recall (95.76 against 95.45%, [−0.58, +1.24]), while accuracy is higher (90.87 against 89.86%). The picture at 128 tokens is similar: higher GQA recall (87.94 against 86.21%) and higher POPE accuracy (91.93 against 90.87%). Over the 12 cells (two benchmarks, three budgets, two measures) it is never worse than pure DPP@8, and against the top-k head with the same weights it is better in 8 cells and worse in none.

You asked whether the weak Qwen result is a limit of top-k selection or of the learned signal. Our reading is that it is partly both.

  • Top-k selection: part of the weakness came from top-k. In the cells where pure DPP@8 beat the top-k head (POPE recall at 192 and 128 tokens), we find no difference from pure DPP@8 once the head is used inside the DPP.
  • The learned signal: it adds to the DPP on GQA, but not measurably on POPE recall. The Qwen labels are noisier than the LLaVA ones, and we trained a single head.

This fills the cross-backbone cell you found missing. On the practical advantage over diversity baselines, the revision will state the claim as preserving answers at a given depth and budget, and will make no claim about compute.

Reviewer_2SEmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SHAPrune uses Shapley-based removal effects computed offline with a frozen VLM to train a lightweight visual token importance predictor for top-k or diversity-based pruning. The paper compares scoring methods, pruning depths, computational costs, and alternative supervision labels. The results show clearer benefits in preserving the original model's responses than in improving task accuracy.

Critical strengths
  1. The evaluation disentangles factors that are often confounded in pruning comparisons. Baselines matched for pruning depth and computational cost, label comparisons, and evaluations on unseen images help distinguish the effects of learned importance scores from those of pruning location. The results suggest that removal-effect supervision provides information beyond raw attention scores.
  2. The design of computing supervision offline and using a lightweight predictor online is practical. It does not require updating the VLM or recomputing Shapley values for each request. The paper also reports actual latency and GPU memory usage, lower-cost supervision alternatives, and unfavorable results on Qwen, providing useful evidence for assessing the method's limitations.
Critical weaknesses
  1. The deployment advantages remain limited. At the same pruning depth, SHAPrune-D shows no statistically significant difference from CLIP similarity scores in 22 of 24 comparisons. Improvements in preserving the original model's responses also do not translate into detectable accuracy gains. Optimized alternatives may offer lower latency and GPU memory usage. The offline labeling cost and limited gains on Qwen further constrain the method's general applicability in deployment.
  2. The current evidence supports the value of removal-effect supervision more strongly than the necessity of Shapley values. Cheaper single-ablation labels generally achieve comparable top-k results, but their effectiveness within the best-performing DPP pipeline has not been established. Furthermore, the labels capture the effects of removing image patches on the first substantive response token, rather than the full impact of removing individual visual tokens.
Questions for authors
  1. Is the primary objective to preserve the original model's responses under a token budget, or to reduce inference costs? Which existing result most strongly supports this objective?
  2. Has the DPP variant been evaluated using the trained single-ablation predictor?
1 comment

We thank the reviewer for the careful reading. Below, recall means the answer recall of the paper (how often the pruned model gives the same answer as the un-pruned model), and brackets are 99% intervals of the difference, in points.

Q1. Is the goal to preserve the model's answers or to reduce cost, and which result supports it?

The goal is to preserve the original model's answers under a token budget. Lower inference cost is not our goal, and we agree that the paper does not show a better compute–quality trade-off: pruning before the LLM is faster and uses less memory (§5.5, §6). Our advice in §7 is to prune after layer 8 when the number of kept tokens is the constraint, and before the LLM when latency or memory is.

We measure recall rather than accuracy because pruning can raise accuracy just by changing answers. At 64 tokens CDPruner changes 478 of the 9,000 POPE answers. It breaks 200 correct ones and fixes 278 wrong ones, so its F1 (87.5) ends up above that of the un-pruned model (85.9). Accuracy by itself therefore cannot tell which selector keeps the model's behavior, and a pruner that is meant to replace the full model should keep it. The compression literature measures the same quantity under the names "flips" (Dutta et al., 2024) and "loyalty" (Wang et al., 2025); see App. B. We report accuracy next to recall in every cell.

Three results support the goal most directly.

  1. Inside the DPP, on images used nowhere else. With the same DPP at the same depth, our learned relevance keeps more answers than CLIP relevance (CDPruner@8): 87.73 against 86.43% on 15,000 fresh images ([+0.66, +1.90]), with no difference in accuracy. On the 10,000-image set alone the numbers are 88.01 and 86.58%, or 143 more questions answered like the full model. The 5,000-image set alone and VizWiz show no significant difference.
  2. The signal itself. When depth, features and the top-k rule are fixed, supervision lifts the same attention readout from 91.4/78.3% of POPE/GQA answers (our best readout without labels) to 95.9/84.6% at 64 tokens (§5.1).
  3. A head trained on a different label (new). We trained the same head on later-layer attention instead. At 64 tokens it keeps 90.17% of POPE answers against 96.32% (both on distinct pairs) and 71.24% of GQA answers against 84.58%. Its accuracy is lower as well: 85.38 against 89.25% and 53.47 against 60.14%.

On the 22 of 24 ties with CLIP relevance: those are benchmark cells. With more images (the 10,000- and 15,000-image fresh sets) the gain becomes measurable, at about 1.3 points of recall.

Q2. Has the DPP variant been evaluated with the single-ablation predictor?

It had not, so we ran it: 10,000 fresh images, 64 tokens, after layer 8. Every row uses the same DPP and only the relevance changes.

Relevance used in the DPPRecall (%)Accuracy (%)
none (pure DPP@8)85.4569.05
CLIP (CDPruner@8)86.5870.46
learned from single-ablation labels (new)87.9470.71
learned from Shapley labels (SHAPrune-D)88.0170.47

Single-ablation relevance keeps more answers than CLIP relevance (+1.36, [+0.61, +2.12]). Compared with pure DPP it keeps more answers (+2.49, [+1.67, +3.29]) and is more accurate (+1.66, [+0.96, +2.34]). We find no difference from Shapley relevance ([−0.50, +0.63]).

So the best DPP pipeline also works with the cheap labels, which cost 3.73 GPU-hours instead of 58.76. Nothing else in the method changes: it is the same removal game, the same head and the same DPP. What matters is how the head is trained. It learns to predict how much the model's answer changes when a block of image tokens is removed. Shapley values and single ablations are two ways of measuring that change, and the cheaper one is enough.

This variant also carries over to Qwen2.5-VL-7B, which was part of your concern. Inside the DPP, the learned head keeps more GQA answers than pure DPP@8 at 64 tokens (81.94 against 79.92%, [+0.71, +3.30]). On POPE we find no difference in recall, while accuracy is higher (90.87 against 89.86%, [+0.14, +1.88]).

You also pointed out that the labels only use the first answer token and patch removal. We treat the removal game as a training target and judge it by the answers the final pruner keeps. A head trained on whole-answer labels changes no TextVQA cell (§6). Single tokens as players give less reliable labels than 3×3 blocks (split-half +0.111 against +0.519). And when we recompute single-ablation labels under the pruning that is actually used (after layer 8), they agree with the input-removal labels at a Pearson ratio of 0.75, and a head retrained on them shows no change in recall at 64 tokens (§3).

In the revision we will state the goal, the measure and the scope of the claim at the start of §1, present the contribution as training on removal effects rather than on Shapley values specifically, and add the new results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.