- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SHAPrune, a visual token pruning method for LLaVA-1.5-7B. It computes block-level Shapley values of 3×3 visual-token blocks with respect to the model’s own first answer token, then amortizes these removal-effect labels into a 256-parameter readout over early-layer text-to-image attention. At inference, tokens are scored after layer 8 and kept either by top-k (SHAPrune) or by using the scores as the relevance of a conditional DPP (SHAPRune-D). On POPE, GQA, MME, and TextVQA, the supervised readout preserves more unpruned answers than FastV and several training-free attention readouts at matched depth, and SHAPRune-D is competitive with CDPruner@8 and pure DPP@8. The paper also reports that other removal-effect labels tie with Shapley labels, that placement after layer 8 explains much of the margin, and that gains are weaker on TextVQA and Qwen2.5-VL.
Critical strengths
-
Placement analysis is independently useful. The paper systematically shows that moving diversity-based pruners (pure DPP, CDPruner) from before the LLM to after layer 8 yields large recall gains, often accounting for most of SHAPrune’s margin. This finding is consequential for VLM pruning regardless of whether removal effects are the better signal.
-
Reusable empirical artifacts and protocol. It releases block Shapley labels, trained heads, per-question outputs, and a detailed evaluation protocol across four benchmarks, three budgets, and multiple baselines. These enable future controlled comparisons with matched-cost learned selectors, which the current paper does not provide.
Critical weaknesses
-
The hypothesis is old and untested. “Removal beats attention” is established (Jain & Wallace, 2019; Covert et al., 2021) and already used by FastSHAP/ViT-Shapley/AT2. Experiments compare supervised labels only with zero-cost training-free readouts, never a learned selector at matched cost, so they show supervision helps—not that removal effects are the better signal. §5.4’s ties further confirm the core claim is never isolated.
-
The fixed, non-overlapping 3×3 partition defines the players, but Shapley values are partition-dependent: supervision therefore measures which geometric windows matter, not which visual entities. Objects spanning blocks are split; object-background blocks share one identity. Block size was chosen for estimator reliability (split-half +0.52 vs +0.11), not semantics. This makes selection coarser than scoring (block averaging discards within-block ranking; k=64 keeps 7 blocks plus one token). TextVQA, the consistent loss, is exactly the fine-grained, small-entity case predicted—yet never analyzed (App. E).
-
As a method that requires offline training (~59 GPU-hours of labels plus a supervised readout), all six published baselines are training-free methods from 2025 or earlier; no contemporary learned selector is compared at matched cost, so the claimed SOTA cannot be verified on fair terms. Architecture coverage is limited to two models: on Qwen2.5-VL-7B the label split-half reliability collapses from +0.52 (LLaVA) to +0.31. The core mechanism may not transfer across architectures. As a method that already incurs offline cost, its efficiency ceiling may be far below trained pre-LLM compressors (LLaVA-Mini ICLR25, VisPCO ACL26), a regime the paper entirely avoids comparing against.
Questions for authors
Please see weaknesses.
We thank the reviewer for the detailed review. Recall means the answer recall of the paper, brackets are 99% intervals of the difference in points, and the new POPE results count its 5,170 distinct image–question pairs.
1. "The hypothesis is old and untested."
We do not claim the idea that removal effects explain a model better than attention; §2, §3 and App. N cite FastSHAP, ViT-Shapley, AT2, Covert et al. and Jain & Wallace. What we add is a controlled test of this idea as a pruning signal for a frozen VLM, and an account of where the gains come from. Pruning after layer 8 explains a large part of the margin over diversity-based pruners, and the revision will list this as a contribution.
On a learned selector at the same cost: the submission has controls with the same head and cost. Trained on labels shuffled across pairs, the head only ties random selection; trained on the average label map, it keeps 86.0/75.4% of POPE/GQA answers. To test a real label that is not a removal effect, we now trained the same head on later-layer attention (layers 8–31, all heads, averaged over blocks), with the same pairs and training. Only the kind of label differs.
The recall gaps are +6.15 ([+5.04, +7.28]) and +13.34 ([+11.89, +14.78]). The accuracy gaps are +3.87 and +6.67, with both intervals above zero. Recall is also higher at 128 and 192 tokens. This is one alternative label and one head seed.
On the ties in §5.4: inside the DPP we also find no difference between single-ablation and Shapley labels (recall 88.01% for Shapley against 87.94% on 10,000 fresh images, [−0.50, +0.63]; CLIP relevance gives 86.58%). The claim we can support is therefore that training the head on how much the answer changes when image tokens are removed gives a better pruning signal than attention, whether the change is measured with Shapley values or with single ablations. The revision will state it this way.
2. The fixed 3×3 partition and TextVQA
The blocks are a grid chosen for reliable labels (App. E), not objects. We ran two checks.
For TextVQA, we split the questions into three equal groups by the area of the OCR boxes that contain the answer (found for 2,631 of 5,000 questions). In these data the 3×3 blocks do not lose more on small text.
3. Training-free baselines, state of the art, backbones and trained compressors
We did not run the published learned selectors. Comparing with them would change the training data, the depth and the features together with the label, whereas the experiment in point 1 changes the label only.
On state of the art: together with the bold entries of Table 3, the "71 of 72" count in §1 can be read as such a claim. We do not make it. The revision moves the count out of §1 and removes the bold.
On the two backbones: the Qwen labels are noisier (+0.312 against +0.515, App. L). Even so, inside the DPP the Qwen head keeps more GQA answers than pure DPP@8 at 64 tokens (81.94 against 79.92%, [+0.71, +3.30]). On POPE we find no difference in recall, while accuracy is higher (90.87 against 89.86%). The signal carries over in part (one head seed).
On LLaVA-Mini and VisPCO: we claim no efficiency advantage over pruning or compressing before the LLM (§6). LLaVA-Mini trains a new model including the LLM, so recall against the frozen model's own answers cannot be computed for it. VisPCO learns how many tokens to keep at each layer for existing in-LLM pruners. It does not choose the importance signal and could be combined with ours. We will add both to the related work.