- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes GroundPrune, a training-free cascaded visual token pruning framework for VLMs. The paper argues that conventional pruning evaluation based only on downstream VQA accuracy can overlook substantial changes in the model’s output distribution and internal representations, particularly for fine-grained grounding tasks. To measure this, the authors introduce output-level KL divergence between the original and pruned models and a representation-level box-probe diagnostic. GroundPrune contains two stages: (1) Visual Evidence Preservation (VEP), which selects tokens using saliency, feature norm, and feature-space diversity; and (2) Behavior-Preserving Decoder Pruning (BPDP), which uses the attention from an offline-selected single routing head for query-conditioned token pruning.
Critical strengths
- The paper raises a meaningful concern that preserving general VQA accuracy alone may not reflect whether token pruning retains fine-grained visual grounding ability.
- The method remains training-free, introduces no learnable parameters, and achieves computational cost and prefill latency comparable to existing pruning baselines.
Critical weaknesses
- The discussion of recent grounding-aware visual token pruning is limited. Only one 2026 paper, Nüwa, is cited. More recent studies on grounding-preserving token reduction should be discussed, such as IVC-Prune: Revealing the Implicit Visual Coordinates in LVLMs for Vision Token Pruning (ICLR 2026).
- The KL-based behavioral preservation objective is not fully justified. Lower KL only means that the pruned model is closer to the vanilla model; it does not imply better accuracy or grounding, since the vanilla model can be wrong.
Questions for authors
- How are the REC bounding-box results obtained for LLaVA-1.5?
- The grounding-specific motivation and contribution are not sufficiently isolated. The proposed methods, VEP (saliency + feature diversity) and BPDP (query-conditioned attention pruning), appear to be generic techniques rather than grounding-specific designs. Could you provide deeper analysis or discussion clarifying exactly why and where the significant gains on the REC task primarily come from? How do these generic mechanisms specifically resolve grounding bottlenecks?
- The paper states that the head is selected once per backbone, but also uses samples “from each benchmark.” It is unclear whether calibration is shared across tasks or tuned separately for each benchmark/retention ratio.
- The paper defines K1 and K2, but does not report values in the experimental setting. What specific values were used for K1 and K2 in the main experimental settings?
Suggestions: Figure 1 is not referenced in the main text and largely duplicates the quantitative results in later tables, so its purpose is unclear. Figure 2 may be more effective if moved to the analysis section and augmented with the corresponding predicted outputs.
2 comments
Response to Reviewer 8FVP (2/2)
Q2. Where do the REC gains come from?
We recorded which tokens each method keeps and compared them with the ground-truth box. This analysis uses LLaVA-1.5-7B with 128 tokens on average and 1,000 RefCOCO val samples. It shows three things.
First, the tokens we keep at the end are on the referred object.
| Method | REC [email protected] | Final kept tokens inside the GT box |
|---|---|---|
| FastV | 8.2 | 17.5% |
| VisionZip | 5.2 | 22.8% |
| Nüwa | 52.7 | 30.5% |
| Ours | 61.0 | 61.0% |
FastV also uses decoder attention, but it averages over all heads at an early layer. Reading one routing head raises the share from 17.5% to 61.0%. This is the effect of BPDP.
Second, VEP keeps more of the target before the query is known. After stage 1, VEP keeps 236 tokens and 61.7% of the target's tokens are still present. Nüwa's stage 1 keeps 224 tokens and 47.3% of the target's tokens are still present. A token removed at this stage cannot be recovered later.
Third, the cascade gives the early layers more tokens at the same cost. In the table below, the average budget is fixed at 128 tokens and only the split between the two stages changes.
| K1 (layers 0–14) | K2 (layers 15–31) | REC [email protected] |
|---|---|---|
| 128 | 128 (no stage 2) | 39.0 |
| 192 | 72 | 54.6 |
| 236 | 33 (paper setting) | 61.0 |
A method that prunes only before the decoder must show 128 tokens to every layer, and it reaches 39.0. The cascade shows 236 tokens to the early layers and 33 tokens to the late layers, and it reaches 61.0. Accuracy rises as more tokens are given to the early layers, so the late layers need few visual tokens for grounding.
In summary, REC fails under pruning in two cases. In the first case, the target's tokens are removed before the query is known. In the second case, the tokens kept in the decoder are not the ones the query refers to. VEP addresses the first case and BPDP addresses the second. General VQA questions mostly concern salient content, so saliency-based methods keep VQA accuracy while failing in both cases.
Q3. Is the calibration shared across tasks, or tuned per benchmark and ratio?
The calibration is shared.
- We use one head per backbone, which is (21, 5) for Qwen2.5-VL-7B and (14, 24) for LLaVA-1.5-7B. We choose it once at one budget, which is R = 25% for Qwen2.5-VL-7B and R = 22.2% for LLaVA-1.5-7B.
- We use the same head for every benchmark and every retention ratio.
- To choose the head, we prune along each candidate head and measure D_out on 7 VQA benchmarks and S_rep on 8 REC splits. We use 500 random samples per benchmark.
- We do not use the labels of these samples. Both measures only compare the pruned model with the original model, and accuracy is never computed during the selection. The box probe is fit on the RefCOCO training split.
The head also works on a dataset that was not used in the selection. The table below reports [email protected] on the val and test splits of Ref-L4 with the head fixed.
| Backbone | R | SparseVLM | Nüwa | Ours | No pruning |
|---|---|---|---|---|---|
| Qwen2.5-VL-7B | 25% | 30.87 / 31.55 | 71.00 / 70.36 | 71.20 / 71.39 | 85.42 / 85.81 |
| 50% | 71.96 / 71.93 | 82.27 / 82.05 | 82.71 / 82.79 | ||
| 75% | 83.19 / 83.19 | 84.09 / 84.33 | 84.85 / 85.50 | ||
| LLaVA-1.5-7B | 11.1% | 0.79 / 0.82 | 21.97 / 21.63 | 22.81 / 22.03 | 40.50 / 40.38 |
| 22.2% | 9.33 / 9.54 | 37.32 / 37.22 | 44.18 / 43.51 | ||
| 33.3% | 17.68 / 17.94 | 43.00 / 42.54 | 49.81 / 49.17 |
Q4. What values of K1 and K2 are used?
K1 = ⌊r·T_v⌋ and K2 = ⌊λ·K1⌋. The table below lists the values for each backbone and budget.
| Backbone | Budget | K1 | K2 |
|---|---|---|---|
| Qwen2.5-VL-7B | R = 75% | 0.85·T_v | 0.45·K1 |
| R = 50% | 0.60·T_v | 0.22·K1 | |
| R = 25% | 0.31·T_v | 0.18·K1 | |
| LLaVA-1.5-7B (T_v = 576) | 192 tokens | 373 | 31 |
| 128 tokens | 236 | 31 | |
| 64 tokens | 100 | 31 | |
| LLaVA-NeXT-7B | R = 22.2% | 0.371·T_v | 0.10·K1 |
| R = 11.1% | 0.190·T_v | 0.10·K1 | |
| R = 5.6% | 0.085·T_v | 0.10·K1 |
T_v varies per image for Qwen2.5-VL and LLaVA-NeXT, so K1 is given as a fraction of T_v. We will add this table to the appendix.
Suggestions on Figures 1 and 2
Thank you for your suggestions. We will reference Figure 1 in the Introduction. We will move Figure 2 to the analysis section and add the predicted boxes to it.
Response to Reviewer 8FVP (1/2)
Thank you for the review. We answer each point below.
W1. Limited discussion of grounding-aware pruning (e.g., IVC-Prune)
We will add IVC-Prune, GAP, and LiteLVLM to Sec. 2.2, next to Nüwa and VScan. Our method differs from IVC-Prune in three ways:
We could not run IVC-Prune under our protocol during the rebuttal period. The token counts of the two papers are also not directly comparable. As we understand it, the percentage reported by IVC-Prune counts the tokens kept after the selection layer, while all tokens are processed before that layer. Our R is the average over all layers, and the IVC-Prune setting corresponds to R ≈ 79% under our Eq. 1. At that setting IVC-Prune reports 99.6% relative REC accuracy on Qwen2.5-VL-7B, and we obtain 99.8% at R = 75%. We will add a comparison with matched counting to the paper.
W2. The KL objective is not justified; the vanilla model can be wrong
KL does not replace accuracy in our paper. Accuracy remains our main evaluation metric, and we report it in every table of the main results. We use KL only to choose the routing head without labels, and we report it as an additional measurement.
Closeness to the original model is the right target for a training-free pruning method. Such a method adds no parameters and no supervision, so it cannot know more than the original model.
We also measured this on real outputs. We compared each pruned model with the original Qwen2.5-VL-7B at R = 25%, question by question. In the table below, "Lost" counts the questions that the original answers correctly and the pruned model answers wrongly. "Fixed" counts the opposite case.
The vanilla model is sometimes wrong, and pruning does fix some of these cases. However, the number of fixed questions is small and almost the same for every method. On GQA it is between 346 and 536. The number of lost questions differs much more between methods. On GQA it is between 474 and 1,880, and it is always larger than the number of fixed questions. The accuracy of a pruned model is therefore decided by how many correct answers of the original it loses.
Two more observations support this. When a pruned model gives a different answer from the original, that answer is correct in only 14% to 37% of the cases. Methods that agree more with the original are also more accurate on GQA, VQAv2, POPE, MME, and ScienceQA. The one exception is MMBench, where Nüwa agrees slightly less than ours (92.2% vs 92.9%) but is slightly more accurate (80.5 vs 79.6).
Q1. How are REC boxes obtained for LLaVA-1.5?
LLaVA-1.5 outputs a box as text. We use the prompt of the lmms-eval
refcoco_bbox_rectask, which is "Bounding box coordinates are specified in the format (top-left x, top-left y, bottom-right x, bottom-right y). All values are floating point numbers bounded between 0 and 1. Please provide the bounding box coordinate of the region this sentence describes: ⟨expression⟩". We use greedy decoding. We take the first box in the output, scale it by the image width and height, and report [email protected]. We use the same protocol for the original model and for all baselines.