acceptodds
Under review as a conference paper at ICLR 2027

GroundPrune: Representation-Preserving Visual Token Pruning for Grounding in VLMs

Abstract

Visual token pruning reduces the inference cost of vision-language models (VLMs), and existing methods are evaluated by whether downstream answer accuracy is preserved. We argue this criterion is insufficient: a pruned model can still answer correctly while its internal representation has already diverged from that of the original model. To measure this directly, we compare pruned and original models at two levels: the KL divergence between their next-token distributions on general VQA, and the IoU between the boxes that a linear probe, trained on the original model, regresses from the last prompt token's hidden state of the pruned and of the original model. Across FastV, SparseVLM, VisionZip, and Nüwa, methods that retain most VQA accuracy still incur substantial distribution shift and degraded box decodability, an erosion masked by VQA accuracy but amplified on referring expression comprehension (REC). We trace it to two causes: saliency mode collapse at the vision stage, where retained tokens cluster around a few dominant hotspots, and the choice of routing point in the decoder, where only a few attention heads can guide pruning without disturbing the original model. We propose GroundPrune, a training-free cascaded framework that addresses each in turn: Visual Evidence Preservation (VEP) selects feature-diverse tokens from a saliency-guided candidate pool, and Behavior-Preserving Decoder Pruning (BPDP) prunes inside the decoder using the attention of a single routing head, located offline as the one whose pruning keeps the model closest to the original, with no supervision at inference. Across eight REC subsets and eight VQA benchmarks, GroundPrune attains the lowest KL divergence and probe error among compared methods; at 25% retention on Qwen2.5-VL-7B it reaches 89.7% relative REC accuracy, 5.1 points above the strongest baseline, while keeping 96.1% relative VQA accuracy, with consistent gains on LLaVA-1.5-7B and LLaVA-NeXT-7B.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
visual token pruning, vision-language models, referring expression comprehension, representation drift, linear probing, efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8FVPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GroundPrune, a training-free cascaded visual token pruning framework for VLMs. The paper argues that conventional pruning evaluation based only on downstream VQA accuracy can overlook substantial changes in the model’s output distribution and internal representations, particularly for fine-grained grounding tasks. To measure this, the authors introduce output-level KL divergence between the original and pruned models and a representation-level box-probe diagnostic. GroundPrune contains two stages: (1) Visual Evidence Preservation (VEP), which selects tokens using saliency, feature norm, and feature-space diversity; and (2) Behavior-Preserving Decoder Pruning (BPDP), which uses the attention from an offline-selected single routing head for query-conditioned token pruning.

Critical strengths
  1. The paper raises a meaningful concern that preserving general VQA accuracy alone may not reflect whether token pruning retains fine-grained visual grounding ability.
  2. The method remains training-free, introduces no learnable parameters, and achieves computational cost and prefill latency comparable to existing pruning baselines.
Critical weaknesses
  1. The discussion of recent grounding-aware visual token pruning is limited. Only one 2026 paper, Nüwa, is cited. More recent studies on grounding-preserving token reduction should be discussed, such as IVC-Prune: Revealing the Implicit Visual Coordinates in LVLMs for Vision Token Pruning (ICLR 2026).
  2. The KL-based behavioral preservation objective is not fully justified. Lower KL only means that the pruned model is closer to the vanilla model; it does not imply better accuracy or grounding, since the vanilla model can be wrong.
Questions for authors
  1. How are the REC bounding-box results obtained for LLaVA-1.5?
  2. The grounding-specific motivation and contribution are not sufficiently isolated. The proposed methods, VEP (saliency + feature diversity) and BPDP (query-conditioned attention pruning), appear to be generic techniques rather than grounding-specific designs. Could you provide deeper analysis or discussion clarifying exactly why and where the significant gains on the REC task primarily come from? How do these generic mechanisms specifically resolve grounding bottlenecks?
  3. The paper states that the head is selected once per backbone, but also uses samples “from each benchmark.” It is unclear whether calibration is shared across tasks or tuned separately for each benchmark/retention ratio.
  4. The paper defines K1​ and K2, but does not report values in the experimental setting. What specific values were used for K1​ and K2 in the main experimental settings?

Suggestions: Figure 1 is not referenced in the main text and largely duplicates the quantitative results in later tables, so its purpose is unclear. Figure 2 may be more effective if moved to the analysis section and augmented with the corresponding predicted outputs.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 8FVP (1/2)

Thank you for the review. We answer each point below.


W1. Limited discussion of grounding-aware pruning (e.g., IVC-Prune)

We will add IVC-Prune, GAP, and LiteLVLM to Sec. 2.2, next to Nüwa and VScan. Our method differs from IVC-Prune in three ways:

  • IVC-Prune keeps token positions derived from the RoPE angles, together with foreground tokens found by value-vector similarity. We keep feature-diverse tokens before the decoder, and then the tokens attended by one decoder head.
  • IVC-Prune removes tokens once, at a middle decoder layer. We remove tokens before the decoder and again inside it.
  • IVC-Prune reports grounding at about 50% of the tokens. We report 75%, 50%, and 25% on Qwen2.5-VL, and down to 11.1% on LLaVA-1.5.

We could not run IVC-Prune under our protocol during the rebuttal period. The token counts of the two papers are also not directly comparable. As we understand it, the percentage reported by IVC-Prune counts the tokens kept after the selection layer, while all tokens are processed before that layer. Our R is the average over all layers, and the IVC-Prune setting corresponds to R ≈ 79% under our Eq. 1. At that setting IVC-Prune reports 99.6% relative REC accuracy on Qwen2.5-VL-7B, and we obtain 99.8% at R = 75%. We will add a comparison with matched counting to the paper.


W2. The KL objective is not justified; the vanilla model can be wrong

KL does not replace accuracy in our paper. Accuracy remains our main evaluation metric, and we report it in every table of the main results. We use KL only to choose the routing head without labels, and we report it as an additional measurement.

Closeness to the original model is the right target for a training-free pruning method. Such a method adds no parameters and no supervision, so it cannot know more than the original model.

We also measured this on real outputs. We compared each pruned model with the original Qwen2.5-VL-7B at R = 25%, question by question. In the table below, "Lost" counts the questions that the original answers correctly and the pruned model answers wrongly. "Fixed" counts the opposite case.

BenchmarkMethodSame answer as originalAccuracyLostFixed
GQAFastV69.4%50.21,880536
SparseVLM73.5%52.41,557493
Nüwa83.3%58.5769461
Ours87.7%59.9474346
VQAv2FastV71.0%70.729,9844,879
SparseVLM75.9%74.122,7694,816
Nüwa84.3%79.910,4524,505
Ours86.8%80.48,8933,785

The vanilla model is sometimes wrong, and pruning does fix some of these cases. However, the number of fixed questions is small and almost the same for every method. On GQA it is between 346 and 536. The number of lost questions differs much more between methods. On GQA it is between 474 and 1,880, and it is always larger than the number of fixed questions. The accuracy of a pruned model is therefore decided by how many correct answers of the original it loses.

Two more observations support this. When a pruned model gives a different answer from the original, that answer is correct in only 14% to 37% of the cases. Methods that agree more with the original are also more accurate on GQA, VQAv2, POPE, MME, and ScienceQA. The one exception is MMBench, where Nüwa agrees slightly less than ours (92.2% vs 92.9%) but is slightly more accurate (80.5 vs 79.6).


Q1. How are REC boxes obtained for LLaVA-1.5?

LLaVA-1.5 outputs a box as text. We use the prompt of the lmms-eval refcoco_bbox_rec task, which is "Bounding box coordinates are specified in the format (top-left x, top-left y, bottom-right x, bottom-right y). All values are floating point numbers bounded between 0 and 1. Please provide the bounding box coordinate of the region this sentence describes: ⟨expression⟩". We use greedy decoding. We take the first box in the output, scale it by the image width and height, and report [email protected]. We use the same protocol for the original model and for all baselines.

AuthorsOpenReview ↗

Response to Reviewer 8FVP (2/2)

Q2. Where do the REC gains come from?

We recorded which tokens each method keeps and compared them with the ground-truth box. This analysis uses LLaVA-1.5-7B with 128 tokens on average and 1,000 RefCOCO val samples. It shows three things.

First, the tokens we keep at the end are on the referred object.

MethodREC [email protected]Final kept tokens inside the GT box
FastV8.217.5%
VisionZip5.222.8%
Nüwa52.730.5%
Ours61.061.0%

FastV also uses decoder attention, but it averages over all heads at an early layer. Reading one routing head raises the share from 17.5% to 61.0%. This is the effect of BPDP.

Second, VEP keeps more of the target before the query is known. After stage 1, VEP keeps 236 tokens and 61.7% of the target's tokens are still present. Nüwa's stage 1 keeps 224 tokens and 47.3% of the target's tokens are still present. A token removed at this stage cannot be recovered later.

Third, the cascade gives the early layers more tokens at the same cost. In the table below, the average budget is fixed at 128 tokens and only the split between the two stages changes.

K1 (layers 0–14)K2 (layers 15–31)REC [email protected]
128128 (no stage 2)39.0
1927254.6
23633 (paper setting)61.0

A method that prunes only before the decoder must show 128 tokens to every layer, and it reaches 39.0. The cascade shows 236 tokens to the early layers and 33 tokens to the late layers, and it reaches 61.0. Accuracy rises as more tokens are given to the early layers, so the late layers need few visual tokens for grounding.

In summary, REC fails under pruning in two cases. In the first case, the target's tokens are removed before the query is known. In the second case, the tokens kept in the decoder are not the ones the query refers to. VEP addresses the first case and BPDP addresses the second. General VQA questions mostly concern salient content, so saliency-based methods keep VQA accuracy while failing in both cases.


Q3. Is the calibration shared across tasks, or tuned per benchmark and ratio?

The calibration is shared.

  • We use one head per backbone, which is (21, 5) for Qwen2.5-VL-7B and (14, 24) for LLaVA-1.5-7B. We choose it once at one budget, which is R = 25% for Qwen2.5-VL-7B and R = 22.2% for LLaVA-1.5-7B.
  • We use the same head for every benchmark and every retention ratio.
  • To choose the head, we prune along each candidate head and measure D_out on 7 VQA benchmarks and S_rep on 8 REC splits. We use 500 random samples per benchmark.
  • We do not use the labels of these samples. Both measures only compare the pruned model with the original model, and accuracy is never computed during the selection. The box probe is fit on the RefCOCO training split.

The head also works on a dataset that was not used in the selection. The table below reports [email protected] on the val and test splits of Ref-L4 with the head fixed.

BackboneRSparseVLMNüwaOursNo pruning
Qwen2.5-VL-7B25%30.87 / 31.5571.00 / 70.3671.20 / 71.3985.42 / 85.81
50%71.96 / 71.9382.27 / 82.0582.71 / 82.79
75%83.19 / 83.1984.09 / 84.3384.85 / 85.50
LLaVA-1.5-7B11.1%0.79 / 0.8221.97 / 21.6322.81 / 22.0340.50 / 40.38
22.2%9.33 / 9.5437.32 / 37.2244.18 / 43.51
33.3%17.68 / 17.9443.00 / 42.5449.81 / 49.17

Q4. What values of K1 and K2 are used?

K1 = ⌊r·T_v⌋ and K2 = ⌊λ·K1⌋. The table below lists the values for each backbone and budget.

BackboneBudgetK1K2
Qwen2.5-VL-7BR = 75%0.85·T_v0.45·K1
R = 50%0.60·T_v0.22·K1
R = 25%0.31·T_v0.18·K1
LLaVA-1.5-7B (T_v = 576)192 tokens37331
128 tokens23631
64 tokens10031
LLaVA-NeXT-7BR = 22.2%0.371·T_v0.10·K1
R = 11.1%0.190·T_v0.10·K1
R = 5.6%0.085·T_v0.10·K1

T_v varies per image for Qwen2.5-VL and LLaVA-NeXT, so K1 is given as a fraction of T_v. We will add this table to the appendix.


Suggestions on Figures 1 and 2

Thank you for your suggestions. We will reference Figure 1 in the Introduction. We will move Figure 2 to the analysis section and add the predicted boxes to it.

Reviewer_YRZzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper argues that visual token pruning in VLMs should be evaluated by how faithfully the pruned model preserves the original model's behavior, rather than by answer accuracy alone. To measure this, the authors propose a two-level diagnostic: the KL divergence between teacher-forced next-token distributions of the pruned and original models on general VQA (DoutD_{\text{out}}), and a linear probe, trained on the original model, that regresses the referred box from the last prompt token's final-layer hidden state (SrepS_{\text{rep}}). With this lens they show that methods retaining most VQA accuracy can still collapse on REC; for example, FastV keeps 87.6% relative VQA accuracy on Qwen2.5-VL-7B at 25% retention but only 26.9% relative REC accuracy. They attribute this damage to saliency mode collapse at the vision stage and to a poor choice of routing point in the decoder, and propose GroundPrune, a training-free two-stage pruner. The first stage (VEP) builds a candidate pool of the top-2K12K_1 tokens scored by vision-attention saliency times feature norm, then picks K1K_1 feature-diverse tokens via farthest point sampling. The second stage (BPDP) prunes inside the decoder using the last-query attention of a single head (L,h)(L, h), selected offline per backbone by minimizing the rank sum of DoutD_{\text{out}} and SrepS_{\text{rep}} over all layer–head pairs. Experiments on Qwen2.5-VL-7B, LLaVA-1.5-7B and LLaVA-NeXT-7B (REC only), covering 8 RefCOCO-family splits and 8 VQA benchmarks against FastV, SparseVLM, VisionZip and Nüwa, report 89.7% relative REC and 96.1% relative VQA on Qwen2.5-VL-7B at RR=25% (5.1 points above Nüwa on REC), large REC gains on LLaVA-1.5 (up to 119.9% relative REC), and efficiency comparable to the baselines. Ablations on the VEP selection rule and the BPDP routing point are also provided.

Critical strengths
  1. A well-documented and practically important failure mode. The paper shows convincingly that standard pruning methods that look nearly lossless on general VQA can be catastrophic on localization. Examples: FastV at 87.6% VQA vs. 26.9% REC on Qwen2.5-VL at RR=25%; VisionZip at 98.3% VQA vs. 12.4% REC on LLaVA-1.5 at 192 tokens. Framing pruning as fidelity to the original model, measured at both the output level (KL) and the representation level (box probe), is a sensible evaluation lens that the community could adopt. This is the main reason the paper is of interest.

  2. Consistent REC improvements with little VQA cost, at negligible inference overhead. On Qwen2.5-VL-7B the gap over the strongest baseline (Nüwa) widens as the budget shrinks: +5.1 points relative REC at RR=25%. Gains also appear on LLaVA-NeXT. The method is simple (score + FPS, plus one attention head) and training-free.

Critical weaknesses
  1. The routing head is selected on the evaluation benchmarks, using the same metrics later used to claim superiority. Sec. 3.4 states that DoutD_{\text{out}} and SrepS_{\text{rep}} are measured "on data randomly sampled from each benchmark," i.e., the same benchmarks reported in Tables 1–4. The claims that GroundPrune "attains the lowest KL divergence and probe error" and that "distribution shift tracks accuracy" are therefore partly true by construction. With 784–1024 candidate heads and an unspecified sample size, overfitting to the selection set is a real concern. It is also unclear whether the selection samples are disjoint from the evaluation splits, and whether the head is re-selected per budget. A held-out selection set and a cross-budget/cross-dataset transfer study are needed before the gains can be trusted.

  2. Baseline results appear unreliable and evaluated under inconsistent protocols. On LLaVA-1.5, FastV, SparseVLM and VisionZip score 52.5, 56.1 and 57.3 on TextVQA at 192 tokens, above the unpruned model (46.0), while GroundPrune and Nüwa score below it. The baseline VQA rows appear to coincide with numbers reported in the SparseVLM and VisionZip papers, which (I believe) use a different TextVQA protocol, so the relative-VQA columns may mix protocols. In addition, VisionZip's REC on LLaVA-1.5 is implausibly low (6.2 on RefCOCO val at 192 tokens, below FastV, and nearly flat from 192 to 64 tokens), which suggests an implementation issue. All baselines should be re-run in the same pipeline as the vanilla model.

  3. Relative REC far above 100% contradicts the paper's own thesis. On LLaVA-1.5, GroundPrune reaches 119.9% (192 tokens) and 109.6% (128 tokens) relative REC, and Nüwa 105.1%. A method whose goal is to follow the original model should not beat it by 20 points on grounding. This points either to an evaluation/output-parsing artifact or to the method substantially changing the model's behavior, and the paper does not discuss it.

  4. The proposed diagnostic does not track accuracy on LLaVA-1.5. At 22.2% retention, GroundPrune and Nüwa have nearly identical probe IoU (34.98 vs. 34.89), yet their relative REC differs by 16.3 points. VisionZip has lower average KL than FastV and SparseVLM but by far the worst REC. The claim that preserving the original behavior carries over to accuracy therefore rests mainly on five methods on a single backbone (Qwen2.5-VL).

  5. The contributions are overstated. The claims of "lowest KL in every setting" and "best or tied-best REC and VQA at every token budget" do not hold at the per-benchmark level: KL is not the lowest on MMMU or TextVQA, LLaVA-1.5 RefCOCO probe IoU is lower than Nüwa's (38.35 vs. 39.18), MME at 64 tokens drops to 1538.5 vs. about 1676–1690 for VisionZip and Nüwa, and LLaVA-NeXT has no VQA results and only one baseline.

  6. Limited novelty and missing comparisons. VEP is close to DivPrune combined with VisionZip/FasterVLM-style saliency, yet DivPrune is cited but not compared. Single-head selection relates to prior work showing that a few "localization heads" suffice for grounding in LVLMs (e.g., Kang et al., CVPR 2025), which is not discussed. PyramidDrop, VScan and other recent baselines are also missing.

Questions for authors
  1. Head-selection protocol. Which data, and how many samples per benchmark, are used to select (L⋆,h⋆)(L^\star, h^\star)? Are they disjoint from all reported evaluation splits, including the RefCOCO val/test splits? Is the head re-selected for each retention ratio? Results with the head selected only on a held-out training split, plus a cross-budget/cross-dataset transfer check, would most directly affect my assessment.

  2. Baseline provenance and relative REC above 100%. Were all baseline numbers obtained by re-running each method in your own lmms-eval pipeline, or were some taken from the original papers? Please explain why FastV, SparseVLM and VisionZip exceed the vanilla model on TextVQA on LLaVA-1.5, why VisionZip's REC is nearly zero, and why GroundPrune reaches 119.9% / 109.6% relative REC. Is the vanilla model evaluated with the same prompt, output parsing and box format as the pruned models?

  3. Hyperparameters and budget matching. What are rr, λ\lambda and LL for each budget and backbone, and how is each baseline matched to the effective RR of Eq. 1 (e.g., FastV keeps all tokens in its early layers)?

  4. Ablations isolating the routing head. Table 7 omits the head-averaged attention at layer 14 itself. Please add it, together with a random head from the same layer and BPDP without VEP, so that the benefit of the selected single head can be separated from the choice of layer and from VEP.

  5. Efficiency at high resolution. VEP needs full vision self-attention maps and BPDP needs explicit attention probabilities, which is incompatible with FlashAttention. What are the latency and memory overheads on Qwen2.5-VL with many visual tokens, and how many GPU-hours does the offline head search take?

2 comments
AuthorsOpenReview ↗

Response to Reviewer YRZz (1/2)

Thank you for the review. We answer each point below.

Q1. Head-selection protocol. For each candidate head we prune along it at one budget (R = 25% for Qwen2.5-VL, 22.2% for LLaVA-1.5) and measure D_out on the seven VQA benchmarks and S_rep on the eight REC splits; every head is scored on 500 random samples per benchmark. The procedure does not search for the head that maximizes accuracy; it searches for the head that best preserves the behavior of the original model. Neither criterion uses the labels of these samples: they only compare the pruned model's output distribution and hidden state with those of the original model. It is therefore not a matter of evaluation labels leaking into the selection, and answer accuracy and [email protected] are not what is optimized. The head is selected once per backbone and reused at every budget, so the 50% and 75% columns of Table 2 are cross-budget transfer. It also transfers across datasets. Table R1 below reports [email protected] (val / test) on Ref-L4 (longer expressions, categories beyond COCO) with the head unchanged.

BackboneRSparseVLMNüwaOursNo pruning
Qwen2.5-VL-7B25%30.87 / 31.5571.00 / 70.3671.20 / 71.3985.42 / 85.81
50%71.96 / 71.9382.27 / 82.0582.71 / 82.79
75%83.19 / 83.1984.09 / 84.3384.85 / 85.50
LLaVA-1.5-7B11.1%0.79 / 0.8221.97 / 21.6322.81 / 22.0340.50 / 40.38
22.2%9.33 / 9.5437.32 / 37.2244.18 / 43.51
33.3%17.68 / 17.9443.00 / 42.5449.81 / 49.17

On the "true by construction" concern: Table 1 of the paper (the KL / probe comparison across methods) is measured on the full benchmarks, not only on the selection samples, and its REC column is IoU with the ground-truth box, not the selection metric S_rep.

Q2. Baseline provenance and relative REC above 100%. Provenance. All REC numbers were produced by running each method ourselves (VisionZip and Nüwa with their released code). You are right about one part: the LLaVA-1.5 VQA rows of FastV, SparseVLM and VisionZip were taken from the VisionZip paper, which uses the official TextVQA protocol with OCR tokens (unpruned: 58.2), whereas our runs use lmms-eval (46.0). That is why they exceed our unpruned model on TextVQA. Normalizing each row by the unpruned model of its own protocol leaves the ordering unchanged (ours 99.0 / 98.5 / 95.0 vs. the best baseline 99.0 / 97.8 / 94.1 at 192 / 128 / 64 tokens), and we will re-run these rows in a common pipeline for the revision. VisionZip REC. This is an independently published result. Nüwa (ICLR 2026, Table 6) reports VisionZip on LLaVA-1.5 at 8.1% / 7.28% of the unpruned model with 128 / 64 tokens (ours: 8.9% / 7.2%), likewise below FastV and flat across budgets, and attributes it to VisionZip re-assigning token positions, which discards the image coordinate frame. Above 100%. This comes from how the unpruned LLaVA-1.5 baseline was scored: LLaVA-1.5 pads the image to a square, and our unpruned run did not account for the padding when mapping boxes back, which lowers the denominator. We could not finish the re-measurement in time and will report the corrected unpruned numbers and relative REC. The ranking of methods remains valid: in [email protected] averaged over the eight REC columns of Table 3, ours scores 62.7 / 57.3 / 31.2 against Nüwa's 55.0 / 48.8 / 29.6, under the same prompt, output parser and box format. Qwen2.5-VL does not pad images and never exceeds 100%.

Q3. Hyperparameters and budget matching. R is the number of visual tokens the LLM processes, averaged over decoder layers and divided by the original count, the same accounting used for in-decoder pruning methods; a fraction r is kept at the vision stage and a fraction λ of those after layer L (Eq. 1). (r, λ) = (0.85, 0.45) / (0.60, 0.22) / (0.31, 0.18) for Qwen2.5-VL at 75 / 50 / 25% (L = 21), and (0.648, 0.086) / (0.411, 0.135) / (0.174, 0.319) for LLaVA-1.5 at 192 / 128 / 64 tokens (L = 14). Baselines are matched on the same layer-averaged count, so tokens kept in early layers are charged: FastV (K = 3) with keep ratio ρ = (R·N_L − K)/(N_L − K), SparseVLM solved the same way, VisionZip by its token count, and Nüwa with its released configuration. Table 5 confirms the match end to end: 2.59 TFLOPs and 39 ms for ours against 2.57–2.84 TFLOPs and 33–42 ms for the pruning baselines, so ours falls within their range.

AuthorsOpenReview ↗

Response to Reviewer YRZz (2/2)

Q4. Ablations. The requested row is already in Table 7; we mixed 1- and 0-indexing. "Layer 15, heads Avg." is the head-averaged attention of 0-indexed layer 14 with the same budget split as ours. Against it, our routing signal gains 4.7 / 3.4 / 4.2 / 3.7 / 3.1 points on the five REC columns at comparable VQA accuracy (within 0.2 points). Table R2 below is Table 7 with consistent 0-indexing.

Routing signal (0-indexed attention layer)GQAMMBSQARefCOCO testARefCOCO testBRefCOCO+ testARefCOCO+ testBRefCOCOg test
Layer 4, heads avg.50.254.370.21.80.81.60.51.9
Layer 9, heads avg.54.059.167.510.87.99.67.38.7
Layer 14, heads avg.60.963.568.167.248.461.438.552.2
Layer 19, heads avg.60.763.168.861.740.056.333.245.4
Layer 24, heads avg.60.363.168.856.936.651.130.041.9
Layer 29, heads avg.59.663.068.453.031.747.226.437.5
Ours (Layer 14, head 24)60.763.768.371.951.865.642.255.3

We will add a row for a random head of the same layer in the revision.

BPDP without VEP cannot reach 22.2%: with r = 1, Eq. 1 would need λ = −0.46, and the minimum is 46.9% (78.6% on Qwen2.5-VL). Table R3 below compares VEP only, BPDP only and VEP+BPDP at R = 25 / 50 / 85%. BPDP only is omitted where it cannot reach the budget because of the lower bound above (R = 25% for LLaVA-1.5; R = 25% and 50% for Qwen2.5-VL), which is also why the upper budget is 85% rather than 75%.

LLaVA-1.5-7B

RVEPBPDPGQAMMBSQARC-tARC-tBRC+-tARC+-tBRCgavg
25.0✓60.063.468.146.326.040.320.533.844.8
25.0✓✓60.963.668.361.645.253.735.550.454.9
50.0✓61.363.468.361.844.953.635.550.054.8
50.0✓60.863.969.465.546.158.236.652.456.6
50.0✓✓61.264.069.565.146.558.136.352.556.7
85.0✓61.864.169.061.244.454.134.750.554.9
85.0✓61.764.169.558.342.151.732.848.153.5
85.0✓✓62.264.169.361.144.453.934.650.455.0

Qwen2.5-VL-7B

RVEPBPDPGQAMMBSQARC-tARC-tBRC+-tARC+-tBRCgavg
25.0✓59.578.684.580.265.975.357.469.771.4
25.0✓✓60.178.784.886.173.580.963.175.675.4
50.0✓60.181.186.689.679.984.970.281.979.2
50.0✓✓60.381.886.891.081.986.272.784.280.6
85.0✓60.683.088.091.384.286.873.685.781.6
85.0✓60.882.688.491.984.187.774.386.082.0
85.0✓✓60.983.088.291.884.187.774.286.382.1

At every budget the average of VEP+BPDP matches or exceeds either stage alone, and the gap is largest under strong pruning (at R = 25%, +10.1 on LLaVA-1.5 and +4.0 on Qwen2.5-VL over VEP only).

Q5. Efficiency and FlashAttention. The decoder runs with FlashAttention (SDPA on LLaVA) at every layer, including the routing layer. There BPDP only computes one extra row, a = softmax(q_last·Kᵀ/√d) for the routing head, from the existing query and key projections: O(T·d), no T×T map and no extra forward pass. It adds 1.7 ms, about 5% of the pruned prefill time. In memory, even for a 16,384-token input (3584 × 3584) at R = 25%, BPDP adds under 0.1 GB, a negligible load. VEP reads the attention of a single vision block (on LLaVA the CLIP [CLS] attention, as VisionZip and Nüwa do). The head search is offline and one-time, about 25 GPU-hours per backbone.

Reviewer_MckXOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes GroundPrune, a method that not only compares the KL divergence between next-token distributions on general VQA, but more importantly, the IoU between the boxes that a linear probe, trained on the original model, regresses from the last prompt token’s hidden state of the pruned and of the original model. This shows the deficiency of current pruning methods on preserving crucial details that helps localization.

Critical strengths

The identification of discrepancy between pruned models and original models on referring expression comprehension task is important, in contrast to general VQA, REC requires more careful visual token pruning strategy, and GroundPrune's performance is superior compared with other methods.

Critical weaknesses
  1. Using attention score each token receives from [CLS] token is not new at all, many works pruning before LLMs typically has explored this usage, and the FPS sampling in Eq. 6 is similar to "DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models".
  2. Prior work "Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding" identifies layer 14, head 24 as its highest-ranked localization head in LLaVA-1.5-7B, though with different methods, GroundPrune has identified exactly the same head as routing point, it is necessary to compare with prior work to demonstrate the difference during selection, and why we need a new way for selection.
  3. The writing is not well formatted and not met with conference standard, with messing logics, misaligned tables and captions, etc.
Questions for authors

See weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer MckX

Thank you for the review. We answer each weakness below.


W1. [CLS] attention is not new, and the FPS in Eq. 6 is similar to DivPrune

The max–min step in Eq. 6 is the same step DivPrune uses, and [CLS] attention is used in prior work. We will state this in Sec. 3.2. The method differs from DivPrune in two ways:

  • DivPrune selects from all visual tokens with no importance score. We select inside a pool of the tokens with the highest saliency score.
  • DivPrune prunes once, before the decoder and before the query is seen. GroundPrune prunes a second time inside the decoder, using the query.

DivPrune's paper does not report grounding results. We evaluated DivPrune on REC with LLaVA-1.5-7B under the same budget as our method, which is 128 visual tokens on average. The table below reports [email protected] on 1,000 random samples from each split.

MethodRefCOCO valRefCOCO+ val
[CLS]-attention top-k28.3–
DivPrune42.436.9
GroundPrune61.054.8
Original model (576 tokens)55.7–

GroundPrune is 18.6 and 17.9 points above DivPrune at the same budget. Selection before the decoder alone stays below the original model (42.4 vs 55.7). The gain comes from the decoder stage, which is the main contribution of the paper.


W2. Kang et al. already find L14-H24 in LLaVA-1.5-7B. Compare, and explain why a new selection is needed

We will cite this work and add the comparison below. The rule of Kang et al. and ours select the same head on LLaVA-1.5-7B. On Qwen2.5-VL-7B, however, the two rules select different heads, and our experiments confirm that our head is better. Pruning along our head is 13 to 15 points higher on RefCOCO val than pruning along any of the heads from Kang et al.'s rule. The two rules differ because they measure different things:

  • Kang et al. score a head by its attention map. A head is selected when it puts enough attention on the image and the attention is spatially compact. The selected heads are then used to read a box from the map. The paper of Kang et al. does not prune tokens, does not evaluate VQA, and does not test Qwen models.
  • We score a head by how much the model's output changes when we prune along it. We measure this on both VQA and REC.

We re-implemented Kang et al.'s rule with 1,000 RefCOCO training samples. As in the released code of Kang et al., the first two layers are excluded. On LLaVA-1.5-7B, the rule returns (14, 24) as the first head, which matches the paper of Kang et al. and our selection. On Qwen2.5-VL-7B, the rule returns (7, 19), (2, 21), and (10, 6). Our head (21, 5) is never selected, because its attention sum on image tokens is 0.057 and the threshold of Kang et al.'s rule is 0.113.

We then pruned along the heads from Kang et al.'s rule on Qwen2.5-VL-7B. Every row in the table below uses the same budget of R = 25% and the same stage 1.

Routing headRank under Kang et al.'s ruleRefCOCO val (n = 1,000)RefCOCO+ val (n = 500)GQA acc. (n = 2,000)GQA: same answer as original
(7, 19)163.963.658.583.8%
(2, 21)263.1–––
(10, 6)365.2–––
Mean of the three–64.059.258.985.9%
(21, 5), oursnot selected by Kang et al.'s rule78.574.860.088.2%

The heads from Kang et al.'s rule are 13 to 15 points below ours on RefCOCO val and 11 to 16 points below on RefCOCO+ val. These heads also change more of the original model's answers on GQA.

This is why a new selection is needed. A head with a compact attention map is good for reading a box from the map. This does not mean that the model's output is preserved when we prune along that head. On Qwen2.5-VL-7B, only a rule that measures the output of the pruned model finds a head that works.


W3. Writing and formatting

Thank you for pointing this out. We will revise the paper as follows:

  • Figure 1 is not referenced in the text. We will reference it in the Introduction.
  • Tables 4 and 5 are placed side by side and their captions run together. We will separate them.
  • The citations in Sec. 2 are not in parentheses. We will fix the citation format.
  • The reference entry for Nüwa contains a broken character. We will fix it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.