- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates visual token pruning for multimodal large language models (MLLMs). The authors argue that token selection for visual grounding involves two distinct objectives:
- “Where to look”: learning query-conditioned target relevance to identify the target region relevant to the query.
- “What to keep”: retaining, under a fixed token budget, the target evidence and spatial context required for accurate coordinate prediction. To address these objectives, the authors propose a two-phase framework:
- Localization Prior Learning (LPL): uses bounding-box supervision to train a query-conditioned token scorer that learns target relevance.
- Grounding-Guided Selection Refinement (GSR): keeps the MLLM backbone frozen and further optimizes the token selector using a policy-gradient objective, where the IoU between the bounding-box predictions produced from candidate token subsets and the ground-truth bounding box serves as the reward signal.
Critical strengths
-
The motivation is clear, and the proposed methodology is effective. The method is simple, practical, and empirically effective. The MLLM backbone remains frozen throughout training, and only a lightweight scorer with approximately 2.1–2.4M parameters is trained, without modifying the backbone itself. During inference, no reinforcement-learning sampling is required; only a single deterministic Top-K selection is performed.
-
The ablation studies are relatively comprehensive. They examine the effects of coordinate features, attention cues, and the probe-based scoring mechanism, as well as different pruning layers, token budgets, and the practical utility of tokens promoted or discarded by GSR.
-
The method is evaluated across multiple datasets and task types. The results indicate that it preserves general-purpose capabilities reasonably well.
-
The paper is well organized and clearly written, with a coherent logical flow.
Critical weaknesses
-
The methodological novelty is limited. LPL is closely related to existing bounding-box-supervised token selectors, while GSR essentially performs discrete Top-K policy optimization using localization IoU as the reward. The paper’s contribution lies primarily in combining these components, diagnosing the task, and demonstrating strong empirical performance, rather than introducing a fundamentally new learning mechanism. The authors should more clearly distinguish the proposed method from learning-based or reinforcement-learning-based pruning methods such as GlimpsePrune and GRIP-VLM.
-
The experimental section should clarify whether the comparisons with related methods are fair, particularly regarding training data, supervision, token budgets, pruning configurations, and other hyperparameters. Moreover, although the paper emphasizes computational savings, it does not provide sufficiently detailed information about the associated computational costs and efficiency measurements.
Questions for authors
- GSR is trained on RefCOCO, RefCOCO+, and RefCOCOg, while the main grounding results are evaluated on these three highly related COCO-based referring-expression datasets, although the paper states that overlapping image IDs have been removed. To better demonstrate generalization and support a fairer comparison, the authors should consider adding results on an independent grounding dataset, such as Flickr30k Entities.
- Were the competing methods in Tables 3, 4, and 5 evaluated using the same training data, supervision, token budgets, pruning-layer configurations, and other relevant hyperparameters?
- Generating 24 candidate subsets for each instance over 4,000 optimization steps appears to require a substantial number of frozen-MLLM rollouts for each backbone. Although the paper emphasizes inference-time savings, it does not report the GPU hours, peak memory usage, energy consumption, or the approximate number of inference requests required to amortize the training cost. Could the authors provide these details?
3 comments
Response to Reviewer m4Fm (2/3)
(2/3) Fairness of the comparison
W2 / Q2. Are the comparisons fair with respect to training data, supervision, token budgets, pruning configurations and other hyperparameters? The efficiency measurements also need more detail.
All methods share the backbone checkpoint, image preprocessing, prompts, output post-processing, the lmms-eval tasks, and token budget. The budget is the final visual-token retention ratio (tokens remaining after the last pruning operation), following the convention of GlimpsePrune and IVC-Prune. The table below lists each method's pruning position (also in Appendix Table 16 of the revised paper) together with its grounding and general-VQA results:
| Method | Type | Pruning position (Qwen / LLaVA) | Grounding [email protected] (Qwen / LLaVA) | General-VQA retention (Qwen / LLaVA) |
|---|---|---|---|---|
| Unpruned | – | – | 85.3 / 51.7 | 100% / 100% |
| VisionZip | training-free, general-purpose | before the LLM | 22.4 / 9.0 | 79.3% / 90.5% |
| DivPrune | training-free, general-purpose | before the LLM | 27.9 / 17.5 | 76.8% / 89.9% |
| Nüwa | training-free, grounding-oriented | pre-LLM merge + decoder layer 14 / 16 | 65.2 / 55.0 | 91.9% / 95.4% |
| IVC-Prune | training-free, grounding-oriented | decoder, after 16 / 16 blocks | 73.7 / 41.1 | 90.0% / 88.8% |
| GlimpsePrune | learned (Visual-CoT; released checkpoint) | decoder, after 19 / 22 blocks | 70.6 / 49.9 | 98.2% / 95.7% |
| Ours | learned (8,000 Visual-CoT + 4,000 RefCOCO rows) | decoder, after 17 / 13 blocks | 82.4 / 60.8 | 98.6% / 98.2% |
Grounding is the mean over the eight RefCOCO/+/g splits and general-VQA retention is the mean over ten benchmarks (Tables 3 and 4 of the paper). Nüwa and IVC-Prune are the two baselines designed specifically to preserve grounding under pruning; ours is 8.7 / 5.8 points above the better of them on each backbone and has the highest general-VQA retention on both.
Because the methods differ in pruning position and in training, we added two controls.
(i) Same pruning position. VisionZip and DivPrune are adapted to prune at our layer with the same final budget (mean RefCOCO+ [email protected]; Appendix Table 10 of the revised paper):
| Same pruning layer, 11.1% final | VisionZip | DivPrune | Ours |
|---|---|---|---|
| Qwen2.5-VL (layer 16) | 52.86 | 60.05 | 78.05 |
| LLaVA-1.5 (layer 12) | 42.41 | 36.63 | 57.05 |
(ii) Same training data. The TPRL comparison and the first-stage swap in General Response G1 use our training rows and final budget.
We also measured the efficiency of VisionZip and DivPrune in their original pre-LLM configuration with the paper's timing protocol. They are faster (1.70–2.20× vs. our 1.44–1.54× end-to-end) but keep only 26–33% (Qwen) and 17–34% (LLaVA) of the unpruned grounding accuracy against 96.6% for ours on Qwen2.5-VL.
Response to Reviewer m4Fm (3/3)
(3/3) Independent grounding data and training cost
Q1. Training and evaluation both use the COCO-based RefCOCO family; please add an independent grounding dataset such as Flickr30k Entities.
We added Flickr30k Entities as you proposed, and additionally the GUI benchmark ScreenSpot; neither shares images with our training data, and we use the same checkpoints, prompts and 11.1% final budget (full tables in General Response G2). On Flickr30k Entities (official test split, 14,481 phrases), ours retains 96.3% of the unpruned accuracy on Qwen2.5-VL (64.73 vs. 67.25) and exceeds the unpruned LLaVA-1.5 (50.22 vs. 39.20), consistent with the RefCOCO results. On ScreenSpot, ours is the best of the three pruned methods we evaluated, and its gap to the unpruned model narrows to within 0.5 points at a 33.3% budget with the same checkpoint. On Qwen2.5-VL, ours outperforms both grounding-oriented baselines (Nüwa and IVC-Prune) on both benchmarks.
Q3. GSR needs many frozen-model rollouts (24 subsets × 4,000 steps). What are the GPU hours, peak memory, energy, and the number of requests needed to amortize the training?
In our training, the backbone stays frozen and only a lightweight scorer (about 2M parameters) is trained, on a single RTX 4090.
| Backbone | LPL | GSR | Total | Peak memory (LPL / GSR) |
|---|---|---|---|---|
| Qwen2.5-VL-7B | 2.27 h | 3.44 h | 5.71 h | 15.9 / 18.4 GiB |
| LLaVA-1.5-7B | 1.54 h | 2.66 h | 4.20 h | 13.9 / 18.9 GiB |
GSR uses 4,000 updates × 24 candidate subsets, i.e. 96,000 frozen-model rollouts per backbone. The 24 candidates of an instance run as one batch without gradients; the policy gradient flows only through the scorer's selection log-likelihood, so no backbone activations are stored.
Amortization. On the paper's efficiency workload (batch 32, 32 output tokens), ours saves 47.6 ms per request on Qwen and 62.5 ms on LLaVA, so the training time is recovered after about 430,000 and 240,000 requests.
Energy. Using the 450 W board power limit as an upper bound, GPU energy for training is at most 2.6 kWh (Qwen) and 1.9 kWh (LLaVA).
We have added the training time and peak memory to Appendix C.2 (Table 15) of the revised paper.
Thank you again for the encouraging and helpful review. We would be glad to answer any further question.
Response to Reviewer m4Fm (1/3)
Thank you very much for the positive assessment and for the concrete questions. We appreciate your recognition that the method is simple, practical and effective, with a frozen backbone and a lightweight scorer, and that the ablation studies are comprehensive and the paper is clearly written. We address each of your concerns below.
(1/3) Novelty
The earlier learned selectors were designed and evaluated for general VQA. Our starting point is a failure their evaluations do not cover. For example, at 11.1%, VisionZip keeps 79.3% of general-VQA performance but only 26.2% of grounding accuracy. We trace this failure to two separable causes, missing target evidence and missing spatial context, and each of our components addresses one of them.
The methods share some building blocks (a learned scorer, a policy-gradient stage), but they start from different problems. The three earlier selectors ask how to learn a better or more adaptive pruning policy for general VQA, and none of their training signals evaluates whether the retained tokens still support localizing the target. Our question is why grounding fails under pruning and how to prevent it. This leads to the design of GSR since box supervision tells the scorer where the target is, but a token outside the target box can only receive credit through the outcome of the whole retained subset, so we reward the IoU of the box that the frozen model predicts from the retained tokens.
The difference also shows empirically (full tables in General Response G1). With the same training data and budget, the same TPRL selector reaches 20.94 on RefCOCO+ with its own recipe and 45.57 with ours (ours: 57.05, LLaVA-1.5-7B). GRIP-VLM's first-stage objective (distillation) reaches 23.8 on RefCOCO+ with our scorer, against 63.3 for LPL (Qwen2.5-VL-7B).
Our core contributions are therefore: