acceptodds
Under review as a conference paper at ICLR 2027

Where to Look, What to Keep: Two-Phase Learning for Grounding-Aware Visual Token Selection

Abstract

Multimodal Large Language Models (MLLMs) encode images into hundreds or thousands of visual tokens, incurring substantial inference overhead. Although visual token pruning mitigates this cost, aggressive pruning can impair visual grounding, which requires both recognizing the referred target and predicting its spatial coordinates. We diagnose this degradation with budget-matched token-addition experiments and find that existing pruners either lose focus on the target or discard the spatial context needed for grounding. These findings motivate two complementary learning objectives: learning query-conditioned target relevance (where to look) and optimizing the retained subset for grounding accuracy (what to keep). We implement these objectives through a two-phase framework that trains a lightweight, query-conditioned token scorer on frozen MLLM backbones. In Phase One, Localization Prior Learning (LPL) uses bounding-box supervision to teach the scorer where to look. In Phase Two, Grounding-Guided Selection Refinement (GSR) optimizes token selections through direct grounding feedback, teaching the scorer what to keep by rewarding subsets that yield accurate bounding-box predictions. At an visual-token retention ratio, our method retains of unpruned grounding accuracy on Qwen2.5-VL-7B and exceeds the unpruned LLaVA-1.5-7B (), while retaining and of general VQA performance across ten benchmarks, respectively. Furthermore, our approach achieves up to end-to-end speedup and prefill speedup, offering a practical solution for efficient MLLM inference. Code and model checkpoints will be publicly released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
VLM, MLLM, Visual Token Pruning, Visual Grounding, Efficient Inference, Reinforcement Learning for Token Selection

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_m4FmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates visual token pruning for multimodal large language models (MLLMs). The authors argue that token selection for visual grounding involves two distinct objectives:

  • “Where to look”: learning query-conditioned target relevance to identify the target region relevant to the query.
  • “What to keep”: retaining, under a fixed token budget, the target evidence and spatial context required for accurate coordinate prediction. To address these objectives, the authors propose a two-phase framework:
  1. Localization Prior Learning (LPL): uses bounding-box supervision to train a query-conditioned token scorer that learns target relevance.
  2. Grounding-Guided Selection Refinement (GSR): keeps the MLLM backbone frozen and further optimizes the token selector using a policy-gradient objective, where the IoU between the bounding-box predictions produced from candidate token subsets and the ground-truth bounding box serves as the reward signal.
Critical strengths
  1. The motivation is clear, and the proposed methodology is effective. The method is simple, practical, and empirically effective. The MLLM backbone remains frozen throughout training, and only a lightweight scorer with approximately 2.1–2.4M parameters is trained, without modifying the backbone itself. During inference, no reinforcement-learning sampling is required; only a single deterministic Top-K selection is performed.

  2. The ablation studies are relatively comprehensive. They examine the effects of coordinate features, attention cues, and the probe-based scoring mechanism, as well as different pruning layers, token budgets, and the practical utility of tokens promoted or discarded by GSR.

  3. The method is evaluated across multiple datasets and task types. The results indicate that it preserves general-purpose capabilities reasonably well.

  4. The paper is well organized and clearly written, with a coherent logical flow.

Critical weaknesses
  1. The methodological novelty is limited. LPL is closely related to existing bounding-box-supervised token selectors, while GSR essentially performs discrete Top-K policy optimization using localization IoU as the reward. The paper’s contribution lies primarily in combining these components, diagnosing the task, and demonstrating strong empirical performance, rather than introducing a fundamentally new learning mechanism. The authors should more clearly distinguish the proposed method from learning-based or reinforcement-learning-based pruning methods such as GlimpsePrune and GRIP-VLM.

  2. The experimental section should clarify whether the comparisons with related methods are fair, particularly regarding training data, supervision, token budgets, pruning configurations, and other hyperparameters. Moreover, although the paper emphasizes computational savings, it does not provide sufficiently detailed information about the associated computational costs and efficiency measurements.

Questions for authors
  1. GSR is trained on RefCOCO, RefCOCO+, and RefCOCOg, while the main grounding results are evaluated on these three highly related COCO-based referring-expression datasets, although the paper states that overlapping image IDs have been removed. To better demonstrate generalization and support a fairer comparison, the authors should consider adding results on an independent grounding dataset, such as Flickr30k Entities.
  2. Were the competing methods in Tables 3, 4, and 5 evaluated using the same training data, supervision, token budgets, pruning-layer configurations, and other relevant hyperparameters?
  3. Generating 24 candidate subsets for each instance over 4,000 optimization steps appears to require a substantial number of frozen-MLLM rollouts for each backbone. Although the paper emphasizes inference-time savings, it does not report the GPU hours, peak memory usage, energy consumption, or the approximate number of inference requests required to amortize the training cost. Could the authors provide these details?
3 comments
AuthorsOpenReview ↗

Response to Reviewer m4Fm (1/3)

Thank you very much for the positive assessment and for the concrete questions. We appreciate your recognition that the method is simple, practical and effective, with a frozen backbone and a lightweight scorer, and that the ablation studies are comprehensive and the paper is clearly written. We address each of your concerns below.

(1/3) Novelty

W1. LPL is close to existing box-supervised selectors, and GSR is discrete top-K policy optimization with an IoU reward; the contribution lies mainly in the combination, the diagnosis and the results. Please distinguish the method more clearly from GlimpsePrune, GRIP-VLM and RL-based selectors.

The earlier learned selectors were designed and evaluated for general VQA. Our starting point is a failure their evaluations do not cover. For example, at 11.1%, VisionZip keeps 79.3% of general-VQA performance but only 26.2% of grounding accuracy. We trace this failure to two separable causes, missing target evidence and missing spatial context, and each of our components addresses one of them.

GlimpsePruneGRIP-VLMTPRLOurs
Problem addressedattention-derived token importance is fragile, and fixed compression ratios cannot adapt to scenes of varying complexitycontinuous relaxations of the discrete token selection lead to sub-optimal solutionshand-engineered scoring rules; progressive pruning is not modeled as a multi-step decisiongrounding collapses under aggressive pruning while general VQA is largely preserved, because target evidence and spatial context are lost
Target taskgeneral VQAgeneral VQAgeneral VQAgrounding + general VQA
Learning signallanguage-modeling loss + box localization loss (Dice + BCE)distillation, then RL on task losslearning from demonstrations, then RL (PPO) on task + efficiency rewardbox prior (LPL), then the frozen model's IoU on the retained subset (GSR)

The methods share some building blocks (a learned scorer, a policy-gradient stage), but they start from different problems. The three earlier selectors ask how to learn a better or more adaptive pruning policy for general VQA, and none of their training signals evaluates whether the retained tokens still support localizing the target. Our question is why grounding fails under pruning and how to prevent it. This leads to the design of GSR since box supervision tells the scorer where the target is, but a token outside the target box can only receive credit through the outcome of the whole retained subset, so we reward the IoU of the box that the frozen model predicts from the retained tokens.

The difference also shows empirically (full tables in General Response G1). With the same training data and budget, the same TPRL selector reaches 20.94 on RefCOCO+ with its own recipe and 45.57 with ours (ours: 57.05, LLaVA-1.5-7B). GRIP-VLM's first-stage objective (distillation) reaches 23.8 on RefCOCO+ with our scorer, against 63.3 for LPL (Qwen2.5-VL-7B).

Our core contributions are therefore:

  1. A diagnosis of why grounding fails under pruning. Budget-matched token-addition experiments (Tables 1–2) separate two causes: adding target tokens helps query-agnostic selectors more than enlarging their own budget, and adding context tokens further helps target-guided selectors.
  2. A two-phase objective that addresses one cause each. LPL learns where to look from box supervision; GSR learns what to keep from the frozen model's grounding IoU on the retained subset. Only a lightweight scorer is trained and the backbone stays frozen.
  3. Grounding and general VQA are preserved together. At 11.1% final retention, mean grounding accuracy is 82.4 / 60.8 (Qwen2.5-VL / LLaVA-1.5) with 98.6% / 98.2% general-VQA retention.
AuthorsOpenReview ↗

Response to Reviewer m4Fm (2/3)

(2/3) Fairness of the comparison

W2 / Q2. Are the comparisons fair with respect to training data, supervision, token budgets, pruning configurations and other hyperparameters? The efficiency measurements also need more detail.

All methods share the backbone checkpoint, image preprocessing, prompts, output post-processing, the lmms-eval tasks, and token budget. The budget is the final visual-token retention ratio (tokens remaining after the last pruning operation), following the convention of GlimpsePrune and IVC-Prune. The table below lists each method's pruning position (also in Appendix Table 16 of the revised paper) together with its grounding and general-VQA results:

MethodTypePruning position (Qwen / LLaVA)Grounding [email protected] (Qwen / LLaVA)General-VQA retention (Qwen / LLaVA)
Unpruned––85.3 / 51.7100% / 100%
VisionZiptraining-free, general-purposebefore the LLM22.4 / 9.079.3% / 90.5%
DivPrunetraining-free, general-purposebefore the LLM27.9 / 17.576.8% / 89.9%
Nüwatraining-free, grounding-orientedpre-LLM merge + decoder layer 14 / 1665.2 / 55.091.9% / 95.4%
IVC-Prunetraining-free, grounding-orienteddecoder, after 16 / 16 blocks73.7 / 41.190.0% / 88.8%
GlimpsePrunelearned (Visual-CoT; released checkpoint)decoder, after 19 / 22 blocks70.6 / 49.998.2% / 95.7%
Ourslearned (8,000 Visual-CoT + 4,000 RefCOCO rows)decoder, after 17 / 13 blocks82.4 / 60.898.6% / 98.2%

Grounding is the mean over the eight RefCOCO/+/g splits and general-VQA retention is the mean over ten benchmarks (Tables 3 and 4 of the paper). Nüwa and IVC-Prune are the two baselines designed specifically to preserve grounding under pruning; ours is 8.7 / 5.8 points above the better of them on each backbone and has the highest general-VQA retention on both.

Because the methods differ in pruning position and in training, we added two controls.

(i) Same pruning position. VisionZip and DivPrune are adapted to prune at our layer with the same final budget (mean RefCOCO+ [email protected]; Appendix Table 10 of the revised paper):

Same pruning layer, 11.1% finalVisionZipDivPruneOurs
Qwen2.5-VL (layer 16)52.8660.0578.05
LLaVA-1.5 (layer 12)42.4136.6357.05

(ii) Same training data. The TPRL comparison and the first-stage swap in General Response G1 use our training rows and final budget.

We also measured the efficiency of VisionZip and DivPrune in their original pre-LLM configuration with the paper's timing protocol. They are faster (1.70–2.20× vs. our 1.44–1.54× end-to-end) but keep only 26–33% (Qwen) and 17–34% (LLaVA) of the unpruned grounding accuracy against 96.6% for ours on Qwen2.5-VL.

AuthorsOpenReview ↗

Response to Reviewer m4Fm (3/3)

(3/3) Independent grounding data and training cost

Q1. Training and evaluation both use the COCO-based RefCOCO family; please add an independent grounding dataset such as Flickr30k Entities.

We added Flickr30k Entities as you proposed, and additionally the GUI benchmark ScreenSpot; neither shares images with our training data, and we use the same checkpoints, prompts and 11.1% final budget (full tables in General Response G2). On Flickr30k Entities (official test split, 14,481 phrases), ours retains 96.3% of the unpruned accuracy on Qwen2.5-VL (64.73 vs. 67.25) and exceeds the unpruned LLaVA-1.5 (50.22 vs. 39.20), consistent with the RefCOCO results. On ScreenSpot, ours is the best of the three pruned methods we evaluated, and its gap to the unpruned model narrows to within 0.5 points at a 33.3% budget with the same checkpoint. On Qwen2.5-VL, ours outperforms both grounding-oriented baselines (Nüwa and IVC-Prune) on both benchmarks.

Q3. GSR needs many frozen-model rollouts (24 subsets × 4,000 steps). What are the GPU hours, peak memory, energy, and the number of requests needed to amortize the training?

In our training, the backbone stays frozen and only a lightweight scorer (about 2M parameters) is trained, on a single RTX 4090.

BackboneLPLGSRTotalPeak memory (LPL / GSR)
Qwen2.5-VL-7B2.27 h3.44 h5.71 h15.9 / 18.4 GiB
LLaVA-1.5-7B1.54 h2.66 h4.20 h13.9 / 18.9 GiB

GSR uses 4,000 updates × 24 candidate subsets, i.e. 96,000 frozen-model rollouts per backbone. The 24 candidates of an instance run as one batch without gradients; the policy gradient flows only through the scorer's selection log-likelihood, so no backbone activations are stored.

Amortization. On the paper's efficiency workload (batch 32, 32 output tokens), ours saves 47.6 ms per request on Qwen and 62.5 ms on LLaVA, so the training time is recovered after about 430,000 and 240,000 requests.

Energy. Using the 450 W board power limit as an upper bound, GPU energy for training is at most 2.6 kWh (Qwen) and 1.9 kWh (LLaVA).

We have added the training time and peak memory to Appendix C.2 (Table 15) of the revised paper.

Thank you again for the encouraging and helpful review. We would be glad to answer any further question.

Reviewer_bMw2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies visual token pruning for visual grounding, where existing pruning methods often lose either the referred target or the spatial context needed to predict accurate box coordinates. Using budget-matched token-addition experiments, the authors show two things: query-agnostic selectors benefit from adding target tokens, and target-guided selectors benefit from adding spatially distributed context tokens. Based on this, they propose a lightweight query-conditioned token scorer inserted after an intermediate decoder layer of a frozen MLLM. It is trained in two phases: Localization Prior Learning (LPL): uses bounding-box supervision to teach the scorer where to look. Grounding-Guided Selection Refinement (GSR): samples candidate token subsets with Gumbel top-K and uses the frozen MLLM's grounding IoU as a reward in a group-relative policy gradient. At inference, the scorer keeps the top-K tokens for the remaining layers. Experiments on Qwen2.5-VL-7B and LLaVA-1.5-7B show strong grounding results while preserving general VQA performance.

Critical strengths
  • The paper addresses a real and under-explored problem. Visual grounding degrades severely under aggressive pruning even when standard VQA benchmarks look fine (e.g., VisionZip and DivPrune keep only 17–34% of unpruned grounding accuracy in Table 3).
  • The token-addition pilot study is well designed and informative, clearly separating the "where to look" and "what to keep" failure modes.
  • Using the frozen MLLM's grounding IoU as a reward is a clean way to optimize the selected token subset directly for the downstream task, rather than relying on proxy scores.
  • The method achieves strong grounding results while preserving general VQA performance. The ablations clearly show that the two training phases complement each other, and the analysis in Appendix A.3 supports that GSR learns to keep more useful tokens.
Critical weaknesses
  • The comparison with training-free methods is a bit unfair. Most of the compared baselines (VisionZip, DivPrune, Nüwa, IVC-Prune) are training-free so the advantage over those baselines doesn’t seem impressive.
  • The method appears to learn more than where to look. For example, GSR is trained on RefCOCO data, and the evaluation is on the same three benchmarks. The images don't overlap, but the data distribution, expression style and box conventions are the same. On LLaVA-1.5-7B, the method reaches 117.6% of the unpruned model's grounding accuracy, this kind of gain suggests the scorer learns a task- and distribution-specific selection policy that compensates for the backbone's weak grounding ability, going beyond pruning redundant tokens. To support the claims, the method should be evaluated on out-of-domain grounding benchmarks not used in training (e.g., GUI grounding benchmarks like ScreenSpot).
  • The 11.1% retention ratio is misleading. The ratio refers only to the final number of retained tokens. Pruning happens after decoder layer 16 on Qwen2.5-VL and layer 12 on LLaVA-1.5, so by the time of pruning, a large portion of the LLM layers have already processed all visual tokens. Averaged over layers, the effective visual-token retention is much higher. This is reflected in the modest efficiency gains on Qwen2.5-VL: only a 1.23× prefill speedup and about 30% KV cache reduction. So beating baselines like VisionZip and DivPrune, which prune before the LLM and keep 11.1% of tokens at every layer, is not surprising, and these baselines are not included in the efficiency comparison.
Questions for authors
  1. Can you evaluate on out-of-domain grounding benchmarks that are not part of the GSR training data?
  2. How would the method be used in a general deployment where the task type is unknown? Can one scorer serve both grounding and VQA well?
5 comments
AuthorsOpenReview ↗

Response to Reviewer bMw2 (1/2)

Thank you very much for the detailed and constructive review. We greatly appreciate your recognition that the paper addresses a real and under-explored problem, that the token-addition pilot study clearly separates the "where to look" and "what to keep" failure modes, and that using the frozen MLLM's grounding IoU as the reward is a clean way to optimize the selected subset. Below we provide several additional experiments and clarifications to address your concerns.

(1/2) Learned baselines and out-of-domain grounding

W1. Most compared baselines (VisionZip, DivPrune, Nüwa, IVC-Prune) are training-free, so the advantage over them is not very impressive.

A comparison with learned selectors is indeed the more informative one. In the submission, we compared with a general training-based baseline GlimpsePrune. At the same 11.1% final budget, ours is 11.7 / 10.9 points higher in mean grounding [email protected] on Qwen2.5-VL / LLaVA-1.5 (82.4 vs. 70.6; 60.8 vs. 49.9) and retains more general VQA (98.6% vs. 98.2%; 98.2% vs. 95.7%).

We have now also compared with the learned selectors TPRL and GRIP-VLM; the full tables and settings are in General Response G1. In brief, on LLaVA-1.5-7B with the same selector architecture, training data and 64-token budget, TPRL's selector reaches 20.94 on RefCOCO+ with its own recipe and 45.57 with our recipe, against 57.05 for ours; with its native dynamic rule TPRL keeps 395 of 576 tokens on average (68.6%) and reaches 52.40. GRIP-VLM has released neither code nor a checkpoint, so we compared the part that transfers to our scorer directly (the first-stage distillation). On Qwen2.5-VL-7B, its distillation warm-up reaches 23.8 on RefCOCO+ and 73.6 on OCRBench, against 63.3 and 79.7 for our LPL.

Our advantage is therefore not limited to training-free baselines; it also holds against learned selectors under the same budget.

W2 / Q1. GSR is trained on RefCOCO and evaluated on the same benchmark family (same expression style and box conventions). Reaching 117.6% of the unpruned LLaVA-1.5 suggests a task- and distribution-specific policy rather than redundancy pruning; please evaluate on out-of-domain grounding benchmarks such as ScreenSpot.

Our scorer both removes redundant tokens and learns a query-conditioned spatial prior. For a backbone whose own grounding is weak, discarding distracting background tokens can therefore make grounding better than with all tokens. Exceeding the unpruned model does not by itself indicate RefCOCO-specific fitting: the training-free Nüwa, never trained on RefCOCO, also exceeds the unpruned LLaVA-1.5 (106.4%). To verify that the learned prior transfers beyond RefCOCO, we tested two out-of-domain benchmarks with the same checkpoints and 11.1% final budget (full tables in General Response G2): Flickr30k Entities (14,481 phrases), whose phrases come from image captions rather than referring expressions, and the GUI benchmark ScreenSpot, whose screenshots differ from natural images.

11.1% finalFlickr30k [email protected]ScreenSpot v1 / v2
Qwen2.5-VL, unpruned67.2578.7 / 80.7
Qwen2.5-VL, best baseline (IVC-Prune)56.3857.8 / 57.8
Qwen2.5-VL, ours64.7367.6 / 69.1
LLaVA-1.5, unpruned39.209.4 / 9.6
LLaVA-1.5, ours50.229.9 / 9.5

On Flickr30k Entities, ours keeps 96.3% of the unpruned accuracy on Qwen2.5-VL and exceeds the unpruned LLaVA-1.5 by 11.0 points, so the effect you noted also appears on data the scorer never saw. On ScreenSpot, ours is the best of the three pruned methods at every budget we tested. There is a real gap to the unpruned Qwen2.5-VL at 11.1%, since GUI targets are small; it narrows with budget to within 0.5 points at 33.3% (78.6 / 80.3) with the same checkpoint, as reported in Appendix A.4. On LLaVA-1.5, ours keeps ScreenSpot accuracy at the unpruned level (9.9 / 9.5 vs. 9.4 / 9.6).

Finally, the same LLaVA-1.5 checkpoint retains 98.2% of general-VQA performance, so the selection does not trade other tasks for grounding.

AuthorsOpenReview ↗

Response to Reviewer bMw2 (2/2)

(2/2) Budget, pruning position, efficiency, and one scorer for all tasks

W3. 11.1% counts only the tokens left after pruning at layer 16 / 12, so the retention averaged over layers is much higher, and the Qwen speedup is modest (1.23× prefill). Beating VisionZip and DivPrune, which prune before the LLM, is therefore not surprising, and they are missing from the efficiency comparison.

We reported the final visual-token retention ratio (tokens remaining after the last pruning operation divided by the original count) because it is the budget convention of the closest decoder-pruning work (GlimpsePrune and IVC-Prune) and it is well defined across pruning positions. In the revised paper we now write final retention ratio throughout and list every method's pruning position (Appendix Table 16).

We agree that the retention averaged over decoder layers is higher than 11.1%. Because the layers before the pruning layer process the full sequence, ours processes on average about 65% (Qwen) and 47% (LLaVA) of the visual tokens, so its compute saving is smaller than that of a pre-LLM method at the same final ratio. This holds for every decoder-pruning method: the number of full-sequence blocks is 16 / 16 for IVC-Prune, 19 / 22 for GlimpsePrune and 17 / 13 for ours. The saving also extends beyond the prefill: after the pruning layer, the KV cache holds only the retained visual tokens, so every generated token attends to a shorter cache. This is why the end-to-end speedup on Qwen (1.44×) is larger than the prefill speedup (1.23×).

We also ran two comparisons to address your concerns:

  1. Same pruning layer. We adapted VisionZip and DivPrune to prune at our layer (layer 16 on Qwen and 12 on LLaVA) with the same final budget, so all three methods process the same number of visual tokens at every layer (Appendix Table 10 of the revised paper).
Same pruning layer, 11.1% finalRefCOCO+ meanOCRBenchChartQAMMStarMMBenchVQAv2-lite
Qwen2.5-VL, VisionZip @1652.8660.278.161.282.879.3
Qwen2.5-VL, DivPrune @1660.0555.864.961.383.174.7
Qwen2.5-VL, Ours78.0579.280.061.283.079.6
LLaVA-1.5, VisionZip @1242.4123.317.933.363.971.4
LLaVA-1.5, DivPrune @1236.6326.316.233.563.965.9
LLaVA-1.5, Ours57.0528.317.333.564.069.4

Pruning position alone therefore does not explain the grounding gap: ours is 18.0–25.2 points (Qwen) and 14.6–20.4 points (LLaVA) higher on RefCOCO+, while general VQA is close on MMStar and MMBench.

  1. Efficiency including VisionZip and DivPrune. We extended the paper's efficiency table with both methods in their original pre-LLM configurations, using the paper's timing protocol (Appendix C.5):
11.1% finalQwen E2E speedupQwen groundingLLaVA E2E speedupLLaVA grounding
Unpruned1.00×85.31.00×51.7
VisionZip (pre-LLM)1.70×22.42.16×9.0
DivPrune (pre-LLM)1.71×27.92.20×17.5
GlimpsePrune1.37×70.61.35×49.9
IVC-Prune1.46×73.71.45×41.1
Ours1.44×82.41.54×60.8

VisionZip and DivPrune are faster, as expected, since they prune before the LLM and every decoder layer processes only the retained tokens. At this budget, however, they keep only 26–33% (Qwen) and 17–34% (LLaVA) of the unpruned grounding accuracy, while ours keeps 96.6% (Qwen) and 117.6% (LLaVA).

We have added these rows to Table 5 of the revised paper and discuss this trade-off in Sec. 5; the pruning-depth ablation (Table 7) shows the same trade-off within our method.

Q2. How would the method be used when the task type is unknown? Can one scorer serve both grounding and VQA?

Yes. Each backbone uses a single checkpoint for all grounding benchmarks, the ten general-VQA benchmarks, and the new out-of-domain evaluations; the scorer is conditioned on the query and needs no task-related label.

Thank you again for the specific and helpful comments.

Dear Reviewer bMw2,

Thank you again for your insightful feedback. We have carefully addressed the concerns raised and provided detailed responses in this rebuttal.

As the discussion period is approaching its end, we would greatly appreciate your feedback on whether our responses have adequately addressed your concerns. We would be happy to provide further clarification if needed.

Reviewer_csjoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a two-stage method for selecting visual tokens and keeping the underlying vision-language model frozen. It first learns which image regions matter for a query, then refines token selection using feedback on predicted bounding boxes. The experiments show improved grounding over the tested pruning methods and largely preserves visual question-answering performance. However, the gain over closely related learned selectors and the specific benefit of the two-stage schedule need stronger evidence.

Critical strengths
  • The token-addition experiments in Tables 1 and 2 compare selections with the same total token budget. They show that keeping the referred object alone can be insufficient and that additional image context helps localization, providing a useful motivation for the method.

  • The evaluation checks grounding, general visual question answering, and measured inference speed on different backbones. This helps establish that the method saves computation while retaining useful capabilities, rather than merely reducing the number of tokens.

Critical weaknesses
  • The main training recipe is close to prior work. GlimpsePrune already uses bounding-box supervision, and GRIP-VLM combines supervised warm-up with reinforcement-learning token selection on a frozen backbone. The paper cites these methods and compares with GlimpsePrune, but does not evaluate GRIP-VLM or TPRL. Its potential gain is the grounding-specific supervision and reward, and comparisons under matched training data and token budgets would better establish what these choices add.

  • The comparison in Table 6 changes two things at once. The training method and the training data. The full method sees document and text-reading examples that the second-stage-only model does not. Its better text-reading performance may therefore come from those extra examples rather than the two-stage training strategy.

Questions for authors

Please see the weaknesses above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer csjo

Thank you for the careful review and the two concrete suggestions. We appreciate your recognition that the token-addition experiments provide a useful motivation for the method, and that the evaluation covers grounding, general VQA and measured inference speed on different backbones. We conducted additional experiments on the baseline comparison and the ablation; all new experiments below use the same evaluation protocol as the paper.

(1/2) Comparison with learned selectors

W1. GlimpsePrune already uses box supervision, and GRIP-VLM combines a supervised warm-up with RL-based token selection on a frozen backbone. The paper does not evaluate GRIP-VLM or TPRL; comparisons under matched training data and token budgets would show what the grounding-specific supervision and reward add.

Since all three reviewers asked about these methods, the full tables and settings are in General Response G1; the main results are:

  • TPRL (LLaVA-1.5-7B, 64 of 576 tokens kept, same training data). TPRL's selector reaches 20.94 on RefCOCO+ with its own recipe and 45.57 when trained with our recipe (LPL→GSR); ours reaches 57.05. So our training recipe alone more than doubles the accuracy of the same selector. With its native dynamic pruning rule, TPRL keeps 395 of 576 tokens on average (68.6%) and still reaches only 52.40.
  • GRIP-VLM (Qwen2.5-VL-7B, our scorer, same data). GRIP-VLM has released neither code nor a checkpoint, so we compared the first stage, whose objective (distillation from the unpruned model) transfers to our scorer directly. The distillation warm-up reaches 23.82 on RefCOCO+ and 73.6 on OCRBench; our box-supervised first stage (LPL) reaches 63.28 and 79.7.

Replacing GRIP-VLM's distillation with our first stage (LPL) gives a much better grounding warm-up (63.28 vs. 23.82), and training the same TPRL selector with our two-phase recipe (box prior + IoU reward) instead of TPRL's own recipe more than doubles its grounding accuracy (45.57 vs. 20.94).

(2/2) Training phase vs. training data

W2. Table 6 changes two things at once, the training method and the training data. The full method sees document and text-reading examples that the GSR-only model does not, so its better text reading may come from those examples rather than from the two-phase strategy.

In Table 6 we trained each phase on its own data (LPL on Visual-CoT, GSR on RefCOCO), so the GSR-only model never saw the document and text-reading samples; the comparison therefore mixed the two factors (data and training phase). We have now trained GSR alone on the data of both phases (8,000 Visual-CoT samples and 4,000 RefCOCO samples).

Qwen2.5-VL-7B, 11.1% finalRefCOCO+ valtestAtestBmeanOCRBenchChartQAMMStarMMBenchVQAv2-lite
GSR only, RefCOCO data (as in Table 6)78.5684.5471.2078.1050.979.061.783.075.9
GSR only*, RefCOCO + Visual-CoT data (new)78.7884.5370.9878.1056.179.461.683.176.9
LPL only65.0269.3355.4963.2879.780.161.682.979.3
LPL → GSR (our full model)78.5785.2470.3478.0579.280.061.283.079.6

Adding the LPL data to GSR training improves text reading and general VQA (OCRBench 50.9 → 56.1, VQAv2-lite 75.9 → 76.9), while grounding stays the same (78.10). Data exposure therefore explains only a small part of the text-reading gap: on the same data, the two-phase model reaches 79.2 on OCRBench, so most of the gain comes from the LPL phase itself (the localization supervision). The grounding reward, in turn, is what raises RefCOCO+ from 63.28 (LPL only) to 78.05. We have added this result to Table 6 of the revised paper (row "GSR only*").

Thank you again for the suggestions; they made the paper's evidence more complete.

Dear Reviewer csjo,

Thank you again for your constructive suggestions. Following your feedback, we conducted additional controlled experiments to better establish the effectiveness of our two-phase training strategy and its advantages over existing learned selectors.

We hope these new results address your main concerns. If any part is still unclear, we’re glad to elaborate. We would be grateful if you could re-evaluate the paper considering these revisions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers (2/2)

G2. Grounding outside the RefCOCO family

We evaluated two benchmarks outside the RefCOCO family, Flickr30k Entities (official test split, 14,481 phrases) and the GUI benchmark ScreenSpot, with the same checkpoints, prompts and final budget as the main tables.

11.1% finalFlickr30k [email protected]ScreenSpot v1 / v2
Qwen2.5-VL, unpruned67.2578.7 / 80.7
Qwen2.5-VL, Nüwa46.6645.0 / 45.3
Qwen2.5-VL, IVC-Prune56.3857.8 / 57.8
Qwen2.5-VL, ours64.7367.6 / 69.1
LLaVA-1.5, unpruned39.209.4 / 9.6
LLaVA-1.5, ours50.229.9 / 9.5

On Flickr30k Entities, ours keeps 96.3% of the unpruned accuracy on Qwen2.5-VL and exceeds the unpruned LLaVA-1.5 by 11.0 points, consistent with the RefCOCO results.

On ScreenSpot, there is a real gap to the unpruned Qwen2.5-VL at 11.1% (67.6 / 69.1 vs. 78.7 / 80.7). GUI targets are usually small icons and text spans, so an aggressive 11.1% budget loses visual details needed to locate them. We therefore also report larger final retention ratios with the same checkpoint (Qwen2.5-VL; v1 / v2; unpruned 78.7 / 80.7):

Final retention ratioNüwaIVC-PruneOurs
11.1%45.0 / 45.357.8 / 57.867.6 / 69.1
22.2%64.5 / 66.569.3 / 70.976.1 / 78.4
33.3%71.9 / 73.374.6 / 76.378.6 / 80.3

Ours is the best of the three pruned methods we evaluated, on both benchmarks and at every budget, and its gap to the unpruned model narrows to within 0.5 points at 33.3% with the same checkpoint. On LLaVA-1.5, ours keeps ScreenSpot accuracy at the unpruned level (9.9 / 9.5 vs. 9.4 / 9.6).

We have added both benchmarks to Appendix A.4 of the revised paper.

AuthorsOpenReview ↗

General Response to All Reviewers (1/2)

We sincerely thank all reviewers for their time and constructive feedback. We are encouraged that the reviewers found the pilot study to cleanly separate the two failure modes of pruning for grounding, missing target evidence and missing spatial context (csjo, bMw2), that rewarding the frozen model's IoU on the retained subset optimizes directly what grounding needs (bMw2), and that our method is lightweight and practical (m4Fm).

Two requests were shared by several reviewers, so we answer them here once:

  • (G1) Does the advantage hold against learned selectors? Yes: in matched comparisons with both TPRL and GRIP-VLM (same training data and token budget), ours is clearly ahead on grounding.
  • (G2) Does the scorer overfit RefCOCO? No: on Flickr30k Entities and ScreenSpot, never used in training, ours is the best pruned method. G1. Comparison with the learned selectors TPRL and GRIP-VLM

GRIP-VLM (arXiv, May 2026) and TPRL (arXiv, March 2026) are preprints without released selector checkpoints, so the submission did not compare with them. We agree that the comparison is informative and conducted these additional experiments here. The training data, the pruning layer, the token budget and the evaluation harness are the same as ours unless explicitly noted.

TPRL (code released). TPRL releases its selector architecture and training code for LLaVA-1.5-7B. We trained TPRL's selector with our two-phase recipe (LPL→GSR) and also trained the same selector with TPRL's own pipeline (learning from demonstrations, then PPO) on the same training data.

LLaVA-1.5-7BVisual tokens keptRefCOCO+ valtestAtestBmeanOCRBenchChartQAMMStarMMBenchVQAv2-lite
TPRL selector + TPRL recipe (dynamic budget)395 avg (68.6%)53.4362.4241.3452.4031.217.533.764.171.3
TPRL selector + TPRL recipe64 (11.1%)20.5222.0020.3120.9419.813.633.663.660.1
TPRL selector + our recipe (LPL→GSR)64 (11.1%)46.0956.2234.4045.5726.215.934.063.864.8
Ours64 (11.1%)57.1268.9145.1057.0528.317.333.564.069.4

With the same training data, recipe and token budget, ours is 11.5 points higher on RefCOCO+ than the TPRL selector (57.05 vs. 45.57), and higher on OCRBench, ChartQA and VQAv2-lite. With its own recipe and its native dynamic pruning rule (keep the tokens with p > 0.5), TPRL keeps 395 of 576 visual tokens on average (68.6%) and reaches 52.40 on RefCOCO+, where ours reaches 57.05 with only 64 tokens.

At the same 11.1% budget, TPRL with its own recipe drops substantially on visual grounding (20.94 on RefCOCO+) and on the text-heavy and VQAv2 tasks (e.g., OCRBench 19.8 vs. 28.3, VQAv2-lite 60.1 vs. 69.4). Training the same selector with our recipe (LPL→GSR) instead reaches 45.57 at the same budget, which indicates the effectiveness of our two-phase training pipeline.

GRIP-VLM (no code or checkpoint released). As no code or checkpoint is released for GRIP-VLM, its selector cannot be reproduced in a verifiable way. Of its two training stages, the distillation objective of the warm-up (the pruned model is trained to match the unpruned one) does not depend on the selector's design and can be applied to our scorer directly. In contrast, GRIP-VLM's RL stage is specified for its own scorer design (independent per-token keep/drop actions, progressive pruning at several decoder layers, a budget-conditioned scorer, and a reward that penalizes exceeding the token budget). We therefore compare the first stage, which can be transferred faithfully, using our scorer with the same pruning layer, data, training steps and token budget on Qwen2.5-VL-7B:

Qwen2.5-VL-7B, our scorer, 11.1%, first stage onlyRefCOCO+ valtestAtestBmeanOCRBenchChartQAMMStarMMBenchVQAv2-lite
Distillation warm-up (GRIP-VLM's first-stage objective)23.9626.5120.9923.8273.678.061.782.979.4
LPL (our first stage)65.0269.3355.4963.2879.780.161.682.979.3

With the same scorer and training data, LPL reaches 63.28 on RefCOCO+ where the distillation warm-up reaches 23.82, and it is also higher on OCRBench (79.7 vs. 73.6) and ChartQA (80.1 vs. 78.0). Distillation teaches the selector to preserve the unpruned model's outputs but gives no signal about where the referred object is.

We have added the conceptual differences (learning signal, action space, target task) to the related-work section of the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.