acceptodds
Under review as a conference paper at ICLR 2027

Ansight: Answer-Sensitive, Spatially Diverse Visual Token Pruning

10 views

Abstract

Modern vision-language models process hundreds to thousands of visual tokens for every query, making inference expensive in both computation and memory. Token pruning methods aim to reduce this cost, but the proxies they commonly use for token importance are not explicitly designed to capture the visual representations that actually matter for the model's answer. We show that the sensitivity of a VLM's answer to each visual representation can provide effective supervision for token selection. We identify Gradient Activation as an effective label-free signal derived from the frozen model's own generated answer, and distill it into Ansight, a lightweight question-conditioned selector that operates before the language model. However, relevance alone can spend a small token budget on a single spatial region or on redundant evidence. We therefore combine Ansight with regional spatial coverage and global feature diversity, retaining visual evidence that is compact but complementary. Extensive experiments across several backbones, benchmarks, and retention budgets show that Ansight achieves the highest average relative performance in 13 of 15 backbone–budget settings, reducing the performance loss of the strongest prior method by 26%. On LLaVA-NeXT-7B, it retains 94.9% of full-model performance with only 5% of visual tokens, while cutting computation by 8.6, reducing KV-cache memory by 94%, and accelerating end-to-end inference by 2.7. We further show that Ansight transfers to long-video question answering without video-specific retraining and extends to multi-turn dialog over a fixed image.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models, Visual Token Pruning, Efficient Multimodal Inference, Gradient-Based Attribution, Label-Free Supervision, Diversity-Aware Token Selection, KV-Cache Compression

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gchgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Ansight, a learned visual-token selector for VLMs. Offline, the frozen VLM generates its own answer for an image–question pair, and a teacher-forced backward pass computes a Gradient × Activation (G×A) attribution of the answer log-probability for each visual token. A small MLP is distilled to predict these attributions before the language model, from the visual token, the mean question embedding, their interaction, and a Fourier position encoding. One selector is trained per backbone on a few hundred DeFacto image–question pairs, without ground-truth answer labels for the targets. At inference, two selection rules sit on top of the predicted relevance: Regional Allocation (RA) splits the budget across fixed 4×4 cells in proportion to predicted relevance mass, and Region-Constrained Global Diversity (RCGD) greedily picks tokens that are both relevant and dissimilar to those already chosen while respecting the per-cell quotas. Experiments on five backbones (LLaVA-1.5, LLaVA-NeXT, LLaVA-OneVision, Qwen2.5-VL, Qwen3-VL), six benchmarks and three budgets (5/11/22%) against nine pruning baselines report the best per-backbone average in 13 of 15 settings, with the largest gains at 5% retention (e.g., +5.4 pp on Qwen3-VL). The paper also reports substantial end-to-end speedups on high-resolution LLaVA backbones (up to 2.8×), ablations of supervision signals and selection rules, transfer to long-video QA without retraining, and partial token reuse in multi-turn dialog.

Critical strengths
  1. A broad, careful evaluation with a practical, low-overhead selector. The comparison spans five backbones, nine recent baselines, three budgets, answer preservation in addition to accuracy, paired bootstrap intervals, and end-to-end timing that charges every method its own selection cost. In this setting, Ansight's selector costs only 2–9 ms, while several strong baselines (CDPruner, ZOO-Prune, SCOPE at high resolution) lose most or all of their speedup to selection overhead. This makes the efficiency comparison more realistic than in most pruning papers.

  2. Well-supported findings on supervision and selection. The ablations show that answer-gradient targets outperform attention-, occlusion- and activation-norm targets for training a pre-LM selector. They also show that self-generated answers give targets nearly identical to ground-truth answers (Spearman 0.92), and that spatial allocation and diversity give consistent gains over global top-k at low budgets. These are useful, reusable lessons for learned token selection.

Critical weaknesses
  1. Several baselines look under-implemented, which makes the margins hard to trust. Except for LLaVA-1.5 (and MMTok/VisionZip on Qwen2.5-VL), baselines are the authors' own ports with substituted components, such as CDPruner without its CLIP relevance model and SCOPE without a class token on OneVision. Many of these ports fall below random selection at 5% retention:

    • Qwen3-VL: random 72.7, versus CDPruner 68.0, VisionZip 65.8, FastV 52.1.
    • LLaVA-OneVision: random 86.0, versus DivPrune 83.4, VisionZip 84.2, CDPruner 79.7.

    On LLaVA-1.5, official baseline scores are rescaled per benchmark by the ratio of full-model scores across different implementations, which is not a justified correction for pruned performance. Since the largest reported gains (5% on the Qwen models and OneVision) are exactly where the ported baselines are weakest, the true margin over well-implemented baselines is unclear.

  2. The headline results pick the best of three Ansight variants per setting. "Highest average in 13 of 15 settings" and the "26% smaller loss" both use the best of top-k, RA and RCGD per backbone and budget. With one fixed rule, RCGD wins 11 of 15 and RA 10. The variants also behave inconsistently across backbones. On Qwen2.5-VL at 5%, Ansight top-k (76.4) is below random (78.3), so the gain there comes from the diversity rule rather than the learned relevance. On LLaVA-1.5, Ansight does not beat CDPruner at 5% or 11% (95.3 vs. 95.5; 96.7 vs. 96.8). Appendix F suggests choosing the rule per backbone from held-out comparisons, which is an additional tuning step that the baselines do not get.

  3. The selector is barely question-conditioned in practice, which undercuts the "answer-sensitive" framing. On MMBench, the variant using only visual and position features is the best input configuration for both top-k and RA (Table 20), i.e., removing the question does not hurt. On long video, a fixed question-independent query performs as well as the current question (81.67% vs. 81.78%). Question features help mainly in multi-turn dialog. On standard single-question benchmarks, Ansight therefore behaves largely as a learned, query-agnostic image saliency model. This should be stated clearly, and compared with query-agnostic distilled selectors.

  4. The methodological novelty is incremental. Distilling explanation-derived relevance into a lightweight pre-LM selector was done by Lei et al. (2026), and MAP distills attention into a pre-LM predictor. G×A (Gradient×Input) is a standard attribution method, and the teacher ablations do not single it out. Gradient saliency and integrated gradients are comparable, and gradient saliency is better on LLaVA-1.5 MMBench and POPE (Tables 16–17). RA resembles the spatial budget allocation of HoloV, RoRA and SCOPE, and RCGD is a constrained version of relevance-weighted diversity selection (DivPrune, ZOO-Prune). The contribution is mainly a well-engineered combination.

  5. The efficiency claims are oversold relative to most backbones. The abstract's 8.6× compute, 94% KV reduction and 2.7× speedup refer to LLaVA-NeXT. On Qwen3-VL, Qwen2.5-VL and LLaVA-1.5, end-to-end speedups are 0.97–1.39×, with a slight slowdown on Qwen3-VL. FLOP reductions on the Qwen models are 48–64%. Timings use two output tokens, and with 64 output tokens the speedup on LLaVA-1.5 drops to 1.03×. The video speedups (up to 7.19×) exclude frame decoding and vision encoding. Since the vision encoder is never pruned, the high-resolution document speedup from raw images is only 1.03–1.20×.

  6. The training requirements weaken the "label-free" and plug-and-play positioning. Unlike the training-free baselines, Ansight requires a separately trained selector per backbone. The default training pool is filtered using LLaVA-1.5's correctness against ground-truth answers and DeFacto's annotated supporting regions. The unfiltered-pool ablation (Table 19) is reported only for Qwen3-VL, plus partially for OneVision. GQA is evaluated on a fixed 800-question subset rather than the full test-dev split, and the dialog experiments use only LLaVA-1.5.

Questions for authors
  1. Baseline fidelity. Why do several ported baselines (CDPruner, VisionZip, FastV, DivPrune) fall below random selection at 5% on Qwen3-VL and LLaVA-OneVision? Can you validate the ports against official implementations where they support these backbones? Can you also report the LLaVA-1.5 comparison without per-benchmark rescaling, i.e., all methods run in the same pipeline? This would most directly affect my assessment of the reported margins.

  2. A single pre-specified method. What are the cross-backbone results if one rule (e.g., RCGD) is fixed in advance for all settings? How should a practitioner choose between top-k, RA and RCGD for a new backbone without access to evaluation data?

  3. Role of the question. Given that visual + position features alone match or beat the full input on MMBench and that a fixed query works as well on video, what does question conditioning contribute on single-image benchmarks? How does Ansight compare with the same selector trained without question features, i.e., a distilled query-agnostic saliency model?

  4. Training cost and transfer. Can a selector trained for one backbone be reused on another of the same family? Does the unfiltered DeFacto pool give the same results on all five backbones? Can you report full GQA test-dev results instead of the 800-question subset?

  5. Efficiency in realistic settings. What are the end-to-end speedups on the Qwen backbones and LLaVA-1.5 with realistic output lengths and batched serving, and for video including encoding? Do you see a path to pruning before or inside the vision encoder?

6 comments

We thank the reviewer for the careful and constructive review, and for recognizing the breadth of the evaluation, the low overhead of the selector, and the value of the supervision and selection findings. Following the reviewer's suggestions, we compare Ansight with the official implementations of the baselines, clarify how to choose among the three selection variants, and add experiments on the role of the question, on training cost, and on efficiency. Points 2 and 3 give the results of RCGD alone on all five backbones, and the question ablation using RCGD.

1. Comparison with the baselines' released code (W1, Q1)

We first clarify the rescaling in Table 6, which changes only the raw scores, not the average relative score. Each rescaled score is the official score multiplied by our full-model score over the official full-model score, so dividing it by our full-model score gives back the official relative score. The average relative scores of the baselines in Table 6 are therefore those of their official code.

To also compare both methods in the same pipeline, as the reviewer requests, we ran Ansight inside the released code of each baseline, wherever a release supports the checkpoint, with one fixed variant, RCGD, and no rescaling. The table gives the score of the baseline and of Ansight at 5% retention, and their difference in pp. A score is the percentage of the full-model score, averaged over the six benchmarks of the paper. Both methods run in the pipeline of the baseline, so the score of Ansight varies slightly between rows. Each release preprocesses the image in its own way and has its own full-model score, so the tokens that Ansight selects from, and the score it is divided by, change from one release to the next. The last two columns divide the end-to-end speedup of Ansight + RCGD by that of the baseline at 5% and at 22% retention, so a value above 1 means that Ansight is faster.

BackboneReleased baselineBaselineAnsightDifferenceSpeed at 5%Speed at 22%
LLaVA-1.5FastV74.894.8+20.01.10×1.14×
LLaVA-1.5VisionZip87.294.0+6.90.98×0.98×
LLaVA-1.5DivPrune92.794.0+1.41.00×1.07×
LLaVA-1.5PruneSID91.695.5+3.81.38×1.54×
LLaVA-1.5ZOO-Prune93.594.0+0.62.34×2.40×
LLaVA-1.5SCOPE93.092.6−0.41.08×1.30×
LLaVA-1.5CDPruner95.594.5−0.91.18×1.53×
LLaVA-1.5MMTok93.492.7−0.71.01×1.08×
LLaVA-NeXTVisionZip87.392.4+5.00.98×0.95×
LLaVA-NeXTDivPrune91.192.4+1.41.00×1.20×
LLaVA-NeXTPruneSID89.992.4+2.41.01×1.52×
LLaVA-NeXTZOO-Prune90.092.3+2.46.24×5.60×
LLaVA-NeXTSCOPE92.992.6−0.41.42×1.95×
LLaVA-NeXTCDPruner97.697.9+0.42.52×2.47×
LLaVA-NeXTMMTok92.692.4−0.31.05×1.26×
Qwen2.5-VLVisionZip79.485.5+6.11.00×0.99×
Qwen2.5-VLDivPrune78.986.3+7.40.99×1.02×
Qwen2.5-VLMMTok80.585.9+5.41.00×1.04×

Here we report a baseline for a backbone only when its release supports that checkpoint.

On Qwen2.5-VL, where our gains at 5% are largest, Ansight is 5.4 to 7.4 pp above all three released selectors. On LLaVA-NeXT, it is above five of the seven released baselines at every budget, and within 1 pp of the other two. On LLaVA-1.5, it is above five of the eight official baselines at 5%, and within 0.9 pp of the other three. On the individual benchmarks, no baseline is significantly better than Ansight on LLaVA-NeXT or Qwen2.5-VL at any budget, where a difference is significant when its paired 95% bootstrap confidence interval excludes zero. It is also faster than CDPruner, SCOPE, ZOO-Prune, and PruneSID at every budget, by up to 2.52×, 1.95×, 6.24×, and 1.54×.

Qwen3-VL and LLaVA-OneVision have no released code, so the baselines there keep the selection signal published in each paper and change only its interface to the backbone, and the revised tables will mark them as such. To check the margins on these two backbones, we ran Ansight + RCGD, random selection, and the strongest baseline of each backbone in the paper, ZOO-Prune and SCOPE, on the benchmarks at 5% retention.

BackboneRandomZOO-PruneSCOPEAnsight + RCGD
Qwen3-VL74.583.777.489.2
LLaVA-OneVision86.588.691.393.2

Both baselines are above random selection on both backbones, and Ansight is 5.5 pp and 2.0 pp above the stronger one. It is significantly better than the stronger baseline on three of the six benchmarks on each backbone, and not significantly worse on any. SCOPE gives the same result against Ansight in our implementation and in its released code on LLaVA-NeXT, at −0.5 pp (Table 9) and −0.4 pp.

Why some baselines fall below random selection.

Wen et al. (ACL Findings 2025) report the same effect for FastV and SparseVLM, and the release of FastV is itself below random selection on LLaVA-1.5 (Table 6). To find the cause in each case, we changed one design element of the baseline and kept everything else fixed on 64 questions per benchmark:

BaselineElement we changedOneVisionQwen3-VL
Random selection82.667.3
FastVattention-selected tokens replaced by random ones67.5 → 82.753.1 → 67.0
VisionZipsaliency values shuffled across patches82.7 → 86.467.2 → 73.5
DivPrunefeature mean removed before cosine selection78.1 → 83.3–
CDPrunerquestion relevance replaced by a uniform one76.9 → 81.366.4 → 77.5

In every case, this one change lifts the baseline to about the level of random selection or above. The cause therefore lies in one element of each method that transfers poorly to these backbones at a very small budget. CDPruner has no official implementation for Qwen3-VL or LLaVA-OneVision, so its question representation has to be set for these backbones. For models without a paired text encoder, its paper specifies the language model's embeddings of the instruction tokens, which our OneVision implementation uses. On Qwen3-VL, CDPruner reaches 66.4% with the language model's pooled states of the question, and 77.8% with the instruction-token embeddings, the same level as with the uniform relevance. The revised Qwen3-VL tables will report the latter.

2. Choosing among top-k, RA, and RCGD (W2, Q2)

Top-k, RA, and RCGD select from the same trained selector and the same relevance scores, so they are three selection options and not three trained methods. The table gives the lead of RCGD alone over the strongest baseline in each setting, in average relative score.

RCGD minus strongest baseline (pp)5%11%22%
Qwen3-VL+5.4+2.7+1.7
Qwen2.5-VL+4.5+2.1+0.4
LLaVA-OneVision+1.5+1.1+0.3
LLaVA-NeXT−0.5−0.1+0.8
LLaVA-1.5−0.2−0.1+0.1

RCGD alone is ahead at every budget on three backbones, and within 0.5 pp of the strongest baseline on LLaVA-NeXT and LLaVA-1.5, for which point 1 gives the comparison in released code.

The variant for a new backbone can be chosen from the number of visual tokens that the backbone produces per image, which is known in advance and needs no evaluation data. On the three backbones with 260 to 576 visual tokens, RCGD is the best variant at every budget, and gains 3.7 to 9.4 pp over top-k at 5%. On the two high-resolution backbones, where 5% still keeps over 100 tokens, RA is the best variant in five of six settings (Tables 1 and 6–9). We will state this as an empirical guideline from the present experiments, in place of the advice in Appendix F. We will also state that "13 of 15" and "26% smaller loss" use the best of the three variants in each setting, and report each variant next to them.

On Qwen2.5-VL at 5%, only 17 tokens remain, and top-k falls below random selection. With the same relevance scores, RCGD is 9.4 pp above top-k and 7.5 pp above random. Diversity alone does not explain this result, as RCGD is 7.4 pp above DivPrune, which selects by diversity without relevance, in the released-code comparison of point 1, and significantly better on four of the six benchmarks. The gain therefore needs both the learned relevance and a selection that spreads the few kept tokens (Section 3.3).

3. Does the selector use the question? (W3, Q3)

The question matters when different questions about the same image need different visual evidence. To test this, we trained an otherwise identical selector without the question features, which is the query-agnostic distilled selector that the reviewer asks for. On LLaVA-1.5 with RA at 5%, the question features make no significant difference on MMBench, at −0.22 pp [−0.99, 0.53], as in Table 20. Here and below, brackets give paired 95% bootstrap confidence intervals. On GQA and POPE, where several questions refer to the same image, they improve answer preservation by 4.08 pp [2.08, 6.10] and by 3.05 pp [2.30, 3.80].

To measure this effect directly, we built a test in which only the question changes. From the GQA validation scene graphs, we built 400 pairs of questions that share an image and ask about different objects. Using RCGD for both selectors at 5%, the question features raise answer preservation from 62.96% to 75.46%, a gain of 12.50 pp [10.00, 15.04], and task accuracy from 49.00% to 56.12%, a gain of 7.12 pp [4.79, 9.50]. Therefore, the selector follows the object that the question asks about, and is not a fixed saliency ranking of the image. Multi-turn dialog shows the same effect, as selecting for the current question preserves 6.75 pp more answers at 5% than selecting from the image alone (Section 4.5). In long-video QA, a fixed query does as well as the question, so the gain there comes from the learned visual relevance. We will clarify that "answer-sensitive" describes the supervision, which is the sensitivity of the model's answer to each token.

4. Contributions and the choice of attribution signal (W4)

Prior work has studied explanation-based relevance, learned selection, spatial allocation, and diversity separately. Ansight shows how to obtain answer-relevant supervision from a frozen VLM, and how to distill it into a question-aware selector that runs before the language model, which gives three contributions.

  • Supervision. With the selector, the training data, and the optimization fixed, answer-gradient targets train better selectors than attention, occlusion, and similarity targets (Tables 15 and 16). On MMBench at 5%, G×A preserves 3.61 pp [2.66, 4.59] more answers than attention rollout. This advantage is largest at the lowest budget, as it is 0.6 pp at 11% and 0.5 pp at 22% (Table 16).
  • No answer labels. Targets from the model's own answers are close to those from ground-truth answers, with a Spearman correlation of 0.92.
  • Selection before the language model. The selector needs no attribution and no language-model attention at inference, and takes under 3 ms with top-k and 2 to 15 ms with RCGD (Tables 1 and 6–9). With the same training data on LLaVA-1.5, Ansight with top-k retains 89.6% of the full-model performance at 5%, against 74.6% for the method of Lei et al., on average over six benchmarks.

RA and RCGD share the principles of spatial allocation and diversity with the regional and diversity-based methods that the reviewer lists. They combine these principles with answer-sensitive relevance, as the token budget of each region follows the relevance predicted for the current image–question pair, and one diversity-aware selection then picks complementary tokens within these budgets before the language model. We will make these differences, and those from Lei et al. and MAP, explicit in the revision.

We agree that G×A is not the best answer-gradient signal in every setting. Integrated gradients preserves as many answers, and sometimes slightly more, but needs 16 backward passes per training example against one for G×A. Gradient saliency costs the same as G×A and is comparable on LLaVA-1.5, but preserves 6.2 to 9.2 pp fewer answers on Qwen3-VL at 5%, and is never significantly better after Holm correction (Table 17). We therefore use G×A, which balances cost and accuracy across backbones.

Reviewer_WJDyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an answer-sensitive visual pruning method for Vision-Language models (VLMs). The key idea is to leverage the gradient and activation of VLMs' answer log-probability to select question-related visual tokens. The authors distill such supervision signals into a small question-conditioned network to facilitate visual token selection. Moreover, region-aware token allocation is introduced to improve token diversity and preserve complementary evidence. Extensive experiments across five VLMs show that the proposed method achieves a good efficiency-accuracy trade-off.

Critical strengths

This paper shows that an answer-sensitive token pruning method can achieve promising performance across diverse tasks while considerably reducing inference cost.

Critical weaknesses

Insufficient comparisons with the closest prior work [1]. The introduction positions the paper as a follow-up study on which supervision signal best identifies question-related visual tokens. However, the paper only compares with [1] in Tables 3 and 16. Since the paper filters 855 pairs of samples that align with the proposed method for training, it is unclear whether Ansight is truly better than [1]. More comprehensive comparisons are necessary to demonstrate the effectiveness and superiority of the proposed method.

Reference

[1] Task-Related Token Compression in Multimodal Large Language Models from an Explainability Perspective. ICLR 2026.

Questions for authors
  1. The label-free supervision is somewhat overclaimed. While the method does not use ground-truth answers during training, the training pairs are selected using ground-truth answers and annotated evidence regions. A true label-free setting should only use a set of images without any annotations. In addition, it would be beneficial to report the training time.

  2. The proposed region-constrained global diversity yields limited methodological novelty. Prior works have already explored max-min diversity (DivPrune [2]) and asymmetric regional pruning (PosPrune [3]). The designs in the paper are similar to previous methods. Moreover, it is unclear whether the proposed RCGD is better than existing designs.

  3. It appears that RCGD does not necessarily outperform the naïve top-k variant. Table 1 shows that the average accuracy of Ansight + RCGD is consistently lower than Ansight top-k. A more in-depth analysis is recommended to demonstrate the necessity and superiority of RCGD.

Reference

[2] DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models. CVPR 2025.

[3] PosPrune: Visual Token Pruning with Positional Bias Correction for Efficient Large Vision-Language Models. AAAI 2026.

6 comments

We thank the reviewer for recognizing Ansight's efficiency–accuracy trade-off across diverse tasks, and for the thoughtful feedback on the comparison with Lei et al., on the label-free claim, and on RCGD. Following these comments, we extend the comparison with Lei et al. to all six benchmarks, clarify the role of annotations in training, and analyze RCGD across all five backbones and budgets.

1. Comparison with Lei et al. (W1)

Lei et al. show that explanation-derived relevance can be distilled into a lightweight predictor for token compression, as we note in Sections 1 and 2. Ansight extends this direction by asking which attribution signal gives effective supervision, and how relevance can be predicted without computing language-model attention at inference. Lei et al. supervise a convolutional predictor with gradient-weighted attention relevance, and its input is the first-layer attention from the instruction to the visual tokens. Ansight takes its supervision from the sensitivity of the answer log-probability to each visual token (G×A), and predicts it from the visual embeddings, the question features, and the token positions, which are all available before the language model.

To make the comparison more complete, we extended it from MMBench (Table 16) to all six benchmarks. The paper of Lei et al. does not come with an official implementation, so we carefully reimplemented their method, following the architecture, the attribution procedure, and the training objective that the paper describes. Both methods are trained on the same image–question pairs, with the same training/validation split and the same random seeds, and they are evaluated under the same conditions. The table gives the mean task score on LLaVA-1.5, as a percentage of the full model's score, over a fixed subset of 600 questions from each benchmark.

Method5%11%22%
Lei et al. (our reimplementation)74.679.186.4
Ansight + top-k89.695.296.4
Ansight + RCGD94.497.098.0

At 5% retention, Ansight with top-k selection retains 15.0 points more of the full model's performance than the reimplementation, and 19.8 points more with RCGD. Ansight + RCGD also preserves 16.1 points more of the full model's answers, with a paired 95% bootstrap interval of [14.8, 17.4]. We will include the results on each benchmark in the revision.

On the concern that the selection of our training pairs may favor Ansight, both methods receive exactly the same pairs in this comparison. The selection is also independent of the attribution scores of either method, as it depends only on how the frozen model's answer responds to masking the annotated supporting region (Appendix A.2). Lei et al. likewise keep only the training samples that the model answers correctly (their Appendix C). Ansight is as effective when trained on unfiltered data (Table 19).

The results published by Lei et al. offer an additional reference on a shared backbone. Their predictor is trained on 8K to 12K examples, depending on the model (their Appendix C). On LLaVA-OneVision-7B, which both papers evaluate, it retains on average 94.5% of the full-model performance at 25% retention on the four benchmarks that the two papers share, MME, POPE, TextVQA, and MMBench (their Table 2). On the same four benchmarks, Ansight + top-k retains 98.7% at 22% retention, and 96.9% at 11% (Table 1). As these results are reported independently, we give them as additional context, and the matched comparison above remains our main controlled comparison.

The paper also isolates the supervision signal from the predictor architecture (Tables 16 and 18). With the same MLP, training data, split, and optimization, replacing attention rollout with G×A raises MMBench answer preservation at 5% retention from 77.65% to 81.27%, a gain of 3.61 points [2.66, 4.59], and G×A is ahead in each of five training seeds.

2. Label-free supervision and training cost (Q1)

We agree with the reviewer's distinction between label-free target generation and annotation-free training, and we will make the terminology precise throughout the paper, including the abstract and the contributions. Ansight generates its supervision without ground-truth answers. For each image–question pair, the frozen VLM generates its own answer, and we compute G×A from the log-probability of that answer. Target generation thus needs image–question pairs, and neither annotated answers nor supporting regions. In our default setup, ground-truth answers and annotated supporting regions are used in a separate step, to choose which pairs enter the training set (Appendix A.2), and they are not used to compute the targets. We will thus describe the method as answer-label-free target generation.

We apply this selection step so that a small training set gives reliable targets, and the experiments in Table 19 show that it is not required. Qwen3-VL selectors trained on the complete unfiltered DeFacto pool, or on a larger mix of DeFacto with seven other VQA datasets, perform on par with the default setup. Selectors can also be trained without DeFacto altogether, as in our multi-turn dialog experiments (Appendix E).

As the reviewer suggests, we also report the training time. Generating the targets takes one answer generation and one backward pass through the frozen VLM per pair (Section 3.2). For Qwen3-VL, generating the targets and the selector inputs takes about 1.5 GPU-seconds per pair on an H100, or about 20 GPU-minutes for the whole training pool (Appendix D). Training a selector for 40 epochs on the default training set then takes 3 to 23 minutes on one GPU, depending on the backbone.

BackboneTraining time (minutes)
LLaVA-1.52.7
Qwen3-VL7.1
Qwen2.5-VL7.5
LLaVA-NeXT16.7
LLaVA-OneVision23.1

3. RCGD and prior work (Q2)

DivPrune and PosPrune are closely related to RCGD, and we will add PosPrune to the related work. Both address redundancy and spatial coverage among visual tokens, and RCGD shares these goals. RCGD differs in how relevance, regional allocation, and diversity are combined, and in where the selection takes place. PosPrune is a two-stage, training-free method. It first prunes by region at the vision encoder, and then applies positional-bias correction and relevance–diversity selection inside the language-model decoder. DivPrune selects tokens by max-min diversity to reduce redundancy. RCGD selects before the language model, with the relevance predicted for the current image–question pair. The token budget of each spatial region follows the predicted relevance in that region, and one global diversity-aware selection then picks complementary tokens within these regional budgets. The principles of spatial allocation and diversity are thus shared with prior work, and the contribution of RCGD is to combine them with answer-sensitive relevance in a single selection step before the language model.

To measure what the regional budgets add, we compare RCGD with relevance-weighted diversity selection without regional budgets. Both use the same predicted relevance scores, so the comparison isolates the regional constraint. On MMBench at 5% retention, the two are within 0.7 points in answer preservation on four backbones, and RCGD is 1.5 points higher on LLaVA-OneVision (Table 4). We will describe RCGD as region-constrained relevance–diversity selection.

4. RCGD compared with top-k (Q3)

Table 1 shows LLaVA-OneVision, where top-k has slightly higher average scores than RCGD. The results on all five backbones give a fuller picture. The table compares the average task score of top-k and RCGD, as a percentage of the full model's score (Tables 1 and 6–9).

Backbone, top-k → RCGD5%11%22%Median tokens kept at 5%
LLaVA-1.591.6 → 95.396.0 → 96.797.5 → 98.329
Qwen3-VL84.4 → 88.192.8 → 94.297.2 → 97.913
Qwen2.5-VL76.4 → 85.888.6 → 92.494.2 → 96.317
LLaVA-NeXT94.3 → 94.196.6 → 96.999.2 → 98.7115
LLaVA-OneVision92.9 → 92.896.7 → 95.998.5 → 97.9135

RCGD has the higher average at all three budgets on LLaVA-1.5, Qwen3-VL, and Qwen2.5-VL. Its largest gains are at 5% retention, where only 13 to 29 visual tokens remain, and range from 3.7 to 9.4 points. This matches the motivation for combining relevance with diversity. With so few tokens, the highest-scoring ones may carry redundant information, and RCGD makes the kept tokens cover complementary evidence. On the two high-resolution backbones, 5% retention still keeps over 100 tokens, and the two methods stay within one point of each other.

Answer preservation, which measures whether pruning changes the full model's answer, shows the same behavior. At 5% retention, with the predicted relevance scores fixed, RCGD preserves more answers than top-k on all five backbones.

BackboneGain in preservation [95% interval]
LLaVA-1.5+2.2 [1.0, 3.3]
Qwen3-VL+4.1 [2.8, 5.3]
Qwen2.5-VL+5.6 [4.3, 7.0]
LLaVA-NeXT+2.3 [1.2, 3.4]
LLaVA-OneVision+1.3 [0.3, 2.4]

These are equal-weight averages over the 600-question subsets of the six benchmarks.

In the revision, we will report all three variants for every backbone and budget, and state the guideline that these results suggest, which is RCGD for backbones with a few hundred visual tokens and Regional Allocation (RA) for the high-resolution backbones.

Reviewer_kKJrOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Ansight, a question-aware visual token pruning framework designed to preserve task-relevant visual information while reducing the computational overhead of vision-language models (VLMs). It uses gradient × activation (G×A) attribution to generate supervision signals for training a lightweight MLP-based selector, and introduces regional allocation and region-constrained global diversity to balance spatial coverage and feature diversity during token selection. Extensive experiments across multiple VLM backbones, benchmarks, and retention budgets demonstrate the effectiveness of the proposed framework.

Critical strengths
  1. The paper writing is clear and easy to follow.

  2. The motivation is reasonable and the design is insightful.

  3. The experimental evaluation is comprehensive.

Critical weaknesses
  1. The choice of G ×\times A as the signal appears to be an intuition. I suggest the authors to provide some theoretical support or qualitative visualizations of G ×\times A attribution maps. The could help clarify the reason of the choice.

  2. The trained MLP is much lighter than the VLM itself. So can this MLP capture fine-grained semantic relations? For example, can it reliably distinguish between subtly different questions, like small changes in wording about direction, color, spatial relations, and etc, about the same image? If it fails to identify the relations, how would this affect the quality of the final output?

Questions for authors

See weaknesses.

2 comments

We thank the reviewer for the positive assessment of the paper's clarity, motivation, and evaluation, and for the two suggestions. Below, we give a first-order derivation for Gradient × Activation (G×A), and new analyses of whether the lightweight selector follows the question.

1. Why G×A

G×A measures how sensitive the log-probability of the answer is to a change in each visual token. Let JJ be the frozen model's log-probability of its generated answer, viv_i a visual token, and gi=∇viJg_i=\nabla_{v_i}J. Consider a bounded relative perturbation δvi=vi⊙η\delta v_i=v_i\odot\eta with ∥η∥∞≤ϵ\|\eta\|_\infty\leq\epsilon. A first-order Taylor expansion gives ΔJ≈(vi⊙gi)⊤η\Delta J\approx(v_i\odot g_i)^\top\eta. By Hölder's inequality,

max⁡∥η∥∞≤ϵ∣(vi⊙gi)⊤η∣=ϵ∥vi⊙gi∥1=ϵ ai,\max_{\|\eta\|_\infty\leq\epsilon}|(v_i\odot g_i)^\top\eta|=\epsilon\|v_i\odot g_i\|_1=\epsilon\,a_i,

with equality at η=ϵ sign(vi⊙gi)\eta=\epsilon\,\mathrm{sign}(v_i\odot g_i), where aia_i is our score in Eq. 1. Our score is thus the largest first-order change in the answer log-probability that a bounded relative change of token ii can cause. The same expansion underlies the Taylor pruning criterion of Molchanov et al. (ICLR 2017), which scores a feature by the product of its gradient and its activation. As a first-order approximation, it motivates how we rank the tokens, and it does not predict the exact effect of deleting one.

The experiments in the paper support this choice. With the selector architecture, the training data, and the optimization fixed, G×A preserves 3.61 points [2.66, 4.59] more MMBench answers than attention rollout at 5% retention, and it is ahead in each of five training seeds (Table 18). It is also ahead of local occlusion and prompt attention, by 8.5 and 15.0 points (Table 16). A selector trained on the gradient alone is similar on LLaVA-1.5 MMBench, with 82.11% preservation at 5% against 81.27% for G×A. On Qwen3-VL, however, it preserves 6.2 to 9.2 points fewer answers at 5% on all three benchmarks, and it is never significantly better than G×A (Table 17).

As a new analysis, we also checked whether G×A highlights the visual evidence for the answer. On DeFacto eval set, with annotated supporting regions, with LLaVA-1.5, G×A separates the tokens inside the supporting region from those outside with an AUROC of 0.815, against 0.777 for attention rollout. We will add attribution maps, including failure cases, to the revision.

2. Can the lightweight selector distinguish fine-grained questions?

The selector's role is to decide which visual tokens to keep for a given question. The language model then receives the complete question, and does the reasoning that produces the answer. What matters is thus whether the selector changes the tokens it keeps when the question needs different visual evidence. We test this on standard benchmarks and with two new controlled experiments.

First, we measure what the question features contribute on standard benchmarks. We train an otherwise identical selector without the question features, and compare it with the full selector on LLaVA-1.5, with RA at 5% retention. On MMBench, as in Table 20, there is no conclusive difference in answer preservation, at +0.22 points [−0.53, 0.99] without the question. We now extend this comparison to GQA and POPE, where the question features add 4.08 points [2.08, 6.10] and 3.05 points [2.30, 3.80]. Unlike MMBench, these two benchmarks ask several questions about the same image, so the selector has to find the evidence for the current question.

Second, we built a controlled test for the case that the reviewer describes, different questions about the same image. To isolate the effect of the question, we need pairs of questions that share an image and differ in the object they ask about. We built 400 such pairs, with 800 questions in total, from GQA validation scene graphs, for example two questions about the colors of two different objects. On LLaVA-1.5 at 5% retention with RCGD, we compare selectors trained with and without the question features, with three training seeds each.

With questionWithout questionDifference [95% interval]
Answer preservation75.46%62.96%+12.50 [10.00, 15.04]
Task accuracy56.12%49.00%+7.12 [4.79, 9.50]

Intervals are paired bootstrap intervals over question pairs, with the seeds averaged. The selector therefore uses the question to find different objects in the same image, and is not a fixed saliency ranking of the image.

Not every change in wording needs different visual tokens. For instance, two questions about the left/right relation between the same two objects both need these two objects, whichever relation they ask about. In a second controlled test with 400 such pairs, removing the question features gives no conclusive difference in task accuracy, at +1.04 points [−0.08, 2.21]. This is consistent with the selector keeping the same evidence, and the language model resolving the relation from the complete question.

Third, we measure what happens when the selector picks the evidence for the wrong question. On the different-object pairs, we select the tokens with the partner question in place of the current one. At 5% retention, this lowers answer preservation by 4.8 points [1.6, 8.0] on LLaVA-1.5, 5.9 [2.9, 8.8] on Qwen3-VL, and 3.4 [0.8, 6.0] on Qwen2.5-VL. The differences on LLaVA-NeXT and OneVision are inconclusive. This measures the cost of mismatched evidence, not how often it occurs in normal inference.

Finally, the reviewer asks whether such a light MLP has enough capacity. In our ablations, wider or deeper MLPs, bilinear scoring, and pooled cross-attention do not improve preservation (Table 21), so we see no sign that its size limits the selector. The question enters as a mean-pooled embedding, which may lose word order, and none of these richer scorers improves on it.

Question conditioning thus matters when questions need different visual evidence, and the language model handles the finer semantic and relational distinctions. We will add the derivation, the attribution maps, and these analyses to the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.