- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Ansight, a learned visual-token selector for VLMs. Offline, the frozen VLM generates its own answer for an image–question pair, and a teacher-forced backward pass computes a Gradient × Activation (G×A) attribution of the answer log-probability for each visual token. A small MLP is distilled to predict these attributions before the language model, from the visual token, the mean question embedding, their interaction, and a Fourier position encoding. One selector is trained per backbone on a few hundred DeFacto image–question pairs, without ground-truth answer labels for the targets. At inference, two selection rules sit on top of the predicted relevance: Regional Allocation (RA) splits the budget across fixed 4×4 cells in proportion to predicted relevance mass, and Region-Constrained Global Diversity (RCGD) greedily picks tokens that are both relevant and dissimilar to those already chosen while respecting the per-cell quotas. Experiments on five backbones (LLaVA-1.5, LLaVA-NeXT, LLaVA-OneVision, Qwen2.5-VL, Qwen3-VL), six benchmarks and three budgets (5/11/22%) against nine pruning baselines report the best per-backbone average in 13 of 15 settings, with the largest gains at 5% retention (e.g., +5.4 pp on Qwen3-VL). The paper also reports substantial end-to-end speedups on high-resolution LLaVA backbones (up to 2.8×), ablations of supervision signals and selection rules, transfer to long-video QA without retraining, and partial token reuse in multi-turn dialog.
Critical strengths
-
A broad, careful evaluation with a practical, low-overhead selector. The comparison spans five backbones, nine recent baselines, three budgets, answer preservation in addition to accuracy, paired bootstrap intervals, and end-to-end timing that charges every method its own selection cost. In this setting, Ansight's selector costs only 2–9 ms, while several strong baselines (CDPruner, ZOO-Prune, SCOPE at high resolution) lose most or all of their speedup to selection overhead. This makes the efficiency comparison more realistic than in most pruning papers.
-
Well-supported findings on supervision and selection. The ablations show that answer-gradient targets outperform attention-, occlusion- and activation-norm targets for training a pre-LM selector. They also show that self-generated answers give targets nearly identical to ground-truth answers (Spearman 0.92), and that spatial allocation and diversity give consistent gains over global top-k at low budgets. These are useful, reusable lessons for learned token selection.
Critical weaknesses
-
Several baselines look under-implemented, which makes the margins hard to trust. Except for LLaVA-1.5 (and MMTok/VisionZip on Qwen2.5-VL), baselines are the authors' own ports with substituted components, such as CDPruner without its CLIP relevance model and SCOPE without a class token on OneVision. Many of these ports fall below random selection at 5% retention:
- Qwen3-VL: random 72.7, versus CDPruner 68.0, VisionZip 65.8, FastV 52.1.
- LLaVA-OneVision: random 86.0, versus DivPrune 83.4, VisionZip 84.2, CDPruner 79.7.
On LLaVA-1.5, official baseline scores are rescaled per benchmark by the ratio of full-model scores across different implementations, which is not a justified correction for pruned performance. Since the largest reported gains (5% on the Qwen models and OneVision) are exactly where the ported baselines are weakest, the true margin over well-implemented baselines is unclear.
-
The headline results pick the best of three Ansight variants per setting. "Highest average in 13 of 15 settings" and the "26% smaller loss" both use the best of top-k, RA and RCGD per backbone and budget. With one fixed rule, RCGD wins 11 of 15 and RA 10. The variants also behave inconsistently across backbones. On Qwen2.5-VL at 5%, Ansight top-k (76.4) is below random (78.3), so the gain there comes from the diversity rule rather than the learned relevance. On LLaVA-1.5, Ansight does not beat CDPruner at 5% or 11% (95.3 vs. 95.5; 96.7 vs. 96.8). Appendix F suggests choosing the rule per backbone from held-out comparisons, which is an additional tuning step that the baselines do not get.
-
The selector is barely question-conditioned in practice, which undercuts the "answer-sensitive" framing. On MMBench, the variant using only visual and position features is the best input configuration for both top-k and RA (Table 20), i.e., removing the question does not hurt. On long video, a fixed question-independent query performs as well as the current question (81.67% vs. 81.78%). Question features help mainly in multi-turn dialog. On standard single-question benchmarks, Ansight therefore behaves largely as a learned, query-agnostic image saliency model. This should be stated clearly, and compared with query-agnostic distilled selectors.
-
The methodological novelty is incremental. Distilling explanation-derived relevance into a lightweight pre-LM selector was done by Lei et al. (2026), and MAP distills attention into a pre-LM predictor. G×A (Gradient×Input) is a standard attribution method, and the teacher ablations do not single it out. Gradient saliency and integrated gradients are comparable, and gradient saliency is better on LLaVA-1.5 MMBench and POPE (Tables 16–17). RA resembles the spatial budget allocation of HoloV, RoRA and SCOPE, and RCGD is a constrained version of relevance-weighted diversity selection (DivPrune, ZOO-Prune). The contribution is mainly a well-engineered combination.
-
The efficiency claims are oversold relative to most backbones. The abstract's 8.6× compute, 94% KV reduction and 2.7× speedup refer to LLaVA-NeXT. On Qwen3-VL, Qwen2.5-VL and LLaVA-1.5, end-to-end speedups are 0.97–1.39×, with a slight slowdown on Qwen3-VL. FLOP reductions on the Qwen models are 48–64%. Timings use two output tokens, and with 64 output tokens the speedup on LLaVA-1.5 drops to 1.03×. The video speedups (up to 7.19×) exclude frame decoding and vision encoding. Since the vision encoder is never pruned, the high-resolution document speedup from raw images is only 1.03–1.20×.
-
The training requirements weaken the "label-free" and plug-and-play positioning. Unlike the training-free baselines, Ansight requires a separately trained selector per backbone. The default training pool is filtered using LLaVA-1.5's correctness against ground-truth answers and DeFacto's annotated supporting regions. The unfiltered-pool ablation (Table 19) is reported only for Qwen3-VL, plus partially for OneVision. GQA is evaluated on a fixed 800-question subset rather than the full test-dev split, and the dialog experiments use only LLaVA-1.5.
Questions for authors
-
Baseline fidelity. Why do several ported baselines (CDPruner, VisionZip, FastV, DivPrune) fall below random selection at 5% on Qwen3-VL and LLaVA-OneVision? Can you validate the ports against official implementations where they support these backbones? Can you also report the LLaVA-1.5 comparison without per-benchmark rescaling, i.e., all methods run in the same pipeline? This would most directly affect my assessment of the reported margins.
-
A single pre-specified method. What are the cross-backbone results if one rule (e.g., RCGD) is fixed in advance for all settings? How should a practitioner choose between top-k, RA and RCGD for a new backbone without access to evaluation data?
-
Role of the question. Given that visual + position features alone match or beat the full input on MMBench and that a fixed query works as well on video, what does question conditioning contribute on single-image benchmarks? How does Ansight compare with the same selector trained without question features, i.e., a distilled query-agnostic saliency model?
-
Training cost and transfer. Can a selector trained for one backbone be reused on another of the same family? Does the unfiltered DeFacto pool give the same results on all five backbones? Can you report full GQA test-dev results instead of the 800-question subset?
-
Efficiency in realistic settings. What are the end-to-end speedups on the Qwen backbones and LLaVA-1.5 with realistic output lengths and batched serving, and for video including encoding? Do you see a path to pruning before or inside the vision encoder?
6 comments
Qwen3-VL and LLaVA-OneVision have no released code, so the baselines there keep the selection signal published in each paper and change only its interface to the backbone, and the revised tables will mark them as such. To check the margins on these two backbones, we ran Ansight + RCGD, random selection, and the strongest baseline of each backbone in the paper, ZOO-Prune and SCOPE, on the benchmarks at 5% retention.
| Backbone | Random | ZOO-Prune | SCOPE | Ansight + RCGD |
|---|---|---|---|---|
| Qwen3-VL | 74.5 | 83.7 | 77.4 | 89.2 |
| LLaVA-OneVision | 86.5 | 88.6 | 91.3 | 93.2 |
Both baselines are above random selection on both backbones, and Ansight is 5.5 pp and 2.0 pp above the stronger one. It is significantly better than the stronger baseline on three of the six benchmarks on each backbone, and not significantly worse on any. SCOPE gives the same result against Ansight in our implementation and in its released code on LLaVA-NeXT, at −0.5 pp (Table 9) and −0.4 pp.
Why some baselines fall below random selection.
Wen et al. (ACL Findings 2025) report the same effect for FastV and SparseVLM, and the release of FastV is itself below random selection on LLaVA-1.5 (Table 6). To find the cause in each case, we changed one design element of the baseline and kept everything else fixed on 64 questions per benchmark:
| Baseline | Element we changed | OneVision | Qwen3-VL |
|---|---|---|---|
| Random selection | 82.6 | 67.3 | |
| FastV | attention-selected tokens replaced by random ones | 67.5 → 82.7 | 53.1 → 67.0 |
| VisionZip | saliency values shuffled across patches | 82.7 → 86.4 | 67.2 → 73.5 |
| DivPrune | feature mean removed before cosine selection | 78.1 → 83.3 | – |
| CDPruner | question relevance replaced by a uniform one | 76.9 → 81.3 | 66.4 → 77.5 |
In every case, this one change lifts the baseline to about the level of random selection or above. The cause therefore lies in one element of each method that transfers poorly to these backbones at a very small budget. CDPruner has no official implementation for Qwen3-VL or LLaVA-OneVision, so its question representation has to be set for these backbones. For models without a paired text encoder, its paper specifies the language model's embeddings of the instruction tokens, which our OneVision implementation uses. On Qwen3-VL, CDPruner reaches 66.4% with the language model's pooled states of the question, and 77.8% with the instruction-token embeddings, the same level as with the uniform relevance. The revised Qwen3-VL tables will report the latter.
2. Choosing among top-k, RA, and RCGD (W2, Q2)
Top-k, RA, and RCGD select from the same trained selector and the same relevance scores, so they are three selection options and not three trained methods. The table gives the lead of RCGD alone over the strongest baseline in each setting, in average relative score.
| RCGD minus strongest baseline (pp) | 5% | 11% | 22% |
|---|---|---|---|
| Qwen3-VL | +5.4 | +2.7 | +1.7 |
| Qwen2.5-VL | +4.5 | +2.1 | +0.4 |
| LLaVA-OneVision | +1.5 | +1.1 | +0.3 |
| LLaVA-NeXT | −0.5 | −0.1 | +0.8 |
| LLaVA-1.5 | −0.2 | −0.1 | +0.1 |
RCGD alone is ahead at every budget on three backbones, and within 0.5 pp of the strongest baseline on LLaVA-NeXT and LLaVA-1.5, for which point 1 gives the comparison in released code.
The variant for a new backbone can be chosen from the number of visual tokens that the backbone produces per image, which is known in advance and needs no evaluation data. On the three backbones with 260 to 576 visual tokens, RCGD is the best variant at every budget, and gains 3.7 to 9.4 pp over top-k at 5%. On the two high-resolution backbones, where 5% still keeps over 100 tokens, RA is the best variant in five of six settings (Tables 1 and 6–9). We will state this as an empirical guideline from the present experiments, in place of the advice in Appendix F. We will also state that "13 of 15" and "26% smaller loss" use the best of the three variants in each setting, and report each variant next to them.
On Qwen2.5-VL at 5%, only 17 tokens remain, and top-k falls below random selection. With the same relevance scores, RCGD is 9.4 pp above top-k and 7.5 pp above random. Diversity alone does not explain this result, as RCGD is 7.4 pp above DivPrune, which selects by diversity without relevance, in the released-code comparison of point 1, and significantly better on four of the six benchmarks. The gain therefore needs both the learned relevance and a selection that spreads the few kept tokens (Section 3.3).
3. Does the selector use the question? (W3, Q3)
The question matters when different questions about the same image need different visual evidence. To test this, we trained an otherwise identical selector without the question features, which is the query-agnostic distilled selector that the reviewer asks for. On LLaVA-1.5 with RA at 5%, the question features make no significant difference on MMBench, at −0.22 pp [−0.99, 0.53], as in Table 20. Here and below, brackets give paired 95% bootstrap confidence intervals. On GQA and POPE, where several questions refer to the same image, they improve answer preservation by 4.08 pp [2.08, 6.10] and by 3.05 pp [2.30, 3.80].
To measure this effect directly, we built a test in which only the question changes. From the GQA validation scene graphs, we built 400 pairs of questions that share an image and ask about different objects. Using RCGD for both selectors at 5%, the question features raise answer preservation from 62.96% to 75.46%, a gain of 12.50 pp [10.00, 15.04], and task accuracy from 49.00% to 56.12%, a gain of 7.12 pp [4.79, 9.50]. Therefore, the selector follows the object that the question asks about, and is not a fixed saliency ranking of the image. Multi-turn dialog shows the same effect, as selecting for the current question preserves 6.75 pp more answers at 5% than selecting from the image alone (Section 4.5). In long-video QA, a fixed query does as well as the question, so the gain there comes from the learned visual relevance. We will clarify that "answer-sensitive" describes the supervision, which is the sensitivity of the model's answer to each token.
4. Contributions and the choice of attribution signal (W4)
Prior work has studied explanation-based relevance, learned selection, spatial allocation, and diversity separately. Ansight shows how to obtain answer-relevant supervision from a frozen VLM, and how to distill it into a question-aware selector that runs before the language model, which gives three contributions.
- Supervision. With the selector, the training data, and the optimization fixed, answer-gradient targets train better selectors than attention, occlusion, and similarity targets (Tables 15 and 16). On MMBench at 5%, G×A preserves 3.61 pp [2.66, 4.59] more answers than attention rollout. This advantage is largest at the lowest budget, as it is 0.6 pp at 11% and 0.5 pp at 22% (Table 16).
- No answer labels. Targets from the model's own answers are close to those from ground-truth answers, with a Spearman correlation of 0.92.
- Selection before the language model. The selector needs no attribution and no language-model attention at inference, and takes under 3 ms with top-k and 2 to 15 ms with RCGD (Tables 1 and 6–9). With the same training data on LLaVA-1.5, Ansight with top-k retains 89.6% of the full-model performance at 5%, against 74.6% for the method of Lei et al., on average over six benchmarks.
RA and RCGD share the principles of spatial allocation and diversity with the regional and diversity-based methods that the reviewer lists. They combine these principles with answer-sensitive relevance, as the token budget of each region follows the relevance predicted for the current image–question pair, and one diversity-aware selection then picks complementary tokens within these budgets before the language model. We will make these differences, and those from Lei et al. and MAP, explicit in the revision.
We agree that G×A is not the best answer-gradient signal in every setting. Integrated gradients preserves as many answers, and sometimes slightly more, but needs 16 backward passes per training example against one for G×A. Gradient saliency costs the same as G×A and is comparable on LLaVA-1.5, but preserves 6.2 to 9.2 pp fewer answers on Qwen3-VL at 5%, and is never significantly better after Holm correction (Table 17). We therefore use G×A, which balances cost and accuracy across backbones.
We thank the reviewer for the careful and constructive review, and for recognizing the breadth of the evaluation, the low overhead of the selector, and the value of the supervision and selection findings. Following the reviewer's suggestions, we compare Ansight with the official implementations of the baselines, clarify how to choose among the three selection variants, and add experiments on the role of the question, on training cost, and on efficiency. Points 2 and 3 give the results of RCGD alone on all five backbones, and the question ablation using RCGD.
1. Comparison with the baselines' released code (W1, Q1)
We first clarify the rescaling in Table 6, which changes only the raw scores, not the average relative score. Each rescaled score is the official score multiplied by our full-model score over the official full-model score, so dividing it by our full-model score gives back the official relative score. The average relative scores of the baselines in Table 6 are therefore those of their official code.
To also compare both methods in the same pipeline, as the reviewer requests, we ran Ansight inside the released code of each baseline, wherever a release supports the checkpoint, with one fixed variant, RCGD, and no rescaling. The table gives the score of the baseline and of Ansight at 5% retention, and their difference in pp. A score is the percentage of the full-model score, averaged over the six benchmarks of the paper. Both methods run in the pipeline of the baseline, so the score of Ansight varies slightly between rows. Each release preprocesses the image in its own way and has its own full-model score, so the tokens that Ansight selects from, and the score it is divided by, change from one release to the next. The last two columns divide the end-to-end speedup of Ansight + RCGD by that of the baseline at 5% and at 22% retention, so a value above 1 means that Ansight is faster.
Here we report a baseline for a backbone only when its release supports that checkpoint.
On Qwen2.5-VL, where our gains at 5% are largest, Ansight is 5.4 to 7.4 pp above all three released selectors. On LLaVA-NeXT, it is above five of the seven released baselines at every budget, and within 1 pp of the other two. On LLaVA-1.5, it is above five of the eight official baselines at 5%, and within 0.9 pp of the other three. On the individual benchmarks, no baseline is significantly better than Ansight on LLaVA-NeXT or Qwen2.5-VL at any budget, where a difference is significant when its paired 95% bootstrap confidence interval excludes zero. It is also faster than CDPruner, SCOPE, ZOO-Prune, and PruneSID at every budget, by up to 2.52×, 1.95×, 6.24×, and 1.54×.