acceptodds
Under review as a conference paper at ICLR 2027

Diagnosing Query Awareness in Visual Token Pruning: From Biased Importance Signals to Calibrated Selection

Abstract

Visual-token pruning for VLMs relies on importance scores that are often treated as proxies for question-relevant evidence. Popular scores can be divided into image-conditioned visual priors and query-conditioned signals represented by early text-visual attention. Their high aggregate fidelity has been reported, yet aggregate fidelity alone cannot reveal what these scores measure and limits understanding. In this work, we conduct a structured diagnostic analysis along three axes: prompt sensitivity, selection-behavior preference, and depth-wise query formation. Across multiple VLMs, early text-visual attention exhibits little query discrimination despite being prompt-conditioned in construction. It behaves as an image-conditioned local-detail prior, while visual priors favor broader spatial coverage; the two families succeed on complementary global- and local-evidence regimes. Diversity-aware selection reduces redundancy but does not remove these inherited biases. Reliable query-discriminative attention emerges only in middle-to-late language-model layers. These findings motivate Query-Calibrated Pruning (Q-Cal), a two-stage framework for multi-round VLM inference. Q-Cal first forms a reusable visual prefix from complementary early priors, then refines it with a calibrated mixture of deeper text-visual attention once query-specific evidence has formed. Across six Qwen-family VLMs and eight benchmarks, Q-Cal improves both the accuracy and efficiency of visual pruning for multi-round questions. Code is available at https://github.com/anonymous733882/q-cal-submission.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Models, Visual Token Pruning, Efficient Multimodal Inference, Query-Aware Token Selection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_baheOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Multimodal large language models pay a large amount of visual tokens for every image, and pruning is the standard way to control inference cost, with importance scores drawn either from image-side priors or from text-to-visual attention. The paper first asks whether these scores really change with the question, and finds that the early text-to-visual attention that is nominally question-conditioned stays almost identical across different questions and favors local detail, whereas visual priors favor global coverage and attention that can truly discriminate between questions appears only in the middle-to-late decoder layers. Based on this diagnosis the paper proposes a two-stage pruning framework that first builds a conservative visual prefix reusable across questions from two complementary early signals and then refines it per question within that prefix using a calibrated combination of middle-to-late attention signals. Across several open-source vision-language models and several visual question-answering benchmarks, the framework is compared with existing pruning methods under a matched cost budget, with ablations over the two stages and the signal sources, and it is reported to achieve a better accuracy-cost trade-off at low retention rates.

Critical strengths
  1. The diagnostic study of existing pruning signals is consistent across models, signals and evidence types, and is the strongest part of the paper.
  2. The paper provides unusually concrete reproduction and comparison material: the full calibration objective (Eq. 6-13), per-backbone Stage-2 mixture weights (p.14 Table 4), the calibration-source substitution study (p.9 Figure 6b: eight single-source and two/three/four-source substitutions, maximum deviation 0.016), and a baseline reproduction audit for 14 methods (p.17 Table 7, e.g., AgilePruner TextVQA 79.80 to 84.38, FastV A-OKVQA 77.00 to 84.33).
Critical weaknesses
  1. The paper's formatting is very poor, with too many lines that are less than 80% filled.
  2. On InViG, Q-Cal at 5-10% visual-token retention is above the unpruned full-token reference on all six backbones (up to about 67% relative gain), and the paper neither explains this column nor defines how it is scored.
  3. At the 5%/10% final budgets, on the same hardware and batch size, what are the measured prefill and decoding latency, throughput and memory of Q-Cal relative to the strongest cost-matched baseline (e.g., SparseVILA, ID-Selection)?
  4. At the six backbones, eight benchmarks and the same matched-C protocol of Table 2, what are the accuracies of (i) Stage 1 only (no Stage 2) and (ii) the best single deep-attention layer replacing the calibrated mixture, and what is the gain of the calibrated mixture over them? I would consider raising my score if the authors could address these issues.
Questions for authors

Please see weakness.

1 comment
AuthorsOpenReview ↗

Response to Reviewer bahe: layout fixes, InViG bug fix, serving measurements, and matched-cost ablations

W1: Formatting and short lines

Thank you for pointing out this shortcoming. We readjusted the layout of the paper to avoid short lines. In the main text (pages 1–9), the number of body lines filled to less than 80% dropped from 23 in the submission to 8. All of them are ordinary paragraph endings or the lead-in of a list.

W2: InViG scores above Full

Thank you for spotting this anomaly; it indeed required an implementation audit. It was caused by a code bug: when the prefix KV cache was reused across the questions of an image, the attention mask was misaligned, which suppressed the scores of Full and of the baselines; InViG was affected the most.

We fixed the bug, unified the pipeline, and reran all cells for all methods. After the correction, Q-Cal no longer exceeds Full on InViG on any backbone. The revised values are in Table 2 (p. 8, l. 378) and Table 9 (p. 19, l. 1005), with all Avg. Fid. values recomputed. We apologize for the original evaluation errors.

W3: Measured latency, throughput and memory

Thank you for the suggestion. We strengthened the efficiency analysis with measured latency, throughput and memory. Appendix H (p. 21, l. 1130) reports prefill latency, decoding latency, throughput and peak memory of Full, Q-Cal and all eleven baselines on the nine backbones (the six Qwen-VL backbones and the three new ones), on the same hardware, at fixed batch sizes B=1,4,16,64B=1,4,16,64 and both budgets (Tables 10–13, p. 23, l. 1188–p. 26, l. 1350). Table R6 summarizes the comparison with the strongest cost-matched baselines.

Table R6: Serving cost at B=64B=64, mean over six Qwen-VL backbones (from Tables 10–13; 5% / 10%).

FullQ-CalID-SelectionSparseVILA
Prefill (ms / question)25282053 / 21761694 / 17711800 / 1822
Decoding (ms / step)113.033.7 / 37.438.8 / 40.936.9 / 36.8
Throughput (questions / s)7.0715.26 / 14.2415.57 / 14.6615.71 / 15.60
Peak memory (GB)25.616.8 / 16.915.8 / 16.116.9 / 17.2

Q-Cal has a clear efficiency advantage over the full-token model: at B=64B=64 it decodes 3.4$\times$ faster, reaches 2.2$\times$ the throughput and uses 1.5$\times$ less memory. Compared with SparseVILA and ID-Selection, Q-Cal is the most efficient in decoding, because its decoding cache holds only the final 5% or 10% of the visual tokens, whereas the one-shot baselines keep 13–16% to match the same cost CC: at 5%, ID-Selection needs 15% and SparseVILA 9% more time per decoding step. Its throughput and memory are on par with these baselines (throughput within 5% of ID-Selection at every batch size). Its prefill is slower, since Q-Cal scores every question at layer SS whereas ID-Selection prunes once before the language model.

W4: Stage 1 only and best single layer at matched cost

Thank you for the suggestion. We added to the appendix a complete CC-matched comparison of Stage 1 only, the best single deep-attention layer and the calibrated mixture on the nine backbones and eight benchmarks at both budgets (Appendix I, p. 27, l. 1404; Table 14, p. 27, l. 1420). Stage 1 only keeps the Stage-1 selection with the number of visual tokens whose total cost matches CC; the single layer keeps Q-Cal's pruning position SS and final budget but scores the tokens with the best single layer. Table R7 summarizes the results; "Wins" counts the benchmark–budget cells (16 per backbone) in which each setting is the most accurate (ties counted for each).

Table R7: Avg. Fid. (%, 5% / 10%) and wins at matched cost (from Table 14).

BackboneStage 1 onlySingle layerCalibrated mixtureWins (S1 / Single / Cal. mix.)
Qwen2-VL-2B95.2 / 96.296.4 / 97.296.8 / 97.02 / 7 / 9
Qwen2-VL-7B97.1 / 97.797.4 / 97.597.0 / 97.44 / 8 / 6
Qwen2.5-VL-3B94.2 / 96.395.2 / 96.896.1 / 96.86 / 4 / 11
Qwen2.5-VL-7B95.0 / 96.897.7 / 98.897.9 / 98.51 / 8 / 11
Qwen3-VL-4B92.8 / 94.597.0 / 97.296.9 / 97.30 / 11 / 10
Qwen3-VL-8B90.9 / 93.196.7 / 97.396.6 / 97.30 / 8 / 10
Qwen3.6-27B93.1 / 95.297.6 / 98.197.5 / 97.70 / 10 / 7
Qwen3.8-27B91.5 / 94.698.0 / 97.496.9 / 97.70 / 8 / 8
Gemma-4-31B82.4 / 86.494.9 / 94.894.7 / 94.90 / 8 / 12
Mean / total92.5 / 94.596.8 / 97.296.7 / 97.213 / 72 / 84

In average fidelity, both the best single layer and the calibrated mixture are clearly better than Stage 1 only. The two Stage-2 signals reach similar average fidelity, but the calibrated mixture wins more cells than the best single layer (84 against 72 of 144). Introducing Stage 2 therefore improves pruning at the same cost, and the best single layer can serve as an approximate substitute for the calibrated mixture.

We hope these revisions address your concerns, and we are happy to answer any further questions.

Reviewer_4cGEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Query-Calibrated Pruning (Q-Cal), a two-stage visual token pruning method for multi-round VLM inference. A diagnostic framework backs its design: early text-visual attention is found to be largely prompt-invariant, while query-specific attention only forms in middle-to-late LM layers. Q-Cal therefore keeps a reusable visual prefix selected by early priors, then prunes it per question with a calibrated mixture of deep attention layers. It shows advantages compared to other baseline methods on six Qwen-family VLMs, though questions remain regarding how general the conclusions are, and several formatting and data errors were also found in the manuscript.

Critical strengths
  1. The method shows better performance compared to the baseline methods.
  2. The method shows extra insights regarding how to setup visual token pruning while preserving model performance.
  3. The visual token pruning task is practical and may be applied to real-world production cases.
Critical weaknesses
  1. It is not clear whether the proposed method generalizes to newer models, for example the Qwen3.6/3.8 families and the Gemma 4 family. Mainstream VLMs are moving towards hybrid or sparse attention, which already performs a form of implicit pruning.
  2. Figures and tables are not well made and contain very small text that is hard to read. The main table is too crowded and overwhelming; it could also be pruned to be clearer.
  3. Some entries in Table 2 appear to be copy-pasted. For example, SparseVILA and FasterVLM have the same GQA, TallyQA and GQA-G performance on Qwen3-8B, and SparseVILA also has the same POPE and TallyQA performance as FasterVLM on Qwen2-7B. Many other copy-paste errors are spotted in Table 2, weakening its credibility.
Questions for authors
  1. Can you evaluate the performance of Q-Cal on recent non-Qwen VLMs?
  2. Please inspect Table 2, as it seems to contain many errors and self-contradictions.
  3. The proposed method consistently surpasses the full-token baseline on InViG. Please verify whether this is caused by a code bug, or explain what the cause would be.
1 comment
AuthorsOpenReview ↗

Response to Reviewer 4cGE: three new backbones (incl. non-Qwen and hybrid attention), revised figures, and audited Table 2

W1 & Q1: Newer Qwen models, non-Qwen VLMs and hybrid attention

Thank you for the suggestion. We added the requested experiments on Qwen3.6-27B, Qwen3.8-27B and Gemma-4-31B (Section 4.1, p. 7, l. 340), with all eleven baselines. The Q-Cal results are in the condensed main table, Table 2 (p. 8, l. 378); the complete comparison is in Table 9 (p. 19, l. 1005) and the efficiency measurements in Tables 10–13 (p. 23, l. 1188–p. 26, l. 1350). Table R4 summarizes them.

Table R4: New backbones (5% / 10%; serving at B=16B=16 vs. Full).

BackboneQ-Cal Avg. Fid.Strongest baselineThroughputPrefill speedup
Qwen3.6-27B97.5 / 97.7PACT 95.1 / 97.21.21 / 1.20$\times$1.89 / 1.80$\times$
Qwen3.8-27B96.9 / 97.7PACT 92.7 / 95.61.17 / 1.14$\times$1.80 / 1.57$\times$
Gemma-4-31B94.7 / 94.9PACT 84.6 / 91.11.03 / 1.03$\times$1.46 / 1.42$\times$

Q-Cal remains strong on these backbones: its average fidelity is about 95% or higher at both budgets (94.7–97.7%), above the strongest baseline. It also accelerates inference. The prefill speedup is even larger than on the six Qwen-VL backbones: 1.4–1.9$\times$ against about 1.35$\times$ at B=16B=16. Throughput rises by up to 1.2$\times$, and on Gemma-4 at B=64B=64 the pruned model fits on two GPUs where the full-token model runs out of memory. The acceleration of throughput and decoding is lower than on the six Qwen-VL backbones (1.4$\times$ at B=16B=16), because these models use hybrid and sparse attention: Qwen3.6/3.8 interleave linear-attention layers whose recurrent state has a fixed size, and Gemma-4 uses sliding-window attention, so the cost of a decoding step hardly depends on the number of visual tokens (1.01–1.04$\times$).

W2: Readability of figures and the main table

Thank you for the comments on our presentation. We remade the tables and figures of the paper: all figures now use larger and uniform printed font sizes, and the main table, Table 2 (p. 8, l. 378), now compares Q-Cal with the four strongest baselines per backbone, while the complete matrix with all eleven baselines is given in Table 9 (p. 19, l. 1005).

W3 & Q2: Repeated entries and errors in Table 2

Thank you for your careful check of Table 2, and we apologize for these errors. We reran the entire main results table (Table 2, p. 8, l. 378; complete matrix in Table 9, p. 19, l. 1005).

Identical SparseVILA and FasterVLM entries. Their close results come from the two methods' design. According to the papers and the official code, the two methods both score visual tokens by the attention they receive in the last ViT layer ([CLS] attention when the encoder has a [CLS] token; for encoders without one, such as the Qwen and Gemma encoders used here, the mean attention each visual token receives). At equal retention they therefore keep the same prefill tokens. The only difference is SparseVILA's question-aware retrieval during decoding, which happens after the first token; their first generated tokens are thus almost always identical, and on the short-answer questions of these benchmarks their results are very close (identical outputs on 99.72% of the questions; at most 0.09 points per cell). Both were run separately (Appendix G, p. 21, l. 1116).

Avg. Fid. not matching the cells, and other inconsistencies. We found a bug in our code: when the prefix KV cache was reused across the questions of an image, the attention mask was misaligned, which affected the baselines to different degrees. InViG was affected the most, and Q-Cal, run with a separate script, was affected differently. The mismatch between Avg. Fid. and the cells of Qwen3-VL-8B FasterVLM came from this bug in its 10% runs, together with a misplacement of two values between the budget columns when the table was assembled. We fixed the KV-cache reuse, reran all results, and recomputed every Avg. Fid. from the cells (Tables 2 and 9).

Q3: Why Q-Cal exceeded Full on InViG

Thank you for raising this, and we apologize for the error. It was caused by the misaligned attention mask in the KV-cache reuse described above, which affected InViG the most and suppressed the scores of Full and of the baselines. After the fix and the rerun, Q-Cal no longer exceeds Full on any backbone (Table R5); the corrected values are in Table 2 (p. 8, l. 378) and Table 9 (p. 19, l. 1005).

Table R5: InViG accuracy after the fix (Q-Cal: 5% / 10%).

BackboneFullQ-CalBackboneFullQ-Cal
Qwen2-VL-2B0.0990.092 / 0.092Qwen3-VL-4B0.0930.083 / 0.085
Qwen2-VL-7B0.1480.137 / 0.137Qwen3-VL-8B0.1180.111 / 0.113
Qwen2.5-VL-3B0.1150.112 / 0.115Qwen3.6-27B0.1590.158 / 0.159
Qwen2.5-VL-7B0.0890.086 / 0.088Qwen3.8-27B0.1450.139 / 0.143
Gemma-4-31B0.1430.124 / 0.124

We hope these revisions address your concerns, and we are happy to answer any further questions.

Reviewer_UgTKOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether visual-token importance scores actually respond to the question. It finds that early text-visual attention changes little across questions about the same image, while deeper attention is more question-dependent. It also compares the kinds of evidence preserved by early attention and visual priors.

Critical strengths
  1. The paper shows that feeding a question into a pruning score does not necessarily make the selected tokens question-specific. Testing different questions on the same image makes this visible, rather than relying only on average answer accuracy (Table 1).

  2. Q-Cal gives the two pruning stages different jobs: keep broad image coverage first, then select evidence for the current question using deeper attention. This is a sensible design for asking several questions about one image, and it preserves strong accuracy at low final token budgets (Table 2).

Critical weaknesses
  1. The query-awareness test measures whether selections change, but not whether they keep the right evidence. Different questions can need the same region, while a small change in selected tokens can be enough to answer a question about a new detail. Table 1 reports overlap and rank correlation without measuring evidence coverage for each question. The visualizations and the separate 240-task study are useful, but they do not show how often the early selector actually misses the evidence needed by a changed question. This limits the stronger conclusion that early attention is not useful for query-specific selection.

  2. The shared prefix can discard the very detail that a later question needs. Stage 1 keeps only 33% of tokens before the per-question round, and Stage 2 can select only from that set (Sections 3.4-3.5). If a small text region or object is removed, deeper attention cannot bring those tokens back. The budget sweep supports 33% on average, but does not show how often individual questions fail because their evidence was removed at Stage 1. This is an important limit for a prefix intended to serve several different questions.

Questions for authors
  1. For question pairs that need different image regions, how often do early and deep attention retain the correct region for each question? Can you separate shared useful tokens from tokens that cause the early selector to miss the new evidence?

  2. How many answer failures come from Stage 1 removing needed evidence, rather than Stage 2 choosing poorly? Does the shared 33% prefix retain evidence for all questions about an image, especially small-text and small-object questions?

  3. Please also clarify the fidelity calculation in Table 2. For Qwen3-VL-8B FasterVLM, all eight listed scores improve from the 5% to the 10% setting, but average fidelity falls from 96.3% to 91.4%. Which entries need correction?

1 comment
AuthorsOpenReview ↗

Response to Reviewer UgTK: evidence retention of the early selector, stage-wise error analysis, and corrected Table 2 cells

W1 & Q1: Does the early selector retain the evidence each question needs?

Thank you for pointing out this limitation. We redesigned the diagnostic on about 1,800 GQA questions over 710 images whose evidence masks are mutually independent (pairwise IoU <0.3<0.3). Besides JkJ_k and ρ\rho, which show the same trend and conclusion as before, we measure at 10% retention the Touch and Cov. of each question's evidence, and of the other questions' evidence when the tokens selected for the original question are used (mis-aligned), summarized in Table R1. Full results: Table 1 (p. 4, l. 162) and Table 6 (p. 15, l. 797).

Table R1: Touch / Cov. at 10% (mean of six backbones).

SignalOwn questionOther question
Early attention (L0)86.3 / 12.186.2 / 11.9
ViT received attention95.8 / 16.095.9 / 15.9
Deep attention (75%)93.9 / 25.474.2 / 8.3

Early attention misses the evidence entirely for 13.7–14.0% of the questions, deep attention for 6.1%, and deep attention covers the most evidence (25.4% vs. 12.0–17.6%). Early attention and the visual priors give a fixed coverage of the image: they hit another question's evidence as often as their own. Deep attention tracks the question (93.9% vs. 74.2% Touch); Figure 1 (p. 4, l. 195) visualizes this. Its Touch is slightly below the visual priors only for small evidence (84.1% vs. 88.1%), and in 72% of these misses its nearest selected token is adjacent to the object.

W2 & Q2: Evidence removed by the shared 33% prefix

Thank you for pointing out this missing analysis. We added two experiments on object (GQA, COCO masks) and text (TextVQA, OCR boxes) evidence.

Budget sweep. Touch and Cov. of each question's evidence in the shared prefix are shown in Figure 5b (p. 9, l. 454) (objects) and Figure 10 (p. 28, l. 1481) (text). They grow with the budget, and 33% is cost-effective: it touches 94.0% of the small-object and 95.2% of the small-text evidence (5%: 69.6%, 65.8%; 50%: +2 points at most).

Error analysis (Table 3, p. 9, l. 432, objects; Table 15, p. 28, l. 1488, text). Among trials the full model answers correctly, a Q-Cal failure at 5% or 10% is attributed to Stage 1 if answering from the same prefix without Stage 2 is also wrong, else to Stage 2; Table R2 summarizes the failures by stage.

Table R2: Failures per stage, percentage of trials (Stage 1 / Stage 2).

Evidence sizeObjectsText
Small4.5 / 0.94.7 / 3.2
Medium4.4 / 1.46.1 / 7.0
Large4.2 / 1.56.7 / 13.2

With the 33% prefix, Stage-1 failures occur in only about 5% of the trials at every scale, for both objects and text. So the prefix keeps sufficient evidence at all scales. Stage-2 errors are fewer for all objects and small text; for medium and large text, which need many contiguous text tokens, they are comparable or more frequent, because the final 5–10% budget covers only 14–26% of the text area.

Q3: Average fidelity of Qwen3-VL-8B FasterVLM in Table 2

Thank you for catching this, and we apologize for this oversight. Two errors affected this row: (i) the 10% VQAv2 and VG-QA runs were hit by a bug in the multi-question prefix KV-cache path (scores 0.564, 0.354); (ii) these two values were placed in the 5% column, while Avg. Fid. used the correct assignment. Hence all cells seemed to improve while the fidelity dropped. We reran all cells of Table 2 and the complete Table 9 (p. 19, l. 1005) and recomputed every Avg. Fid.; Table R3 shows only the cells of this row discussed here (VQAv2, VG-QA and its Avg. Fid.).

Table R3: Qwen3-VL-8B FasterVLM, affected cells of Table 2 (5% / 10%).

EntryAs printedTrue columnsRevised
VQAv20.564 / 0.7220.722 / 0.5640.723 / 0.728
VG-QA0.354 / 0.4730.473 / 0.3540.469 / 0.475
Avg. Fid.96.3 / 91.496.2 / 91.595.9 / 96.9

We hope these revisions address your concerns, and we are happy to answer any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions for the Area Chair

Dear Area Chair,

We thank you and the reviewers for the careful reviews. Their valuable and constructive suggestions have helped us improve the paper, and all of them have been addressed in the revised manuscript. We summarize the main changes below; details are in our individual responses.

1. Implementation error in Table 2 (R2, R3; also R1-Q3). The anomalies noted by the reviewers (InViG scores above Full and inconsistent Avg. Fid.) were caused by a bug in the prefix KV-cache reuse, which misaligned the attention mask. We sincerely apologize for this oversight. We fixed the bug and reran every cell of Tables 2 and 9; the main conclusions are unchanged.

2. Generality beyond the original backbones (R2). We added Qwen3.6-27B, Qwen3.8-27B and Gemma-4-31B (including non-Qwen and hybrid attention) with all eleven baselines. Q-Cal reaches 94.7–97.7% average fidelity, above the strongest baseline.

3. Evidence retention and the 33% Stage-1 budget (R1). New evidence-retention measurements show that deep text-visual attention tracks each question's own evidence. The shared 33% Stage-1 prefix touches about 95% of the evidence for small objects and small text, and Stage-1 failures occur in only about 5% of trials at every evidence size, confirming that this budget is sufficient.

4. Measured efficiency and matched-cost ablations (R3). Appendix H reports measured latency, throughput and memory for all methods on nine backbones; at B=64, Q-Cal decodes 3.4× faster and reaches 2.2× the throughput of the full-token model. Appendix I adds the requested matched-cost comparison with Stage 1 only and the best single layer.

5. Presentation (R2, R3). All figures and tables were remade with larger, uniform fonts, and the short lines were fixed.

We are grateful for the reviewers' feedback and happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.