acceptodds
Under review as a conference paper at ICLR 2027

QUEST: Query-guided Evidence Selection for Visual Token reduction

Abstract

Long visual token sequences increase the inference cost of the large language model (LLM) in vision-language models (VLMs). Effective visual token reduction requires preserving the visual information needed to answer the textual query. An external vision-language alignment model (VLAM) can provide relevance between image regions and the query. However, directly mapping this relevance to the visual tokens of the target VLM is difficult because the VLAM and the target VLM may use different vision encoders and visual representations. To address this issue, we propose QUEST, a training-free Pre-LLM visual token selection method. QUEST uses the relevance from an external VLAM to identify query-relevant Evidence Seed units and uses the corresponding vision encoder tokens as Evidence Seeds. It constructs Evidence Attention from the attention between the Evidence Seeds and other visual tokens at an intermediate layer of the target VLM's vision encoder. QUEST also selects an Evidence Source Layer based on layer-wise attention distance. Evidence Attention is combined with Visual Guidance, followed by diversity-aware selection to choose the final visual tokens under a given token budget. We evaluate QUEST on LLaVA-NeXT-7B and Qwen2.5-VL-7B across multiple VLAMs, token budgets, and nine benchmarks. The results show high average performance retention relative to the full-token VLMs across the evaluated token budgets.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Vision-Language Models, Visual Token Selection, Query-Guided Token Selection, Vision-Language Alignment, Pre-LLM Token Selection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FwRoOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

QUEST studies how query relevance from an external vision-language alignment model can guide visual token selection before the language model. It first identifies query-relevant seed regions, then uses attention from an intermediate vision-encoder layer to assign evidence scores to other visual tokens. These scores are fused with visual saliency and supplied to the diversity-aware selection procedure adopted from SFPruner. The method requires no model training and chooses the attention source layer using an image-based statistic. Experiments on LLaVA-NeXT and Qwen2.5-VL compare alignment models and token budgets, evaluate accuracy and computational cost, and analyze the components used to construct the evidence signal.

Critical strengths
  1. The paper identifies a concrete problem in transferring external query relevance into a target VLM: spatial correspondence alone need not capture contextualized visual information. Using externally identified seeds to obtain an attention-based evidence signal is a reasonable way to combine alignment with the target representation. Qwen2.5-VL provides a relevant test case because it lacks a paired CLIP text encoder.

  2. The source-layer analysis and architecture-specific implementation details are useful. The layer is selected from images without question annotations, and the tested intermediate layers outperform the final layer on average. The experiments also disclose modest gains and additional runtime/memory costs. On Qwen, the default method improves average retention over SFPruner by 0.43 and 0.41 percentage points at 30% and 20% retention, respectively.

Critical weaknesses
  1. The evidence does not isolate the paper's main new mechanism. SG-RLS and directional masking are inherited, while the ablations vary how evidence attention is computed rather than comparing it with direct relevance, visual-only guidance, or non-semantic seed controls under an otherwise identical selection pipeline. Moreover, Appendix C.1 states that the LLaVA implementation computes a global attention map in the VLAM and interpolates it to the high-resolution tiles, instead of extracting attention separately from those target-VLM tile representations. This is an important qualification of the claimed target-representation adaptation. Given the small main-table margins, controlled mechanism ablations are necessary to establish that query-guided evidence expansion is responsible for the improvement.

  2. The accuracy-cost comparison is not yet sufficiently controlled. LLaVA baseline results in Table 1 are taken from prior reports, while QUEST fixes a 2x2 tile layout and evaluates its own pipeline. On Qwen, where the baselines are reproduced, the gain at 10% retention is only 0.02 points over CDPruner. At 20% retention, Table 6 reports 183.7 ms prefilling for QUEST versus 153.5 ms for SFPruner, and 17.0 versus 16.2 GB memory. That table labels the baselines' VLAM as absent, although Sec. 4.2 states that relevance-based baselines use the same external CLIP. The benefit at matched total resources is therefore unclear, especially for such small accuracy differences.

Questions for authors
  1. With the same VLAM, preprocessing, visual guidance, and SG-RLS/DM implementation, what is the incremental effect of evidence attention over direct interpolated relevance, visual-only guidance, and random or visual-saliency seed selection? Please report paired differences on the same evaluation examples.

  2. For LLaVA, how are the 576 base-tile tokens handled when the appendix defines guidance for only the 2304 spatial-tile tokens? Does extracting evidence attention from the actual target-VLM tiles change the result, and were the reported baselines evaluated with the same fixed layout?

  3. Were external VLAM inference and attention extraction charged consistently to every method in Table 6? Please also clarify the common cache/backend, batch-size, and decoding protocol underlying the 197.1 ms/token full-model value versus 24.0 ms/token after pruning. These details could materially change the efficiency interpretation.

8 comments
AuthorsOpenReview ↗

Response Plan and Clarified Questions

We sincerely thank Reviewer FwRo for the detailed and constructive feedback. To facilitate our responses, we have organized the key questions below, using W1–W2 to refer to the corresponding weaknesses. We are currently conducting additional experiments and will update our responses to each point as soon as possible.

  1. (W1) With the same VLAM, preprocessing, visual guidance, and SG-RLS/DM implementation, what is the incremental effect of evidence attention over direct interpolated relevance, visual-only guidance, and random or visual-saliency seed selection? Please report paired differences on the same evaluation examples.
  2. (W1, W2) For LLaVA, how are the 576 base-tile tokens handled when the appendix defines guidance for only the 2304 spatial-tile tokens? Does extracting evidence attention from the actual target-VLM tiles change the result, and were the reported baselines evaluated with the same fixed layout?
  3. (W2) Were external VLAM inference and attention extraction charged consistently to every method in Table 6? Please also clarify the common cache/backend, batch-size, and decoding protocol underlying the 197.1 ms/token full-model value versus 24.0 ms/token after pruning. These details could materially change the efficiency interpretation.
AuthorsOpenReview ↗

Response to Q1

We thank the reviewer for the detailed and constructive review. Common settings are given in the general response.

Q1. (W1) With the same VLAM, preprocessing, visual guidance, and SG-RLS/DM implementation, what is the incremental effect of evidence attention over direct interpolated relevance, visual-only guidance, and random or visual-saliency seed selection? Please report paired differences on the same evaluation examples.

The increment over random and visual-saliency seeds and over visual-only guidance is significant. The increment over direct interpolated relevance is not significant on the paper benchmarks, and it is significant on the document and OCR benchmarks.

We fixed the VLAM, preprocessing, Visual Guidance, and SG-RLS/DM, and changed only the Evidence Attention part or the seeds (Qwen2.5-VL-7B).

Table R3-1. Paired differences on the paper benchmarks at 20%: average retention difference (pp, 95% CI), QUEST minus variant.

Compared variantDifference
Direct interpolated relevance+0.17 [-0.53, +0.88]
Visual-only guidance+1.18 [+0.40, +1.95]
Random seeds+2.12 [+1.32, +2.92]
Visual-saliency seeds+1.78 [+1.05, +2.53]

At 10% and 30%, the difference from direct interpolated relevance is +0.43 [-0.42, +1.28] and +0.09 [-0.53, +0.73].

Table R3-2. Held-out benchmarks, QUEST minus direct interpolated relevance (score difference, 95% CI).

BudgetDocVQA (ANLS)OCRBench
10%+2.59 [+1.57, +3.58]+3.60 [+1.10, +6.00]
20%+1.79 [+0.99, +2.59]+3.50 [+1.40, +5.70]
30%+1.52 [+0.87, +2.23]+3.20 [+1.20, +5.20]

Over the 18 held-out benchmark-budget cells, QUEST is significantly higher than direct interpolated relevance in 6 cells and lower in 1 (MME-RW-Lite, 30%), and higher than visual-only guidance in 7 cells with no lower cell. No seed control is significantly higher than QUEST in any cell. We also repeated the random seeds with five seeds, and all five are significantly lower than QUEST (94.25 ± 0.15% vs. 96.16%, six paper benchmarks without HBench).

AuthorsOpenReview ↗

Response to Q2

Q2. (W1, W2) For LLaVA, how are the 576 base-tile tokens handled when the appendix defines guidance for only the 2304 spatial-tile tokens? Does extracting evidence attention from the actual target-VLM tiles change the result, and were the reported baselines evaluated with the same fixed layout?

The base tile is scored with its own attention and receives 1/5 of the budget. We did not run a per-tile variant on LLaVA-NeXT, and all baselines are now re-run with the same fixed layout.

We apologize that the appendix omitted the handling of the base tile.

Base tile. The 576 tokens of the base tile are scored with the base tile's own CLIP attention and Evidence Seeds, and the interpolated global maps are used only for the 2304 spatial-tile tokens. Of the token budget B = 2880 × r, the base tile receives B/5 (64 for 320 tokens and 128 for 640), which is one tile's share in LLaVA AnyRes, and the spatial tiles receive the rest. Ranking and the diversity computation are done separately in the two groups. The revision adds this to Appendix C.1.

Actual tiles. We did not run a per-tile variant on LLaVA-NeXT. Since the VLAM and the target VLM share the same CLIP encoder in LLaVA-NeXT, we compute the maps once on the whole image (Appendix C.1), and the revision states this clearly. InternVL3.5-4B, which we added, processes each tile separately in InternViT, so its Evidence Attention is extracted from the actual tiles. On InternVL, QUEST is the highest at 20% and 30% on the paper benchmarks, and comparable to or lower than the baselines at 10% (average retention: 89.66% vs. 89.01% and 89.19% at 20%, 92.63% vs. 91.62% and 91.87% at 30%, with source layer 21).

Baseline layout. The baselines in the original Table 1 were reported numbers. We re-ran all baselines with the same fixed layout, token count, budget split, and evaluation as QUEST (Table R3-3). The fixed layout does not significantly change the full model (-0.11 pp [-0.46, +0.23] against the native AnyRes layout). QUEST is comparable to SFPruner and CDPruner and significantly higher than DivPrune and VisionZip at both budgets.

Table R3-3. LLaVA-NeXT-7B, average retention difference (pp, 95% CI), QUEST minus baseline.

TokensSFPrunerCDPrunerDivPruneVisionZip
640-0.32 [-0.69, +0.04]-0.13 [-0.60, +0.34]+1.46 [+0.85, +2.09]+1.36 [+0.63, +2.09]
320-0.31 [-0.83, +0.20]+0.29 [-0.37, +0.94]+2.27 [+1.56, +3.00]+3.01 [+2.12, +3.90]
Reviewer_5VEQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes QUEST, a training-free token pruning method that uses relevance scores from an external vision–language alignment model to identify query-relevant evidence units and selects the corresponding visual encoder tokens as evidence seeds. The method achieves promising results on LLaVA-NeXT-7B and Qwen2.5-VL-7B.

Critical strengths
  1. Rather than directly treating external relevance scores as final token importance, QUEST identifies evidence seeds and expands them through attention relationships among visual tokens. This provides a training-free approach to transferring relevance across different encoders, resolutions, and spatial merging mechanisms.

  2. The experiments are reasonably comprehensive, including analyses of the evidence source layer, seed count, and guidance fusion weight.

Critical weaknesses
  1. The gains over strong baselines are modest and come at higher computational cost, leaving insufficient evidence of a competitive performance–efficiency trade-off. In Table 1, QUEST matches SFPruner at 11.1% token retention and improves average performance retention by only 0.15 percentage points at 22.2% retention.

  2. The current ablations primarily examine source layers, seed counts, VLAM choices, aggregation strategies, and a narrow range of fusion weights. However, they do not directly compare key variants: using interpolated VLAM relevance directly, using Visual Guidance alone, expanding randomly selected seeds, and the full QUEST method.

Questions for authors

Please refer to weaknesses.

6 comments
AuthorsOpenReview ↗

Response Plan and Clarified Questions

We sincerely thank Reviewer 5VEQ for the constructive feedback. To facilitate our responses, we have summarized the two key concerns as questions below, labeled Q1–Q2 to correspond to W1-W2. We are currently conducting additional experiments and will update our responses to each point as soon as possible.

  1. (W1) Does QUEST provide a competitive performance–efficiency trade-off compared to strong baselines, given its modest performance gains and additional computational cost?
  2. (W2) How does the full QUEST method compare with direct VLAM relevance, Visual Guidance alone, and random seed expansion under the same experimental setup?
AuthorsOpenReview ↗

Response to Q1

We thank the reviewer for the constructive review. Common settings are given in the general response.

Q1. (W1) Does QUEST provide a competitive performance-efficiency trade-off compared to strong baselines, given its modest performance gains and additional computational cost?

The cost of QUEST is practically the same as that of baselines using the same VLAM (+0.04 TFLOPs, +3.8 ms prefill, +0.05 GiB memory). In accuracy, QUEST is comparable to SFPruner and CDPruner under the same conditions, higher than DivPrune and VisionZip, and significantly higher than SFPruner on the held-out document and OCR benchmarks.

LLaVA-NeXT. The figures cited by the reviewer (a tie at 11.1% and +0.15 pp at 22.2% in Table 1) came from comparing QUEST with numbers reported by each baseline paper. We re-ran all methods with the same layout, token count, budget split, and evaluation. QUEST is comparable to SFPruner and CDPruner, and significantly higher than DivPrune and VisionZip at both budgets (Table R2-1). The revision replaces Table 1 with this comparison.

Table R2-1. LLaVA-NeXT-7B, average retention difference (pp, 95% CI), QUEST minus baseline.

TokensSFPrunerCDPrunerDivPruneVisionZip
640-0.32 [-0.69, +0.04]-0.13 [-0.60, +0.34]+1.46 [+0.85, +2.09]+1.36 [+0.63, +2.09]
320-0.31 [-0.83, +0.20]+0.29 [-0.37, +0.94]+2.27 [+1.56, +3.00]+3.01 [+2.12, +3.90]

Cost. The efficiency numbers in the original Table 6 had measurement errors, and we corrected them. We re-measured all methods on Qwen2.5-VL-7B with the same protocol (FP16, KV cache, batch size 1, 10 warmup and 100 measured requests, medians), including the CLIP cost for every method that uses a VLAM.

Table R2-2. Efficiency at 20% (median over 100 requests).

MethodFLOPs (T)Peak memory (GiB)Prefill (ms)Decode (ms/token)
Full model19.115.68196.226.7
SFPruner + CLIP-inv9.116.45184.926.4
QUEST9.116.50188.726.5

The extra cost of QUEST over SFPruner with the same VLAM is Evidence Attention only (0.04 TFLOPs, 3.8 ms). The 0.8 GiB over the full model comes from the CLIP weights.

Accuracy on Qwen2.5-VL. On the Table 2 benchmarks, QUEST is comparable to SFPruner, and paired CIs confirm that the differences are not significant (Table R2-3). On the held-out benchmarks, QUEST has the highest average retention at all budgets (Table R2-4), and it is significantly higher than SFPruner on DocVQA (+1.5 to +2.8) and OCRBench (+2.7 to +3.1) at all budgets. Since the total resources are the same for methods using the same VLAM, what remains is the accuracy difference, and it favors QUEST on document and OCR inputs.

Table R2-3. Paired CIs on the Table 2 benchmarks: average retention difference (pp), QUEST minus baseline.

Budgetvs. SFPrunervs. CDPruner
30%+0.43 [-0.19, +1.06]+1.03 [+0.29, +1.77]
20%+0.40 [-0.31, +1.09]+0.80 [-0.00, +1.58]
10%+0.35 [-0.51, +1.19]+0.04 [-0.90, +0.98]

Table R2-4. Average retention (%) on the six held-out benchmarks.

BudgetCDPrunerSFPrunerQUEST
30%86.5291.7691.97
20%81.2985.3686.29
10%72.8273.6375.30
AuthorsOpenReview ↗

Response to Q2

Q2. (W2) How does the full QUEST method compare with direct VLAM relevance, Visual Guidance alone, and random seed expansion under the same experimental setup?

QUEST is significantly higher than random seeds and Visual Guidance alone. Compared with direct VLAM relevance, it is similar on average on the paper benchmarks and significantly higher on DocVQA and OCRBench at all budgets.

We ran these variants in the same pipeline, changing only the Evidence Attention part or the seeds and keeping everything else fixed (Qwen2.5-VL-7B).

  • Direct VLAM relevance (Visual + Raw): the VLAM relevance map replaces Evidence Attention.
  • Visual Guidance alone (Visual only): no query is used.
  • Random seed expansion: the seeds are chosen at random.
  • We also added seeds selected by visual saliency without the query, and seeds from CLIP without the sign inversion.

Table R2-5. Average retention (%). In parentheses: QUEST minus variant (pp) with 95% CI.

VariantPaper benchmarks (20%)Held-out (10%)Held-out (20%)Held-out (30%)
QUEST95.6275.3086.2991.97
Direct VLAM relevance95.45 (+0.17 [-0.53, +0.88])73.6285.0991.65
Visual Guidance alone94.44 (+1.18 [+0.40, +1.95])71.4784.5891.32
Random seeds93.50 (+2.12 [+1.32, +2.92])71.0983.8990.50
Visual saliency seeds93.84 (+1.78 [+1.05, +2.53])71.4784.4891.00
CLIP without inversion91.94 (+3.68 [+2.85, +4.50])68.8582.4389.60

Table R2-6. QUEST minus direct VLAM relevance (score difference, 95% CI).

BudgetDocVQA (ANLS)OCRBench
10%+2.59 [+1.57, +3.58]+3.60 [+1.10, +6.00]
20%+1.79 [+0.99, +2.59]+3.50 [+1.40, +5.70]
30%+1.52 [+0.87, +2.23]+3.20 [+1.20, +5.20]

On the held-out benchmarks, none of the seed controls is significantly higher than QUEST in any benchmark-budget cell. We also repeated the random seeds with five seeds on the paper benchmarks without HBench, and all five are significantly lower than QUEST (94.25 ± 0.15% vs. 96.16%).

Reviewer_eeKrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Training-free, pre-LLM, query-aware visual token selection for VLMs whose vision encoder isn't text-aligned (e.g. Qwen2.5-VL). An external vision-language model (CLIP by default) marks a few query-relevant locations ("seeds"). The target encoder's own attention from those seeds gives the relevance score, mixed with standard CLS/mean attention. Final selection reuses SFPruner's procedure. Evaluated on LLaVA-NeXT-7B and Qwen2.5-VL-7B, with best or tied-best average retention.

Critical strengths

1. Sensible idea for a real gap.

  • Most query-aware pruners assume a text-aligned (CLIP-style) encoder, which doesn't hold for many deployed VLMs.
  • The external model only needs to be roughly right at a few points; the target encoder does the rest. Training-free.

2. Fair setup, honest reporting.

  • Baselines re-run under one protocol, with the same relevance source; includes a Random baseline.
  • Efficiency numbers include the external-model cost, even though that doesn't flatter the method.
Critical weaknesses

1. The new component is not isolated, and the gains are within noise.

  • Relevance source, visual guidance and the whole selection stage come from prior work; only seed-and-expand + the layer rule are new.
  • Missing ablation: same pipeline, using the external relevance map directly instead of the seed-expanded attention (plus no-query / query-only extremes).
  • Margin over the closest method (SFPruner) is a few tenths of a point, single runs, with hyperparameters tuned on the evaluation benchmarks.

2. Query signal is fragile; the core setting is under-tested.

  • Negated CLIP similarity is adopted without analysis; the best external model differs across backbones, and some needed modifications to work.
  • The non-aligned-encoder case, which motivates the paper, is tested on a single model (on LLaVA-NeXT the external model is the encoder).

3. Efficiency story is weak and partly inconsistent.

  • Small prefill saving vs. the closest baseline, and peak memory above the unpruned model.
  • Some cost numbers (decode speedup, FLOPs scaling) are hard to reconcile; the LLaVA-NeXT protocol differs from the baselines'.
Questions for authors
  1. Ablation replacing the seed-expanded attention with the raw relevance map (and the no-query / query-only extremes), everything else fixed?
  2. Random / non-negated / query-agnostic seeds: how much does the query actually matter?
  3. Results on another non-CLIP backbone (e.g. InternVL)?
  4. Breakdown of the external-model cost, and an explanation of the decode / FLOPs numbers?
  5. Were the full model and baselines on LLaVA-NeXT run under the same image layout as QUEST?
10 comments
AuthorsOpenReview ↗

Response Plan and Clarified Questions

We sincerely thank Reviewer eeKr for the detailed and constructive feedback. To facilitate our responses, we have organized the key questions below, using W1-W3 to refer to the corresponding weaknesses. We are currently conducting additional experiments and will update our responses to each point as soon as possible.

  1. (W1) Ablation replacing the seed-expanded attention with the raw relevance map (and the no-query / query-only extremes), everything else fixed?
  2. (W1, W2) Random / non-negated / query-agnostic seeds: how much does the query actually matter?
  3. (W2) Results on another non-CLIP backbone (e.g. InternVL)?
  4. (W3) Breakdown of the external-model cost, and an explanation of the decode / FLOPs numbers?
  5. (W3) Were the full model and baselines on LLaVA-NeXT run under the same image layout as QUEST?
AuthorsOpenReview ↗

Response to Q1

We thank the reviewer for the detailed and constructive review. Common settings are given in the general response.

Q1. (W1) Ablation replacing the seed-expanded attention with the raw relevance map (and the no-query / query-only extremes), everything else fixed?

Yes. QUEST is comparable to Visual + Raw on the paper benchmarks and significantly higher on DocVQA and OCRBench at all budgets. Both extremes are significantly lower than QUEST.

We changed only the Evidence Attention part and kept everything else fixed.

  • Visual + Raw: the VLAM relevance map replaces Evidence Attention.
  • Visual only (no-query extreme): no query is used.
  • Raw only (query-only extreme): only the VLAM relevance map is used, without Visual Guidance.

On the paper benchmarks, the average retention of QUEST and Visual + Raw is similar, and Visual + Raw is significantly higher on MME.

Table R1-1. Average retention (%). In parentheses: QUEST minus variant (pp) with 95% CI.

VariantPaper benchmarks (20%)Held-out (10%)Held-out (20%)Held-out (30%)
QUEST95.6275.3086.2991.97
Visual + Raw95.45 (+0.17 [-0.53, +0.88])73.6285.0991.65
Visual only94.44 (+1.18 [+0.40, +1.95])71.4784.5891.32
Raw only93.99 (+1.64 [+0.79, +2.46])---

Table R1-2. QUEST minus Visual + Raw (score difference, 95% CI).

BudgetDocVQA (ANLS)OCRBench
10%+2.59 [+1.57, +3.58]+3.60 [+1.10, +6.00]
20%+1.79 [+0.99, +2.59]+3.50 [+1.40, +5.70]
30%+1.52 [+0.87, +2.23]+3.20 [+1.20, +5.20]

Over the 18 held-out benchmark-budget cells, QUEST is significantly higher than Visual + Raw in 6 cells and lower in 1 (MME-RW-Lite, 30%), and higher than Visual only in 7 cells with no lower cell. Raw only is 1.64 pp lower than QUEST on average (significantly lower on AI2D, TextVQA, and MMStar, and significantly higher on POPE).

AuthorsOpenReview ↗

Other points in W1

Other points in W1

Scope of the contribution. We agree with the reviewer's description. The relevance source (CLIP-inv), Visual Guidance, and the selection stage (SG-RLS and DM) come from prior work. Our contribution is how the VLAM relevance is transferred to the target VLM. The relevance is used only to pick seeds, and the seeds are expanded with the attention of the target VLM's vision encoder (Evidence Attention), together with the choice of its source layer. Since the VLAM and the target VLM use different encoders and representations, this transfer is the main problem we address (Sec. 1). We kept the other components fixed to isolate this part, and the controlled results are given in Q1 and Q2.

Margins and tuning. On the paper benchmarks, the differences from SFPruner are not significant. On held-out benchmarks with all hyperparameters fixed, QUEST has the highest average at all budgets.

We agree that the paper benchmarks alone cannot rule out noise or tuning effects. Paired CIs on the Table 2 benchmarks confirm that the differences from SFPruner are not significant (Table R1-3). We therefore evaluated the six held-out benchmarks with all hyperparameters fixed (Table R1-4). QUEST has the highest average retention at all budgets, and the differences are significant on DocVQA (+1.5 to +2.8 over SFPruner) and OCRBench (+2.7 to +3.1). Over the 18 cells, QUEST is significantly higher than SFPruner in 6 cells and lower in 1, and higher than CDPruner in 7 cells with no lower cell. The main point of the paper is the method of using VLAM relevance as seeds rather than a win in every setting, and we report the conditions in which the gain appears.

Table R1-3. Paired CIs on the Table 2 benchmarks: average retention difference (pp), QUEST minus baseline.

Budgetvs. SFPrunervs. CDPruner
30%+0.43 [-0.19, +1.06]+1.03 [+0.29, +1.77]
20%+0.40 [-0.31, +1.09]+0.80 [-0.00, +1.58]
10%+0.35 [-0.51, +1.19]+0.04 [-0.90, +0.98]

For this analysis, the full model and CDPruner were re-run in the same environment, so the CDPruner scores differ slightly from Table 2.

Table R1-4. Average retention (%) on the six held-out benchmarks.

BudgetCDPrunerSFPrunerQUEST
30%86.5291.7691.97
20%81.2985.3686.29
10%72.8273.6375.30

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers

General response to all reviewers

We thank all reviewers for the detailed and constructive reviews. We apologize for the late response, as the experiments took some time with our limited GPU resources. To address the reviewers' questions, we ran additional experiments with the same code and settings as in the paper, without changing any hyperparameter.

Common settings

  • Held-out benchmarks: six benchmarks that are not in the paper (VizWiz, DocVQA, MMMU, InfographicVQA, OCRBench, MME-RealWorld-Lite).
  • Statistics: paired bootstrap confidence intervals on the same evaluation samples, with the full model resampled jointly (10,000 resamples for the paper benchmarks and 5,000 for the held-out benchmarks).
  • Backbone: all results are on Qwen2.5-VL-7B unless stated otherwise.
  • Notation: we write CLIP-inv for CLIP† in the paper (CLIP with the negated image-text similarity).

Revised manuscript

We have reflected all additional experiments and corrections from the rebuttal in the revised manuscript, and all changes are marked in blue.

  • Additional experiments: held-out benchmarks, paired CIs, controlled ablations in the same pipeline (including repeated random seeds), InternVL3.5-4B, and the controlled comparison on LLaVA-NeXT (Appendix D, Table 1).
  • Re-measurements and corrections: efficiency (Table 6), the LLaVA-NeXT part of Tables 3 and 4, Figure 3 (the selected layers are unchanged), and parts of the text and notation (Sec. 3.2, Sec. 4, Appendix C).

We answer each reviewer's questions in separate comments below.

AuthorsOpenReview ↗

Summary of the discussion and additional results during the rebuttal period

We sincerely thank all reviewers and the AC for their time and the careful discussion. The reviewers' questions pushed us to make the evaluation fairer and more reliable, and we believe the paper is better for it. Below, we briefly summarize what we did and clarify two points.

What we did during the discussion

  • Fair comparison: In our revised comparisons, all methods use the same layout, token budget, budget split, and evaluation. Baselines whose original numbers came from different settings were re-run in our environment. SFPruner has no public code. Therefore, we implemented it ourselves from the paper and gave it the same VLAM as QUEST, since we wanted to compare against the strongest version we could build.
  • Reliability: Like most prior token pruning studies, including our baselines, our submission reported results only on the evaluation benchmarks. We followed this practice for comparability. However, a common practice is not a justification in itself, and we are grateful to the reviewers for pointing this out. Therefore, we added six held-out benchmarks with all hyperparameters fixed, and we report 95% paired-bootstrap CIs on the same samples for all main comparisons. We hope that this extended evaluation protocol can also serve as a useful reference for the field.
  • Another backbone: We added InternVL3.5-4B, whose vision encoder is not CLIP-based, with all baselines run in the same environment. We believe that these results on a non-CLIP backbone can also support future studies on visual token pruning.
  • We also added controlled ablations of Evidence Attention and corrected the efficiency measurement.

Clarification 1: performance against the strongest baseline (SFPruner)

While preparing this summary, we checked our SFPruner implementation against the paper once more. For backbones without a paired text encoder, the SFPruner paper uses only visual saliency. In our responses, the SFPruner results on Qwen2.5-VL and InternVL3.5-4B include the CLIP-inv relevance that we added. To be transparent, we now report both settings. In the table, "higher" and "lower" mean that the 95% paired-bootstrap CI excludes zero.

BackboneBenchmarksQUEST vs. SFPruner (original)QUEST vs. SFPruner + CLIP-inv
Qwen2.5-VL-7BPaper (7)Higher at all budgets (+1.15 to +3.94 pp)Not significant (+0.35 to +0.43 pp)
Held-out (6)Higher in 7 of 18 cells, no lower cellHigher in 6 cells, lower in 1 cell
InternVL3.5-4B (layer 21)Paper (7)Higher at all budgets (+1.10 to +2.46 pp)Higher at 30%, not significant at 10% and 20%
Held-out (6)Higher in 6 of 18 cells, no lower cellHigher in 3 cells, lower in 4 cells
LLaVA-NeXT-7BPaper (6)-Not significant (-0.32 and -0.31 pp)

On LLaVA-NeXT, the SFPruner paper uses CLIP and does not mention a sign inversion of the similarity. Without the sign inversion, SFPruner is the lowest among all methods, which differs from the ranking reported in the SFPruner paper. With CLIP-inv, the ranking agrees with the paper, and we kept this implementation.

Clarification 2: where the gain comes from

  • VLAM relevance: Adding CLIP-inv relevance improves SFPruner on Qwen2.5-VL at all budgets.
  • Query-based Evidence Seeds: Replacing only the seeds with random, visual-saliency, or un-negated CLIP seeds lowers the average retention by 2.12, 1.78, and 3.68 pp (Qwen2.5-VL, 20%, all CIs exclude zero). No seed control is higher than QUEST in any held-out cell.
  • Evidence Attention against direct VLAM relevance: On the paper benchmarks, the difference is not significant on Qwen2.5-VL and LLaVA-NeXT, and we acknowledge this as a limitation. That said, on the held-out benchmarks, QUEST is higher on DocVQA and OCRBench at all budgets (Qwen2.5-VL). On InternVL3.5-4B (layer 21), it is higher at 30% on the paper benchmarks.

We will include every result from the rebuttal in the manuscript, and we will release the full source code to reproduce all results upon publication.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.