- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
QUEST studies how query relevance from an external vision-language alignment model can guide visual token selection before the language model. It first identifies query-relevant seed regions, then uses attention from an intermediate vision-encoder layer to assign evidence scores to other visual tokens. These scores are fused with visual saliency and supplied to the diversity-aware selection procedure adopted from SFPruner. The method requires no model training and chooses the attention source layer using an image-based statistic. Experiments on LLaVA-NeXT and Qwen2.5-VL compare alignment models and token budgets, evaluate accuracy and computational cost, and analyze the components used to construct the evidence signal.
Critical strengths
-
The paper identifies a concrete problem in transferring external query relevance into a target VLM: spatial correspondence alone need not capture contextualized visual information. Using externally identified seeds to obtain an attention-based evidence signal is a reasonable way to combine alignment with the target representation. Qwen2.5-VL provides a relevant test case because it lacks a paired CLIP text encoder.
-
The source-layer analysis and architecture-specific implementation details are useful. The layer is selected from images without question annotations, and the tested intermediate layers outperform the final layer on average. The experiments also disclose modest gains and additional runtime/memory costs. On Qwen, the default method improves average retention over SFPruner by 0.43 and 0.41 percentage points at 30% and 20% retention, respectively.
Critical weaknesses
-
The evidence does not isolate the paper's main new mechanism. SG-RLS and directional masking are inherited, while the ablations vary how evidence attention is computed rather than comparing it with direct relevance, visual-only guidance, or non-semantic seed controls under an otherwise identical selection pipeline. Moreover, Appendix C.1 states that the LLaVA implementation computes a global attention map in the VLAM and interpolates it to the high-resolution tiles, instead of extracting attention separately from those target-VLM tile representations. This is an important qualification of the claimed target-representation adaptation. Given the small main-table margins, controlled mechanism ablations are necessary to establish that query-guided evidence expansion is responsible for the improvement.
-
The accuracy-cost comparison is not yet sufficiently controlled. LLaVA baseline results in Table 1 are taken from prior reports, while QUEST fixes a 2x2 tile layout and evaluates its own pipeline. On Qwen, where the baselines are reproduced, the gain at 10% retention is only 0.02 points over CDPruner. At 20% retention, Table 6 reports 183.7 ms prefilling for QUEST versus 153.5 ms for SFPruner, and 17.0 versus 16.2 GB memory. That table labels the baselines' VLAM as absent, although Sec. 4.2 states that relevance-based baselines use the same external CLIP. The benefit at matched total resources is therefore unclear, especially for such small accuracy differences.
Questions for authors
-
With the same VLAM, preprocessing, visual guidance, and SG-RLS/DM implementation, what is the incremental effect of evidence attention over direct interpolated relevance, visual-only guidance, and random or visual-saliency seed selection? Please report paired differences on the same evaluation examples.
-
For LLaVA, how are the 576 base-tile tokens handled when the appendix defines guidance for only the 2304 spatial-tile tokens? Does extracting evidence attention from the actual target-VLM tiles change the result, and were the reported baselines evaluated with the same fixed layout?
-
Were external VLAM inference and attention extraction charged consistently to every method in Table 6? Please also clarify the common cache/backend, batch-size, and decoding protocol underlying the 197.1 ms/token full-model value versus 24.0 ms/token after pruning. These details could materially change the efficiency interpretation.
8 comments
Response to Q1
We thank the reviewer for the detailed and constructive review. Common settings are given in the general response.
Q1. (W1) With the same VLAM, preprocessing, visual guidance, and SG-RLS/DM implementation, what is the incremental effect of evidence attention over direct interpolated relevance, visual-only guidance, and random or visual-saliency seed selection? Please report paired differences on the same evaluation examples.
The increment over random and visual-saliency seeds and over visual-only guidance is significant. The increment over direct interpolated relevance is not significant on the paper benchmarks, and it is significant on the document and OCR benchmarks.
We fixed the VLAM, preprocessing, Visual Guidance, and SG-RLS/DM, and changed only the Evidence Attention part or the seeds (Qwen2.5-VL-7B).
Table R3-1. Paired differences on the paper benchmarks at 20%: average retention difference (pp, 95% CI), QUEST minus variant.
| Compared variant | Difference |
|---|---|
| Direct interpolated relevance | +0.17 [-0.53, +0.88] |
| Visual-only guidance | +1.18 [+0.40, +1.95] |
| Random seeds | +2.12 [+1.32, +2.92] |
| Visual-saliency seeds | +1.78 [+1.05, +2.53] |
At 10% and 30%, the difference from direct interpolated relevance is +0.43 [-0.42, +1.28] and +0.09 [-0.53, +0.73].
Table R3-2. Held-out benchmarks, QUEST minus direct interpolated relevance (score difference, 95% CI).
| Budget | DocVQA (ANLS) | OCRBench |
|---|---|---|
| 10% | +2.59 [+1.57, +3.58] | +3.60 [+1.10, +6.00] |
| 20% | +1.79 [+0.99, +2.59] | +3.50 [+1.40, +5.70] |
| 30% | +1.52 [+0.87, +2.23] | +3.20 [+1.20, +5.20] |
Over the 18 held-out benchmark-budget cells, QUEST is significantly higher than direct interpolated relevance in 6 cells and lower in 1 (MME-RW-Lite, 30%), and higher than visual-only guidance in 7 cells with no lower cell. No seed control is significantly higher than QUEST in any cell. We also repeated the random seeds with five seeds, and all five are significantly lower than QUEST (94.25 ± 0.15% vs. 96.16%, six paper benchmarks without HBench).
Response to Q2
Q2. (W1, W2) For LLaVA, how are the 576 base-tile tokens handled when the appendix defines guidance for only the 2304 spatial-tile tokens? Does extracting evidence attention from the actual target-VLM tiles change the result, and were the reported baselines evaluated with the same fixed layout?
The base tile is scored with its own attention and receives 1/5 of the budget. We did not run a per-tile variant on LLaVA-NeXT, and all baselines are now re-run with the same fixed layout.
We apologize that the appendix omitted the handling of the base tile.
Base tile. The 576 tokens of the base tile are scored with the base tile's own CLIP attention and Evidence Seeds, and the interpolated global maps are used only for the 2304 spatial-tile tokens. Of the token budget B = 2880 × r, the base tile receives B/5 (64 for 320 tokens and 128 for 640), which is one tile's share in LLaVA AnyRes, and the spatial tiles receive the rest. Ranking and the diversity computation are done separately in the two groups. The revision adds this to Appendix C.1.
Actual tiles. We did not run a per-tile variant on LLaVA-NeXT. Since the VLAM and the target VLM share the same CLIP encoder in LLaVA-NeXT, we compute the maps once on the whole image (Appendix C.1), and the revision states this clearly. InternVL3.5-4B, which we added, processes each tile separately in InternViT, so its Evidence Attention is extracted from the actual tiles. On InternVL, QUEST is the highest at 20% and 30% on the paper benchmarks, and comparable to or lower than the baselines at 10% (average retention: 89.66% vs. 89.01% and 89.19% at 20%, 92.63% vs. 91.62% and 91.87% at 30%, with source layer 21).
Baseline layout. The baselines in the original Table 1 were reported numbers. We re-ran all baselines with the same fixed layout, token count, budget split, and evaluation as QUEST (Table R3-3). The fixed layout does not significantly change the full model (-0.11 pp [-0.46, +0.23] against the native AnyRes layout). QUEST is comparable to SFPruner and CDPruner and significantly higher than DivPrune and VisionZip at both budgets.
Table R3-3. LLaVA-NeXT-7B, average retention difference (pp, 95% CI), QUEST minus baseline.
| Tokens | SFPruner | CDPruner | DivPrune | VisionZip |
|---|---|---|---|---|
| 640 | -0.32 [-0.69, +0.04] | -0.13 [-0.60, +0.34] | +1.46 [+0.85, +2.09] | +1.36 [+0.63, +2.09] |
| 320 | -0.31 [-0.83, +0.20] | +0.29 [-0.37, +0.94] | +2.27 [+1.56, +3.00] | +3.01 [+2.12, +3.90] |
Response Plan and Clarified Questions
We sincerely thank Reviewer FwRo for the detailed and constructive feedback. To facilitate our responses, we have organized the key questions below, using W1–W2 to refer to the corresponding weaknesses. We are currently conducting additional experiments and will update our responses to each point as soon as possible.