- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SQ-Pruner, a training-free visual token pruning framework for large vision-language models. Its two-stage selection combines Spectral Saliency Anchoring, which derives pixel-level structural importance from the input image, with Query-Guided Relevance Selection, which adaptively fuses centered visual–text similarity and query-to-visual attention to select additional tokens. Pruning is performed after the first two LLM decoder blocks by default. Validation spans multiple model families and scales, image and video understanding benchmarks, different token budgets, component ablations, and reported inference latency measurements
Critical strengths
-
Complete and coherent framework design. SQ-Pruner integrates pixel-level structural saliency, adaptive query-relevance scoring, and budget allocation into a complete training-free token-selection pipeline. The two stages have clearly defined, complementary roles: preserving structural information and supplementing query-relevant content.
-
Strong empirical performance under aggressive compression. On Qwen2.5-VL-7B, retaining only 64 of 1,296 visual tokens achieves 81.6% normalized average performance retention, compared with 77.3% for the strongest reported baseline, AnchorPrune. These results constitute the paper’s strongest contribution, demonstrating a meaningful accuracy advantage when the visual token budget is severely constrained.
-
Broad validation across models and modalities. Experiments cover Qwen2.5-VL at both 7B and 32B scales, LLaVA-1.5, LLaVA-NeXT, and Video-LLaVA, with multiple compression budgets and diverse image and video tasks. This breadth provides evidence that the method’s benefits extend beyond a single model or evaluation setting, strengthening the significance of its empirical results.
Critical weaknesses
-
Confounded comparison between Pixel SR and Token SR. Token SR first reduces each high-dimensional visual token to its norm, discarding all directional information, and then applies spectral residual analysis to the resulting scalar grid. The comparison therefore changes the input representation, spatial resolution, and dimensionality-reduction procedure simultaneously. Consequently, the observed performance gap does not isolate the benefit of pixel-level analysis and may instead reflect the information loss introduced by the Token SR baseline.
-
Limited methodological novelty. Combining visual importance with text relevance is already common in token compression, while DocPrune uses grayscale image information to guide pruning. SSA applies an established spectral residual saliency algorithm to token selection, following the existing idea of extracting low-level structural priors directly from input images. The paper insufficiently distinguishes its technical contribution from these prior approaches or establishes a substantive methodological advance beyond combining existing techniques.
-
Insufficient validation of practical acceleration. SQ-Pruner requires explicit query-to-visual attention scores and changes sequence lengths inside the LLM. Compatibility with FlashAttention and optimized serving frameworks such as vLLM and SGLang is not demonstrated. Table 9 does not specify the attention backend, inference framework used for SQ-Pruner and its baselines, making the reported speedup difficult to reproduce and its practical deployment benefits unclear.
-
Insufficient justification and validation of adaptive fusion. The fixed-weight comparison covers only a narrow range, without reporting adaptive-weight distributions across inputs or tasks. And, concentrated responses are not necessarily reliable: attention sinks and biased attention can also produce sharply peaked distributions. The proposed entropy-based weighting may therefore amplify these errors, yet no targeted analysis validates concentration as a reliable basis for fusion.
Questions for authors
-
Methodological novelty. What is the substantive technical advance over existing methods that combine visual importance with text relevance or use grayscale image information for pruning, such as DocPrune?
-
Practical inference efficiency. How exactly is token pruning implemented in LLMs? What inference framework, attention backend were used in Table 9? Does the reported speedup persist with FlashAttention, and has compatibility with vLLM or SGLang been evaluated?
-
Adaptive fusion. How do the adaptive weights vary across inputs and tasks? What evidence establishes that lower response entropy indicates more reliable relevance estimates, rather than attention sinks or concentrated spurious responses?
More question please refer to Weakness. And author's rebuttal will significantly influence my final rating.
7 comments
Response to Reviewer VbHp (Part II)
W2. Limited methodological novelty.
Thank you for your valuable feedback.
Following your suggestion, we have carefully reviewed the DocPrune [1] paper. Both methods leverage grayscale image information and query relevance. The methodological contribution of SQ-Pruner lies in its two-stage selection under a fixed token budget. SSA reserves part of the budget for structural information, while QRS selects visual evidence relevant to the current query from the remaining candidates.
SSA preserves structural anchors using spectral residual saliency. The background pruning module of DocPrune identifies and removes redundant background based on the proportion of pixels close to the dominant grayscale value within each image patch. In contrast, SSA builds on spectral residual saliency detection from computer vision [2]. It computes pixel-level saliency before visual encoding, applies local divisive normalization and maps the resulting scores to the visual token grid through region-wise averaging. SSA prioritizes structural information such as fine-grained text, chart elements and contours within its allocated budget. Although both methods use grayscale images, their objectives differ. DocPrune uses grayscale statistics to identify removable background regions, whereas SSA uses spectral saliency to identify structural anchors for retention.
QRS integrates two query-relevance signals within the model. The question-aware module of DocPrune computes cosine similarity using embeddings from the retrieval stage. Its comprehension-aware module further determines the pruning layer based on the model's estimated comprehension level. Inspired by centering techniques for word representations in NLP [3], QRS aggregates local visual features at the pruning layer of the LVLM and centers visual and query features separately. It then adaptively combines centered similarity with query-to-visual attention based on the response concentration of each signal. QRS uses the fused relevance scores to fill the remaining token budget.
On Qwen2.5-VL-7B with only 64 visual tokens retained, SSA-only and QRS-only achieve average performance of 77.5% and 80.7%, while the complete two-stage method achieves 81.6%. With SSA fixed, using only attention or centered similarity in the second stage yields 80.2% and 80.3%, while their adaptive fusion through QRS improves performance to 81.6%. These ablations support the complementary roles of structural anchoring and query relevance. They also show that combining centered similarity with attention improves token selection when the structural anchors are held fixed. We will add a citation to DocPrune in the revised manuscript and clarify the differences in structural scoring, relevance estimation and pruning procedures.
[1] Choi J, Lee S, Kim J, et al. DocPrune: Efficient Document Question Answering via Background, Question and Comprehension-aware Token Pruning.
[2] Hou X, Zhang L. Saliency Detection: A Spectral Residual Approach.
[3] Mu J, Bhat S, Viswanath P. All-but-the-Top: Simple and Effective Postprocessing for Word Representations.
Response to Reviewer VbHp (Part III)
W3. Insufficient validation of practical acceleration.
Thank you for your valuable comment.
Speedup with FlashAttention 2. The latency measurements in the original manuscript were conducted using PyTorch 2.7.0+cu128 and Hugging Face Transformers 4.55.4 on a single GPU with BF16 precision and a batch size of 1. All methods use the same model weights, input preprocessing and decoding configurations. We separately verified the attention backends used by the vision encoder and language model at runtime, confirming that all methods use consistent backends within the corresponding modules. Under this FlashAttention 2 configuration, SQ-Pruner reduces the prefill latency of Qwen2.5-VL-7B from 192.56 ms to 135.63 ms, achieving a 1.42× speedup. The Eager attention backend is used only for separate prefill FLOPs measurements, which account for the full visual encoding cost and include an analytical estimate of Pixel SR computation.
Pruning implementation within the LLM. QRS reuses the position-encoded Q/K representations at the pruning layer and only performs an additional attention computation from query tokens to all causally visible keys, followed by aggregation of the attention weights assigned to visual keys. The main attention computation continues to use FlashAttention 2. At the output of decoder block 1, SQ-Pruner gathers the retained visual and text hidden states in their original order, preserves their original position IDs and updates the attention mask. Subsequent layers directly process the shortened sequence.
End-to-end efficiency for long-form generation. We conducted additional generation experiments on DetailCaps, allowing generation to terminate naturally at EOS. E2E latency is measured from input preprocessing to the conversion of generated outputs into text, covering SSA precomputation, full visual encoding, QRS scoring, token selection, prefill and decoding. CUDA synchronization is performed at the timing boundaries. Model loading, warm-up and quality evaluation are excluded from the measured interval.
Table 2. Long-form captioning quality and end-to-end latency on DetailCaps.
| Method | CAPTURE ↑ | CIDEr ↑ | Avg. Output Tokens | E2E Latency ↓ |
|---|---|---|---|---|
| Vanilla | 63.51 | 0.136 | 181.3 | 5.695 |
| FastV | 53.26 | 0.063 | 145.2 | 4.486 |
| DART | 58.03 | 0.082 | 156.0 | 4.796 |
| DivPrune | 58.44 | 0.083 | 156.0 | 4.822 |
| VisionZip | 58.59 | 0.081 | 143.7 | 4.408 |
| HiPrune | 57.58 | 0.073 | 143.4 | 4.505 |
| AnchorPrune | 58.85 | 0.081 | 150.9 | 4.725 |
| Ours | 59.03 | 0.088 | 156.2 | 4.805 |
As shown in Table 2, with only 64 visual tokens retained, SQ-Pruner achieves the highest CAPTURE and CIDEr scores among the evaluated pruning methods. The average E2E latency decreases from 5.695 s to 4.805 s, representing a 15.6% reduction. Under natural generation, the average output length also decreases from 181.3 to 156.2 tokens.
Table 3. Inference efficiency on DetailCaps under controlled conditions.
| Method | Prefill FLOPs ↓ | Prefill Latency (ms)↓ | Decode Latency (ms/token)↓ | GPU Memory ↓ |
|---|---|---|---|---|
| Vanilla | 26.68 | 194.97 | 29.96 | 16.94 |
| FastV | 10.39 | 128.12 | 29.41 | 16.98 |
| DART | 9.80 | 130.18 | 29.52 | 16.98 |
| DivPrune | 8.55 | 128.76 | 30.07 | 16.93 |
| VisionZip | 8.62 | 137.49 | 29.36 | 17.17 |
| HiPrune | 8.69 | 151.32 | 29.76 | 17.17 |
| AnchorPrune | 8.62 | 142.55 | 29.54 | 17.17 |
| Ours | 9.75 | 135.06 | 29.77 | 17.00 |
Table 3 further evaluates efficiency under controlled conditions. Compared with Vanilla, SQ-Pruner reduces prefill FLOPs from 26.68 T to 9.75 T and prefill latency from 194.97 ms to 135.06 ms . These results demonstrate that SQ-Pruner substantially reduces prefill computational cost and measured latency.
Compatibility with serving frameworks. Our current validation covers the Transformers with FlashAttention 2 inference pipeline. Compatibility with vLLM and SGLang has not yet been evaluated. In future work, we will investigate how to support inter-layer compression in these frameworks. We will clarify the software versions, actual attention backends and timing methodology in the revised manuscript.
Response to Reviewer VbHp (Part I)
We sincerely thank you for your careful reading and valuable feedback. We will highlight the corresponding revisions in blue in the revised manuscript for clarity. We hope that our responses and the revised manuscript will fully address your concerns. Due to OpenReview's length limit, we have divided our response into multiple parts for better readability. Thank you for your understanding.
Thank you for your valuable time and thorough review of our paper.
The original experiments compared two saliency estimation pipelines. Token SR constructs a single-channel grid using the ℓ2 norm of visual tokens, while Pixel SR computes saliency directly from the grayscale image. To further investigate the effects of norm projection and spatial aggregation, we conducted two additional experiments on Qwen2.5-VL-7B, as shown in Table 1.
Norm Projection. Multi-channel Token SR (MC Token SR) computes SR independently for each channel of the visual token grid, followed by normalization and aggregation of the resulting saliency maps. At token budgets of 256, 128 and 64, MC Token SR does not achieve higher average performance than the original norm-based Token SR. This suggests that removing norm projection before SR computation does not reduce the performance gap between Token SR and Pixel SR in the tested implementation.
Spatial Aggregation. Token-resolution Pixel SR (TR Pixel SR) applies average pooling to the same grayscale image to match the visual token grid resolution before computing SR. Compared with TR Pixel SR, Pixel SR computes SR before pooling and achieves 4.3%, 3.6% and 2.8% higher average performance at token budgets of 256, 128 and 64. This controlled comparison using the same input supports the benefit of computing saliency before spatial aggregation.
We will include these controlled comparisons in the revised manuscript and clarify that the original experiments compared the complete Pixel SR and norm-based Token SR pipelines.
Table 1. Comparison of different SR input representations and computation resolutions. Results show the average performance relative to the full model across 12 benchmarks on Qwen2.5-VL-7B. The full model uses 1,296 visual tokens.