acceptodds
Under review as a conference paper at ICLR 2027

SQ-Pruner: Spectral Saliency and Query-Guided Visual Token Pruning in Large Vision-Language Models

27 views

Abstract

Redundant visual tokens impose substantial computational overhead on Large Vision-Language Models (LVLMs). Prior studies typically estimate token importance from visual representations after patchification or spatial aggregation, potentially weakening fine-grained structural information. Moreover, existing query-guided pruning methods often rely on a single relevance signal, which may not fully capture the correspondence between visual tokens and the input query. To address these issues, we propose Spectral and Query-Guided Pruner (SQ-Pruner), a training-free visual token pruning framework that adopts a two-stage pipeline: 1) Spectral Saliency Anchoring (SSA) estimates pixel-level spectral saliency before visual encoding to preserve structurally informative tokens as anchors. 2) Query-Guided Relevance Selection (QRS) adaptively fuses query-related signals to select visual tokens from the remaining candidates. Extensive experiments demonstrate the effectiveness of our framework across diverse LVLMs and tasks in both image and video modalities. On Qwen2.5-VL, SQ-Pruner retains 81.6% of the original performance while pruning 95.1% of visual tokens, outperforming the strongest prior method by 4.3%. SQ-Pruner also achieves a 1.42 prefilling speedup over the original model, demonstrating a favorable accuracy-efficiency trade-off for LVLM inference.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Model, Token Reduction

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_VbHpOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SQ-Pruner, a training-free visual token pruning framework for large vision-language models. Its two-stage selection combines Spectral Saliency Anchoring, which derives pixel-level structural importance from the input image, with Query-Guided Relevance Selection, which adaptively fuses centered visual–text similarity and query-to-visual attention to select additional tokens. Pruning is performed after the first two LLM decoder blocks by default. Validation spans multiple model families and scales, image and video understanding benchmarks, different token budgets, component ablations, and reported inference latency measurements

Critical strengths
  • Complete and coherent framework design. SQ-Pruner integrates pixel-level structural saliency, adaptive query-relevance scoring, and budget allocation into a complete training-free token-selection pipeline. The two stages have clearly defined, complementary roles: preserving structural information and supplementing query-relevant content.

  • Strong empirical performance under aggressive compression. On Qwen2.5-VL-7B, retaining only 64 of 1,296 visual tokens achieves 81.6% normalized average performance retention, compared with 77.3% for the strongest reported baseline, AnchorPrune. These results constitute the paper’s strongest contribution, demonstrating a meaningful accuracy advantage when the visual token budget is severely constrained.

  • Broad validation across models and modalities. Experiments cover Qwen2.5-VL at both 7B and 32B scales, LLaVA-1.5, LLaVA-NeXT, and Video-LLaVA, with multiple compression budgets and diverse image and video tasks. This breadth provides evidence that the method’s benefits extend beyond a single model or evaluation setting, strengthening the significance of its empirical results.

Critical weaknesses
  • Confounded comparison between Pixel SR and Token SR. Token SR first reduces each high-dimensional visual token to its ℓ2\ell_2 norm, discarding all directional information, and then applies spectral residual analysis to the resulting scalar grid. The comparison therefore changes the input representation, spatial resolution, and dimensionality-reduction procedure simultaneously. Consequently, the observed performance gap does not isolate the benefit of pixel-level analysis and may instead reflect the information loss introduced by the Token SR baseline.

  • Limited methodological novelty. Combining visual importance with text relevance is already common in token compression, while DocPrune uses grayscale image information to guide pruning. SSA applies an established spectral residual saliency algorithm to token selection, following the existing idea of extracting low-level structural priors directly from input images. The paper insufficiently distinguishes its technical contribution from these prior approaches or establishes a substantive methodological advance beyond combining existing techniques.

  • Insufficient validation of practical acceleration. SQ-Pruner requires explicit query-to-visual attention scores and changes sequence lengths inside the LLM. Compatibility with FlashAttention and optimized serving frameworks such as vLLM and SGLang is not demonstrated. Table 9 does not specify the attention backend, inference framework used for SQ-Pruner and its baselines, making the reported speedup difficult to reproduce and its practical deployment benefits unclear.

  • Insufficient justification and validation of adaptive fusion. The fixed-weight comparison covers only a narrow range, without reporting adaptive-weight distributions across inputs or tasks. And, concentrated responses are not necessarily reliable: attention sinks and biased attention can also produce sharply peaked distributions. The proposed entropy-based weighting may therefore amplify these errors, yet no targeted analysis validates concentration as a reliable basis for fusion.

Questions for authors
  1. Methodological novelty. What is the substantive technical advance over existing methods that combine visual importance with text relevance or use grayscale image information for pruning, such as DocPrune?

  2. Practical inference efficiency. How exactly is token pruning implemented in LLMs? What inference framework, attention backend were used in Table 9? Does the reported speedup persist with FlashAttention, and has compatibility with vLLM or SGLang been evaluated?

  3. Adaptive fusion. How do the adaptive weights vary across inputs and tasks? What evidence establishes that lower response entropy indicates more reliable relevance estimates, rather than attention sinks or concentrated spurious responses?

More question please refer to Weakness. And author's rebuttal will significantly influence my final rating.

7 comments
AuthorsOpenReview ↗

Response to Reviewer VbHp (Part I)

We sincerely thank you for your careful reading and valuable feedback. We will highlight the corresponding revisions in blue in the revised manuscript for clarity. We hope that our responses and the revised manuscript will fully address your concerns. Due to OpenReview's length limit, we have divided our response into multiple parts for better readability. Thank you for your understanding.

W1. Confounded comparison between Pixel SR and Token SR.

Thank you for your valuable time and thorough review of our paper.

The original experiments compared two saliency estimation pipelines. Token SR constructs a single-channel grid using the ℓ2\ell_2 norm of visual tokens, while Pixel SR computes saliency directly from the grayscale image. To further investigate the effects of norm projection and spatial aggregation, we conducted two additional experiments on Qwen2.5-VL-7B, as shown in Table 1.

Norm Projection. Multi-channel Token SR (MC Token SR) computes SR independently for each channel of the visual token grid, followed by normalization and aggregation of the resulting saliency maps. At token budgets of 256, 128 and 64, MC Token SR does not achieve higher average performance than the original norm-based Token SR. This suggests that removing norm projection before SR computation does not reduce the performance gap between Token SR and Pixel SR in the tested implementation.

Spatial Aggregation. Token-resolution Pixel SR (TR Pixel SR) applies average pooling to the same grayscale image to match the visual token grid resolution before computing SR. Compared with TR Pixel SR, Pixel SR computes SR before pooling and achieves 4.3%, 3.6% and 2.8% higher average performance at token budgets of 256, 128 and 64. This controlled comparison using the same input supports the benefit of computing saliency before spatial aggregation.

We will include these controlled comparisons in the revised manuscript and clarify that the original experiments compared the complete Pixel SR and norm-based Token SR pipelines.

Table 1. Comparison of different SR input representations and computation resolutions. Results show the average performance relative to the full model across 12 benchmarks on Qwen2.5-VL-7B. The full model uses 1,296 visual tokens.

Method256 Tokens128 Tokens64 Tokens
Token SR78.3%68.7%62.5%
MC Token SR77.1%67.9%62.0%
TR Pixel SR87.9%80.4%72.7%
Pixel SR92.2%84.0%75.5%
AuthorsOpenReview ↗

Response to Reviewer VbHp (Part II)

W2. Limited methodological novelty.

Thank you for your valuable feedback.

Following your suggestion, we have carefully reviewed the DocPrune [1] paper. Both methods leverage grayscale image information and query relevance. The methodological contribution of SQ-Pruner lies in its two-stage selection under a fixed token budget. SSA reserves part of the budget for structural information, while QRS selects visual evidence relevant to the current query from the remaining candidates.

SSA preserves structural anchors using spectral residual saliency. The background pruning module of DocPrune identifies and removes redundant background based on the proportion of pixels close to the dominant grayscale value within each image patch. In contrast, SSA builds on spectral residual saliency detection from computer vision [2]. It computes pixel-level saliency before visual encoding, applies local divisive normalization and maps the resulting scores to the visual token grid through region-wise averaging. SSA prioritizes structural information such as fine-grained text, chart elements and contours within its allocated budget. Although both methods use grayscale images, their objectives differ. DocPrune uses grayscale statistics to identify removable background regions, whereas SSA uses spectral saliency to identify structural anchors for retention.

QRS integrates two query-relevance signals within the model. The question-aware module of DocPrune computes cosine similarity using embeddings from the retrieval stage. Its comprehension-aware module further determines the pruning layer based on the model's estimated comprehension level. Inspired by centering techniques for word representations in NLP [3], QRS aggregates local visual features at the pruning layer of the LVLM and centers visual and query features separately. It then adaptively combines centered similarity with query-to-visual attention based on the response concentration of each signal. QRS uses the fused relevance scores to fill the remaining token budget.

On Qwen2.5-VL-7B with only 64 visual tokens retained, SSA-only and QRS-only achieve average performance of 77.5% and 80.7%, while the complete two-stage method achieves 81.6%. With SSA fixed, using only attention or centered similarity in the second stage yields 80.2% and 80.3%, while their adaptive fusion through QRS improves performance to 81.6%. These ablations support the complementary roles of structural anchoring and query relevance. They also show that combining centered similarity with attention improves token selection when the structural anchors are held fixed. We will add a citation to DocPrune in the revised manuscript and clarify the differences in structural scoring, relevance estimation and pruning procedures.

[1] Choi J, Lee S, Kim J, et al. DocPrune: Efficient Document Question Answering via Background, Question and Comprehension-aware Token Pruning.

[2] Hou X, Zhang L. Saliency Detection: A Spectral Residual Approach.

[3] Mu J, Bhat S, Viswanath P. All-but-the-Top: Simple and Effective Postprocessing for Word Representations.

AuthorsOpenReview ↗

Response to Reviewer VbHp (Part III)

W3. Insufficient validation of practical acceleration.

Thank you for your valuable comment.

Speedup with FlashAttention 2. The latency measurements in the original manuscript were conducted using PyTorch 2.7.0+cu128 and Hugging Face Transformers 4.55.4 on a single GPU with BF16 precision and a batch size of 1. All methods use the same model weights, input preprocessing and decoding configurations. We separately verified the attention backends used by the vision encoder and language model at runtime, confirming that all methods use consistent backends within the corresponding modules. Under this FlashAttention 2 configuration, SQ-Pruner reduces the prefill latency of Qwen2.5-VL-7B from 192.56 ms to 135.63 ms, achieving a 1.42× speedup. The Eager attention backend is used only for separate prefill FLOPs measurements, which account for the full visual encoding cost and include an analytical estimate of Pixel SR computation.

Pruning implementation within the LLM. QRS reuses the position-encoded Q/K representations at the pruning layer and only performs an additional attention computation from query tokens to all causally visible keys, followed by aggregation of the attention weights assigned to visual keys. The main attention computation continues to use FlashAttention 2. At the output of decoder block 1, SQ-Pruner gathers the retained visual and text hidden states in their original order, preserves their original position IDs and updates the attention mask. Subsequent layers directly process the shortened sequence.

End-to-end efficiency for long-form generation. We conducted additional generation experiments on DetailCaps, allowing generation to terminate naturally at EOS. E2E latency is measured from input preprocessing to the conversion of generated outputs into text, covering SSA precomputation, full visual encoding, QRS scoring, token selection, prefill and decoding. CUDA synchronization is performed at the timing boundaries. Model loading, warm-up and quality evaluation are excluded from the measured interval.

Table 2. Long-form captioning quality and end-to-end latency on DetailCaps.

MethodCAPTURE ↑CIDEr ↑Avg. Output TokensE2E Latency ss ↓
Vanilla63.510.136181.35.695
FastV53.260.063145.24.486
DART58.030.082156.04.796
DivPrune58.440.083156.04.822
VisionZip58.590.081143.74.408
HiPrune57.580.073143.44.505
AnchorPrune58.850.081150.94.725
Ours59.030.088156.24.805

As shown in Table 2, with only 64 visual tokens retained, SQ-Pruner achieves the highest CAPTURE 590˙359\.03 and CIDEr 00˙880\.088 scores among the evaluated pruning methods. The average E2E latency decreases from 5.695 s to 4.805 s, representing a 15.6% reduction. Under natural generation, the average output length also decreases from 181.3 to 156.2 tokens.

Table 3. Inference efficiency on DetailCaps under controlled conditions.

MethodPrefill FLOPs TT↓Prefill Latency (ms)↓Decode Latency (ms/token)↓GPU Memory GBGB ↓
Vanilla26.68194.9729.9616.94
FastV10.39128.1229.4116.98
DART9.80130.1829.5216.98
DivPrune8.55128.7630.0716.93
VisionZip8.62137.4929.3617.17
HiPrune8.69151.3229.7617.17
AnchorPrune8.62142.5529.5417.17
Ours9.75135.0629.7717.00

Table 3 further evaluates efficiency under controlled conditions. Compared with Vanilla, SQ-Pruner reduces prefill FLOPs from 26.68 T to 9.75 T and prefill latency from 194.97 ms to 135.06 ms 14˙4×speedup1\.44× speedup. These results demonstrate that SQ-Pruner substantially reduces prefill computational cost and measured latency.

Compatibility with serving frameworks. Our current validation covers the Transformers with FlashAttention 2 inference pipeline. Compatibility with vLLM and SGLang has not yet been evaluated. In future work, we will investigate how to support inter-layer compression in these frameworks. We will clarify the software versions, actual attention backends and timing methodology in the revised manuscript.

Reviewer_VQz6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes SQ-Pruner, a training-free visual token pruning framework that combines pixel-level spectral saliency with query-guided relevance estimation. The method first identifies structurally informative visual tokens using spectral residual analysis and then selects additional query-relevant tokens by adaptively combining attention and centered cosine similarity. Experiments across multiple vision-language models and image/video benchmarks demonstrate improved performance preservation under aggressive pruning, along with reduced inference costs.

Critical strengths
  1. SQ-Pruner does not require additional training or learnable parameters, making it straightforward to integrate with existing VLM architectures.
  2. The ablation study indicates that combining SSA and QRS yields better performance than either mechanism independently. The efficiency analysis also reports reduced total FLOPs and prefill latency, supporting the practical value.
Critical weaknesses
  1. From my understanding, SSA was applied to the raw input image instead of tokens. Figure 2 visualizes SSA in each Transformer layer, which is not consistent with the method.
  2. Although the combination of pixel-level spectral saliency and query-guided token selection is interesting, the individual components are primarily based on existing techniques. Spectral residual saliency is a well-established image-processing method, while attention-based and similarity-based token importance estimation have already been extensively explored in visual token pruning.
  3. The paper demonstrates that pixel-level spectral saliency performs well empirically, but it remains unclear whether the selected tokens consistently correspond to task-relevant visual evidence. It would be better to quantify this using VQA benchmarks that contain bounding-box annotations indicating where the task-relevant visual evidence is in pixel space. I suggest the authors consider RefCOCOg.
Questions for authors

See weakness.

5 comments
AuthorsOpenReview ↗

Response to Reviewer VQz6 (Part I)

We sincerely thank you for your careful reading and valuable feedback. We will highlight the corresponding revisions in blue in the revised manuscript for clarity. We hope that our responses and the revised manuscript will fully address your concerns. Due to OpenReview's length limit, we have divided our response into multiple parts for better readability. Thank you for your understanding.

W1. Figure 2 visualizes SSA in each Transformer layer, which is not consistent with the method.

Thank you for your careful reading and constructive feedback.

SSA saliency scores are computed from the input image before visual encoding and mapped onto the visual token grid. At the pruning layer, SQ-Pruner uses these precomputed scores to select structural anchors, then computes QRS scores to select additional tokens from the remaining candidates and fill the token budget.

In the original Figure 2, the enlarged inset illustrates both SSA score precomputation and token selection within the pruning layer. We will clarify the input source and precomputation process of SSA in the revised manuscript to provide a more accurate representation of the actual method pipeline.

W2. Although the combination of pixel-level spectral saliency and query-guided token selection is interesting, the individual components are primarily based on existing techniques.

We agree that spectral residual saliency have been studied in prior work. The novelty of SQ-Pruner lies in allocating a fixed visual token budget using pre-encoding pixel-level structural priors and query relevance estimated at the pruning layer. SSA uses pixel-level saliency to reserve part of the budget for fine-grained visual structures. QRS then selects query-relevant visual tokens from the remaining candidates. This two-stage design allows structural saliency and query relevance to independently determine their respective portions of the retained token budget.

SSA reserves token budget for fine-grained visual structures. We build on the spectral residual saliency method of Hou et al. [1]. SSA computes pixel-level saliency before visual encoding, applies local divisive normalization and aggregates the resulting scores onto the visual token grid. These scores guide the retention of text, chart elements and object contours independently of the encoded features. To further evaluate this design, we extend the original Pixel SR and Token SR comparison with two controlled ablations.

Norm Projection. Multi-channel Token SR (MC Token SR) computes SR separately for each channel on the visual token grid. It then normalizes and aggregates the resulting saliency maps.

Spatial Aggregation. Token-resolution Pixel SR (TR Pixel SR) applies average pooling to the same grayscale image to match the resolution of the visual token grid. It then computes SR on the pooled image.

As shown in Table 1, Pixel SR achieves the highest average performance at all three token budgets. These results provide further support for our design choice of extracting structural priors before visual encoding.

Table 1. Comparison of SR input representations and computation resolutions. We report average performance across 12 benchmarks on Qwen2.5-VL-7B relative to the full model with 1,296 visual tokens.

Method256 Tokens128 Tokens64 Tokens
Token SR78.30%68.70%62.50%
MC Token SR77.10%67.90%62.00%
TR Pixel SR87.90%80.40%72.70%
Pixel SR92.20%84.00%75.50%

QRS adaptively fuses two signals to select additional query-relevant tokens. Inspired by the feature-centering study of Mu et al. [2], QRS aggregates local visual features at the pruning layer and separately centers visual and query features. It then adaptively fuses centered similarity with query-guided attention based on the response distributions of the relevance signals, selecting additional visual tokens relevant to the current query from the candidates not selected by SSA. Unlike simply ranking tokens through weighted combinations of multiple scores, the two-stage mechanism provides distinct selection pathways for structural saliency and query relevance.

Under the 64-token budget on Qwen2.5-VL-7B, our SQ-Pruner achieves an average performance of 81.6%, outperforming SSA-only 775˙77\.5% and QRS-only 807˙80\.7%. This demonstrates the benefits of structural anchors and query-guided selection. With SSA fixed, the complete QRS also outperforms using Attention alone 802˙80\.2% or Centered Similarity alone 803˙80\.3%, further supporting the effectiveness of two-signal fusion.

[1] Hou X, Zhang L. Saliency Detection: A Spectral Residual Approach.

[2] Mu J, Bhat S, Viswanath P. All-but-the-Top: Simple and Effective Postprocessing for Word Representations.

AuthorsOpenReview ↗

Response to Reviewer VQz6 (Part II)

Furthermore, during the rebuttal period, we conducted additional experiments with different fusion coefficients β\beta. Adaptive fusion achieves 94.75%, 88.73% and 81.60% under the 256-, 128- and 64-token budgets, consistently outperforming the best fixed-weight configurations 945˙994\.59%, 88\.46% and 81\.28%. It also improves over random-weight fusion by 0.38%, 0.51% and 1.29%, respectively. These results demonstrate that our proposed response-distribution-based adaptive fusion consistently improves token selection across different token budgets. We will further clarify the contributions of the two-stage selection mechanism and adaptive fusion in the revised manuscript.

Table 2. Comparison of fixed β\beta over a wider range.

MethodRetained TokensMME (Total)GQAPOPE (F1 score)ScienceQAMMB ENENMMB CNCNTextVQADocVQAOCRBenchAI2DChartQAMMMUAvg.
Vanilla1,2962326.959.885.987.883.880.282.594.174583.080.550.7100.0%
β=0.12562299.058.386.585.581.177.777.373.563581.572.548.894.13%
β=0.252562308.958.386.585.780.977.877.973.864281.472.748.894.35%
β=0.52562323.458.586.285.780.577.278.774.364381.273.449.294.56%
β = 0.82562291.057.886.186.381.177.379.073.864581.673.548.294.35%
β = 1.02562305.457.886.486.281.077.579.373.664281.673.148.494.41%
β = 1.22562283.857.786.086.181.077.779.474.164881.773.148.494.42%
β=22562286.458.085.785.481.277.779.575.364281.573.449.294.59%
β=42562280.157.585.085.580.777.179.376.364881.273.749.294.46%
β=102562263.157.884.585.380.677.479.374.964581.673.549.194.24%
Ours2562294.158.486.385.980.877.479.575.164681.673.649.194.75%
β=0.11282250.157.786.182.578.074.670.255.052478.461.747.687.70%
β=0.251282242.757.685.983.277.875.171.155.152678.662.548.388.07%
β=0.51282237.357.685.383.177.774.272.355.953179.162.847.888.13%
β = 0.81282257.156.785.582.678.074.072.055.954078.362.947.588.01%
β = 1.01282254.256.585.282.478.174.872.756.054378.063.947.388.17%
β = 1.21282248.856.485.182.477.975.273.656.254178.364.047.288.26%
β=21282239.556.984.582.778.274.873.856.554578.964.147.588.46%
β=41282194.956.483.982.977.674.774.156.955179.464.647.688.38%
β=101282185.556.383.482.678.175.273.755.254279.263.748.088.01%
Ours1282246.457.285.582.777.874.973.556.755579.063.847.888.73%
β=0.1642146.355.885.181.576.671.858.738.138974.848.445.280.14%
β=0.25642154.855.285.481.176.371.159.537.540075.449.345.780.36%
β=0.5642132.055.484.881.676.471.261.638.441274.350.945.580.76%
β=0.8642118.254.483.781.175.970.661.537.139873.951.845.880.13%
β=1642148.454.683.481.475.970.962.537.640974.452.645.280.60%
β=1.2642144.954.383.081.675.471.063.337.341074.552.545.080.52%
β=2642174.154.282.781.476.071.564.438.541976.053.745.181.28%
β=4642098.654.181.381.275.471.666.339.342474.753.845.481.04%
β=10642041.653.579.181.275.871.065.936.842774.753.445.380.23%
Ours642165.855.384.081.176.571.564.538.942475.852.945.481.60%

Table 3. Performance comparison with random fusion.

MethodRetained TokensMME (Total)GQAPOPE (F1 score)ScienceQAMMB ENENMMB CNCNTextVQADocVQAOCRBenchAI2DChartQAMMMUAvg.
Vanilla1,2962326.959.885.987.883.880.282.594.174583.080.550.7100.0%
random2562293.458.285.385.280.076.979.275.664281.273.649.294.37%
random1282198.756.383.882.677.474.273.956.254678.464.748.788.22%
random642117.353.881.880.173.769.564.838.542774.352.245.980.31%

Table 4. Average adaptive fusion coefficient across datasets and token budgets.

DatasetK=256K=128K=64
MME1.3161.3891.402
GQA0.8300.7850.761
POPE0.9100.8710.863
ScienceQA1.2021.2351.253
TextVQA1.5231.5571.563
OCRBench1.3651.3601.343
ChartQA1.2031.1731.147
AuthorsOpenReview ↗

Response to Reviewer VQz6 (Part III)

W3. The paper demonstrates that pixel-level spectral saliency performs well empirically, but it remains unclear whether the selected tokens consistently correspond to task-relevant visual evidence.

Thank you for your helpful suggestion.

Following your suggestion, to further evaluate whether the visual information retained by SQ-Pruner supports task-relevant object localization, we conducted additional referring expression localization experiments on RefCOCOg. We use [email protected] and mIoU to evaluate the agreement between predicted bounding boxes and ground-truth target regions. SSA-only isolates the contribution of the structural prior, while the full method further evaluates the effect of incorporating query-guided selection.

Table 5. Grounding performance on RefCOCOg under different visual token budgets.

MethodRetained TokensVal [email protected]Test [email protected]Val mIoUTest mIoU
Vanilla1,29686.1085.4978.9378.65
FastV25639.4338.7439.5239.09
DART25638.7638.2139.5539.36
AnchorPrune25655.8154.8350.1049.91
SSA-only25649.1348.8445.4044.80
Ours25660.7760.4752.9552.74
FastV12815.6816.1922.4722.46
DART12816.1415.6424.2823.81
AnchorPrune12826.4825.5232.9832.63
SSA-only12825.9425.3031.3130.62
Ours12828.8228.0834.3634.00
FastV648.327.6715.8015.11
DART648.478.4016.4716.35
AnchorPrune6413.4612.9321.9421.59
SSA-only6412.5811.8022.3821.59
Ours6413.5113.3122.3821.74

As shown in Table 5, with 256, 128 and 64 visual tokens retained, SQ-Pruner achieves the best results on both [email protected] and mIoU across the validation and test sets. These results demonstrate that the structural priors extracted by SSA help preserve visual information for object localization. Furthermore, QRS incorporates query relevance to better align the retained visual information with the current task, enabling more effective visual token selection.

Reviewer_5SntOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SQ-Pruner, a training-free visual token pruning method. It first uses pixel-level spectral saliency to select structural anchors, then combines query attention and centred similarity to select additional tokens. Pruning is applied after a shallow LLM layer, reducing the visual sequence processed by later layers. Experiments across several image and video models show a useful trade-off between benchmark performance and inference cost.

Critical strengths
  1. The design is simple and addresses a practical problem. Combining image structure with query relevance is a sensible way to avoid relying on either signal alone. The method requires no additional training and can be added to existing models. Its main contribution is a useful combination of pixel-level spectral cues and query-guided selection.

  2. The experiments provide good support for the proposed combination. The paper compares several pruning baselines and includes ablations on the two stages, fusion weights, and pruning location. On Qwen2.5-VL-7B, retaining 64 of 1,296 visual tokens preserves 81.6% of the original performance on average, versus 77.3% for AnchorPrune. The measured prefill speedup also supports the practical value of the method.

Critical weaknesses
  1. The efficiency evidence does not yet cover open-ended, long-form generation. SSA is computed from pixels before visual encoding, but tokens are actually removed after Layer 1. The full visual encoder and earlier LLM computation are therefore still required.

Table 9 reports both prefill and inference latency, but on POPE, a short-answer object-presence task. These results do not show the quality and total latency of longer multimodal responses. Open-ended visual QA or detailed image/video description would be useful, with actual output lengths and end-to-end timing that includes visual encoding, pruning, prefill, and decoding stages.

  1. The scoring rules are empirically useful, but their reliability is not fully explained. The paper provides no theorem or error bound establishing that spectral anchors preserve necessary semantic evidence, that centring improves cross-modal alignment, or that a lower-entropy relevance signal is more reliable.

The ablations support the design, but more direct analysis or examples would help explain when these assumptions hold.

Questions for authors
  1. Could video pruning make better use of temporal information? The current SQ-Pruner selects tokens independently in each of eight frames. It does not explicitly use repetition or changes across frames to remove redundant content or prioritise important moments. A comparison with a simple temporally aware selection rule may help assess this limitation, especially for longer videos.

  2. How well does the method transfer to newer architectures? The existing results span several models, but the same pruning location and relevance signals may behave differently with newer visual encoders, token merging schemes, or video architectures. Testing on newer architecture would strengthen the transferability claim.

8 comments
AuthorsOpenReview ↗

Response to Reviewer 5Snt (Part I)

We sincerely thank you for your careful reading and valuable feedback. We will highlight the corresponding revisions in blue in the revised manuscript for clarity. We hope that our responses and the revised manuscript will fully address your concerns. Due to OpenReview's length limit, we have divided our response into multiple parts for better readability. Thank you for your understanding.

W1. The efficiency evidence does not yet cover open-ended, long-form generation.

Thank you for raising this important concern.

Following your suggestion, we conducted experiments on DetailCaps to evaluate long-form responses. All methods were evaluated using the same model, hardware and decoding configurations with generation terminating naturally upon reaching the EOS token.

Table 1. Long-form captioning quality and end-to-end performance on DetailCaps.

MethodCAPTURE ↑CIDEr ↑Avg. Output TokensE2E Latency ss ↓
Vanilla63.510.136181.35.695
FastV53.260.063145.24.486
DART58.030.082156.04.796
DivPrune58.440.083156.04.822
VisionZip58.590.081143.74.408
HiPrune57.580.073143.44.505
AnchorPrune58.850.081150.94.725
Ours59.030.088156.24.805

As shown in Table 1, with only 64 visual tokens retained, SQ-Pruner achieves the highest CAPTURE 590˙359\.03 and CIDEr 00˙880\.088 scores among the evaluated pruning methods. The average E2E latency decreases from 5.695 s to 4.805 s, representing a 15.6% reduction. Under natural generation, the average output length also decreases from 181.3 to 156.2 tokens.

Table 2. Inference efficiency on DetailCaps under controlled conditions.

MethodPrefill FLOPs TT↓Prefill Latency (ms)↓Decode Latency (ms/token)↓GPU Memory GBGB ↓
Vanilla26.68194.9729.9616.94
FastV10.39128.1229.4116.98
DART9.80130.1829.5216.98
DivPrune8.55128.7630.0716.93
VisionZip8.62137.4929.3617.17
HiPrune8.69151.3229.7617.17
AnchorPrune8.62142.5529.5417.17
Ours9.75135.0629.7717.00

Table 2 further evaluates efficiency under controlled conditions. Compared with Vanilla, SQ-Pruner reduces prefill FLOPs from 26.68 T to 9.75 T and prefill latency from 194.97 ms to 135.06 ms 14˙4×speedup1\.44× speedup. These results demonstrate that SQ-Pruner substantially reduces prefill computational cost and measured latency.

AuthorsOpenReview ↗

Response to Reviewer 5Snt (Part II)

W2. The scoring rules are empirically useful, but their reliability is not fully explained.

Thank you for your valuable feedback. Below, we address each of your suggestions with detailed explanations and additional experimental results.

aa Ability of spectral anchors to preserve task-relevant visual evidence.

In our SQ-Pruner, SSA extracts structural priors from pixel-level spectral information to preserve fine-grained details that may be weakened after visual encoding, while QRS further incorporates query relevance. Following your suggestion, we conducted additional experiments on the RefCOCOg benchmark to evaluate whether the visual information preserved by spectral anchors supports downstream localization. This task requires the model to localize a target region based on a textual description, making it suitable for assessing whether retained visual tokens preserve information necessary for object localization.

Table 3. RefCOCOg localization results.

MethodRetained TokensVal [email protected]%↑Test [email protected]%↑Val mIoU%↑Test mIoU%↑
Vanilla1,29686.1085.4978.9378.65
FastV25639.4338.7439.5239.09
DART25638.7638.2139.5539.36
SSA-only25649.1348.8445.4044.80
FastV12815.6816.1922.4722.46
DART12816.1415.6424.2823.81
SSA-only12825.9425.3031.3130.62
FastV648.327.6715.8015.11
DART648.478.4016.4716.35
SSA-only6412.5811.8022.3821.59

At visual token budgets of 256, 128 and 64, SSA-only achieves Test [email protected] scores of 48.84%, 25.30% and 11.80%, outperforming both FastV and DART across all budgets. Consistent improvements are also observed on the validation set. These results suggest that the structural priors extracted by SSA help preserve visual information necessary for object localization, even under aggressive token pruning. However, SSA-only still performs substantially below Vanilla. This further motivates the incorporation of QRS to combine structural priors with query relevance for more effective visual token selection.

bb Role of centering in cross-modal relevance estimation.

We apply centering separately to visual and query features to reduce the influence of shared components within each modality on cosine similarity. Let the locally pooled visual features and query features be expressed as vi=μV+siv_i=\mu_V+s_i and tj=μT+qjt_j=\mu_T+q_j, where μV\mu_V and μT\mu_T represent shared additive components within their corresponding modalities. After subtracting the modality-specific means, we obtain vi−vˉ=si−sˉv_i-\bar{v}=s_i-\bar{s} and tj−tˉ=qj−qˉt_j-\bar{t}=q_j-\bar{q}. Thus, centering removes the shared additive components under this decomposition. When these components dominate the original feature directions, the resulting cosine similarity can better capture token-specific directional differences.

Table 4. Ablation study of the two-stage selection.

stage1stage2Avg%
--256 Tokens128 Tokens64 Tokens
SSA-93.285.677.5
QRS-93.687.880.7
SSAAttention93.587.680.2
SSACosine93.286.077.6
SSACentered94.388.080.3
SSAQRS94.788.781.6

We have discussed this in the paper and provided ablation experiments to further validate the effectiveness of centering. As shown in Table 4, with SSA and all other experimental settings kept unchanged, Centered Similarity achieves 94.3%, 88.0% and 80.3% under the 256-, 128- and 64-token budgets. Centered Similarity outperforms standard Cosine Similarity by 1.1%, 2.0% and 2.7%. These results demonstrate the effectiveness of centering for relevance estimation, with larger gains under tighter token budgets.

AuthorsOpenReview ↗

Response to Reviewer 5Snt (Part III)

cc Adaptive fusion based on response concentration.

We use entropy to measure the response concentration of relevance signals rather than directly equating it with semantic reliability. Lower entropy generally indicates stronger selectivity, with responses concentrated on fewer visual tokens, whereas higher entropy reflects a more uniform distribution. Such selectivity may help identify important visual information when the concentrated responses correspond to task-relevant regions. We fully agree that low entropy does not guarantee semantic correctness. Following your suggestion, we conducted additional experiments to evaluate the effectiveness of adaptive fusion.

Different tasks favor different values of β\beta. As shown in Table 5, we extended the search range of the fixed fusion coefficient β\beta from 0.8–1.2 to 0.1–10. Under the 64-token budget, GQA and POPE achieve their best results at β=0.1\beta=0.1 and β=0.25\beta=0.25, favoring centered similarity over attention. In contrast, TextVQA performs best at β=4\beta=4, placing greater weight on attention. As β\beta increases from 0.1 to 4, TextVQA improves from 58.7 to 66.3, while the POPE F1 score decreases from 85.1 to 81.3.

Table 5. Comparison of fixed β\beta over a wider range.

MethodRetained TokensMME (Total)GQAPOPE (F1 score)ScienceQAMMB ENENMMB CNCNTextVQADocVQAOCRBenchAI2DChartQAMMMUAvg.
Vanilla1,2962326.959.885.987.883.880.282.594.174583.080.550.7100.0%
β=0.12562299.058.386.585.581.177.777.373.563581.572.548.894.13%
β=0.252562308.958.386.585.780.977.877.973.864281.472.748.894.35%
β=0.52562323.458.586.285.780.577.278.774.364381.273.449.294.56%
β = 0.82562291.057.886.186.381.177.379.073.864581.673.548.294.35%
β = 1.02562305.457.886.486.281.077.579.373.664281.673.148.494.41%
β = 1.22562283.857.786.086.181.077.779.474.164881.773.148.494.42%
β=22562286.458.085.785.481.277.779.575.364281.573.449.294.59%
β=42562280.157.585.085.580.777.179.376.364881.273.749.294.46%
β=102562263.157.884.585.380.677.479.374.964581.673.549.194.24%
Ours2562294.158.486.385.980.877.479.575.164681.673.649.194.75%
β=0.11282250.157.786.182.578.074.670.255.052478.461.747.687.70%
β=0.251282242.757.685.983.277.875.171.155.152678.662.548.388.07%
β=0.51282237.357.685.383.177.774.272.355.953179.162.847.888.13%
β = 0.81282257.156.785.582.678.074.072.055.954078.362.947.588.01%
β = 1.01282254.256.585.282.478.174.872.756.054378.063.947.388.17%
β = 1.21282248.856.485.182.477.975.273.656.254178.364.047.288.26%
β=21282239.556.984.582.778.274.873.856.554578.964.147.588.46%
β=41282194.956.483.982.977.674.774.156.955179.464.647.688.38%
β=101282185.556.383.482.678.175.273.755.254279.263.748.088.01%
Ours1282246.457.285.582.777.874.973.556.755579.063.847.888.73%
β=0.1642146.355.885.181.576.671.858.738.138974.848.445.280.14%
β=0.25642154.855.285.481.176.371.159.537.540075.449.345.780.36%
β=0.5642132.055.484.881.676.471.261.638.441274.350.945.580.76%
β=0.8642118.254.483.781.175.970.661.537.139873.951.845.880.13%
β=1642148.454.683.481.475.970.962.537.640974.452.645.280.60%
β=1.2642144.954.383.081.675.471.063.337.341074.552.545.080.52%
β=2642174.154.282.781.476.071.564.438.541976.053.745.181.28%
β=4642098.654.181.381.275.471.666.339.342474.753.845.481.04%
β=10642041.653.579.181.275.871.065.936.842774.753.445.380.23%
Ours642165.855.384.081.176.571.564.538.942475.852.945.481.60%

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response to the Area Chair and All Reviewers

We sincerely thank the Area Chair and all reviewers for their detailed feedback, constructive suggestions and positive reassessments of our work. Following additional experiments and clarifications, the review scores increased from [3, 2, 3] to [3, 3, 3]. We are encouraged that our rebuttal has received positive feedback.

The reviewers generally expressed positive views of our paper. We appreciate their positive assessments of our work, as reflected in the following comments:

  • Reviewer 5Snt: "The design is simple and addresses a practical problem." "The experiments provide good support for the proposed combination."

  • Reviewer VQz6: "SQ-Pruner does not require additional training or learnable parameters." "The efficiency analysis also reports reduced total FLOPs and prefill latency, supporting the practical value."

  • Reviewer VbHp: "Complete and coherent framework design." "Strong empirical performance under aggressive compression." "Broad validation across models and modalities."

In response to the constructive suggestions, we have uploaded a revised manuscript with significant revisions marked in blue. The key improvements are summarized below:

  1. Long-form generation quality and efficiency. We added experiments on DetailCaps to evaluate long-form generation. At 64-token budget, SQ-Pruner achieves the highest CAPTURE and CIDEr scores among the compared pruning methods, while reducing end-to-end latency by 15.6% compared with the unpruned model with natural EOS termination. Under controlled conditions, prefill FLOPs are reduced by 63.5%. We also verified the attention backend and pruning implementation, confirming a 1.42× prefill speedup with Transformers and FlashAttention 2.

  2. Preservation of task-relevant visual information. We conducted experiments on RefCOCOg to evaluate whether the preserved visual information supports downstream localization. Across all three token budgets, SQ-Pruner achieves the highest validation and test [email protected] among the compared pruning methods, along with the best or tied-best mIoU.

  3. Controlled analysis of pixel-level saliency. We added comparisons involving multi-channel Token SR and computing SR before or after pooling the grayscale image to examine the effects of norm projection and spatial aggregation. The results show that removing norm projection does not close the performance gap between Token SR and Pixel SR. Furthermore, computing SR before pooling improves average performance by 4.3%, 3.6% and 2.8% at 256, 128 and 64 tokens, supporting the importance of extracting structural saliency before spatial aggregation.

  4. Effectiveness of adaptive fusion. We expanded the search range of the fixed fusion weight ββ to 0.1–10, added a random-weight baseline and analyzed the average adaptive weights across different tasks. The adaptive weights show trends consistent with task preferences. At all three token budgets, adaptive fusion achieves higher average performance than both the best fixed-weight and random-weight strategies.

  5. Generalization across diverse models. We have incorporated comprehensive experiments on InternVL3-8B, Qwen3-VL-8B, LLaVA-OneVision-1.5-8B and DeepSeek-VL2-small. SQ-Pruner achieves substantial gains over competing pruning methods across the evaluated benchmarks and token budgets. At the lowest token budget, SQ-Pruner improves average performance by 5.7%–8.0% over the corresponding strongest baseline.

  6. Clarification of contributions and key design. We further clarified how SQ-Pruner differs from DocPrune and explained how the two-stage design allocates separate token budgets to structural anchors and query-relevant information. We also explained how centering reduces the influence of shared feature components on similarity estimation. Our results show that two-stage selection outperforms either component alone, while centering improves average performance by 1.1%–2.7% over standard cosine similarity.

We once again thank the Area Chair and all reviewers for their valuable time and constructive feedback. The suggestions have helped us improve the manuscript and better demonstrate the value of SQ-Pruner in preserving visual information while improving inference efficiency without additional training. We believe these experiments and analyses have significantly strengthened the paper, and we hope they will be helpful in the final evaluation. We remain available to answer any further questions during the final discussion phase.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.