acceptodds
Under review as a conference paper at ICLR 2027

Borrowing Diffusion’s Gaze for VLM Visual Token Pruning

Abstract

Vision–language models (VLMs) encode images into many visual tokens, most of which are query-irrelevant yet computationally expensive. Existing methods perform one-shot or layer-wise pruning based on visual sensitivity, feature redundancy, or text–visual attention. However, these signals are largely derived from the target VLM itself and may be affected by structural biases such as attention sinks, limiting their reliability as indicators of query-relevant visual evidence. We introduce a Layer-0 pruning framework that draws on the strong text–image alignment of pretrained diffusion models to improve query-relevant visual evidence selection. Specifically, we distill diffusion text–image attention into DiffScorer, a lightweight scorer that provides an external signal for selecting query-relevant visual tokens before LLM prefill. To preserve this strong distilled prior while correcting its residual attention biases, counterfactual Swap-GRPO explores budget-preserving token exchanges around DiffScorer's deterministic selection and optimizes the swap trajectories with answer rewards from a frozen VLM. Only DiffScorer is updated in both stages. Extensive experiments across VLM architectures, pruning ratios, and visual understanding benchmarks demonstrate that DiffScorer consistently outperforms competitive one-shot, layer-wise, and learned pruning methods. Notably, for high-resolution inputs containing over 10K visual tokens, DiffScorer achieves a improvement in end-to-end generation throughput while preserving 99.5% of the unpruned accuracy.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Model, Visual Token Pruning, Diffusion Model, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JJxwOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a method named DiffScorer, which uses diffusion attention to supervise Layer-0 visual token pruning. The proposed method then uses Swap-GRPO to improve token selection based on answer rewards. Experiments are conducted on Qwen3-VL and LLaVA to evaluate the performance of DiffScorer compared to baselines. The results show that the method retains more of the original accuracy after pruning and improves throughput on high-resolution inputs.

Critical strengths
  • The idea of using diffusion attention to supervise early visual token selection is well-motivated and interesting. The proposed scorer outperforms several existing methods even without RL training.

  • The proposed method shows good performance on high-resolution inputs. On HR-Bench 8K, it achieves 1.50 times the end-to-end throughput while retaining 99.5% of the accuracy of the model without pruning.

Critical weaknesses
  • It is not fully clear if the performance gains come from using diffusion models for supervision. The Box+SAM baseline uses the same training data and scorer architecture. However, its supervision differs from diffusion attention in how much contextual information it includes and how it uses the query. Therefore, the improvement may come from more information in the grounding supervision. It does not clearly show the contribution of using diffusion models.

  • The performance gains from Swap-GRPO are small. Table 5 shows 98.04% average performance retention, compared to 97.40% for GRIP-VLM-style rollouts. The paper does not show results with different training seeds.

  • The performance gains do not consistently hold across different VLM architectures. On LLaVA-1.5-7B, the average performance retention only improves by 0.06 to 0.48 percentage points over VLM-Pruner across the three pruning ratios. At 90% pruning, DiffScorer also performs worse on GQA, with a score of 58.31 compared to 60.39 for VLM-Pruner.

Questions for authors
  • Can the authors compare diffusion supervision with a strong query-conditioned grounding model using the same scorer and training data? This would help show the contribution of using diffusion models for supervision.

  • Can the authors provide results with different training seeds to show if Swap-GRPO consistently outperforms GRIP-VLM-style rollouts? It would be helpful to show how much the results change for the full method and the two RL ablations.

  • Why are the performance gains over VLM-Pruner much smaller on LLaVA than on Qwen3-VL? Can the authors clarify if diffusion supervision works differently with different visual encoders or token representations?

5 comments
AuthorsOpenReview ↗

Response to Reviewer JJxw — Reply 1/2

We thank the reviewer for the constructive feedback. We address the supervision control with language-guided SAM 3, evaluate rollout strategies across three training seeds, and examine the smaller gains on LLaVA through a representation-level diagnostic.

Q1. Comparison with a strong query-conditioned grounding teacher

We add SAM 3 as a strong language-guided grounding teacher, using evidence keywords extracted from the same question–answer pairs, together with their corresponding bounding boxes, to generate masks. These are converted into teacher targets using the same procedure as the original Box+SAM control.

We keep the training data, scorer architecture, SFT objective, optimization settings, and training budget fixed. Both variants use SFT only, without RL.

Teacher / trainingGQAMMBMMEPOPESQATextVQASEED-ImageA-OKVQAAvg. retention
Unpruned Qwen3-VL-8B64.1289.74237488.9194.4983.3078.8488.03100.00%
Retain 30% tokens (70% pruning)—————————
Keywords+Box+SAM 3 (SFT)58.3486.12219487.4689.3475.4575.5683.4994.19%
Diffusion (SFT)62.0087.13229288.3790.0879.9777.0785.4196.98%
Retain 20% tokens (80% pruning)—————————
Keywords+Box+SAM 3 (SFT)56.1684.08216687.0386.9668.2472.5882.1091.21%
Diffusion (SFT)60.5585.59222487.7288.4577.5275.0984.1094.95%
Retain 10% tokens (90% pruning)—————————
Keywords+Box+SAM 3 (SFT)51.1777.75193885.8384.8862.7467.3375.8185.16%
Diffusion (SFT)57.3182.63209586.8184.4372.4572.2979.8390.76%

Diffusion supervision exceeds this stronger control by 2.79, 3.74, and 5.60 percentage points in average retention. Its advantage therefore persists against language-guided, query-derived grounding under matched downstream training. This strengthens the evidence beyond the original geometric control.

AuthorsOpenReview ↗

# Response to Reviewer JJxw — Reply 2/2

Q2. Multi-seed stability of Swap-GRPO

We repeat the rollout comparison with training seeds 42, 43, and 44 on Qwen3-VL-8B at 70% pruning, keeping the scorer architecture, data, and training budget matched. Full per-benchmark results are shown below; the final two rows report mean ± sample standard deviation across the three seeds.

SeedRollout strategyGQAMMBMMEPOPESQATextVQASEED-ImageA-OKVQAAvg. retention
—Unpruned Qwen3-VL-8B64.1289.74237488.9194.4983.3078.8488.03100.00%
42GRIP-VLM-style62.3788.08229088.6390.1380.7277.0385.9497.40%
42Swap-GRPO (full)62.8688.61230488.9891.1381.4877.4786.2098.04%
43GRIP-VLM-style61.8388.102294.2488.4290.6380.9776.7485.1597.23%
43Swap-GRPO (full)62.9288.472319.4388.5691.9280.7175.8286.1197.77%
44GRIP-VLM-style62.9587.942294.0688.7889.3980.8077.2885.3397.40%
44Swap-GRPO (full)62.2587.942287.7188.9691.3280.7777.4686.8197.75%
Mean ± SDGRIP-VLM-style62.38 ± 0.5688.04 ± 0.092292.77 ± 2.4088.61 ± 0.1890.05 ± 0.6280.83 ± 0.1377.02 ± 0.2785.47 ± 0.4197.34% ± 0.10 pp
Mean ± SDSwap-GRPO (full)62.68 ± 0.3788.34 ± 0.352303.71 ± 15.8688.83 ± 0.2491.46 ± 0.4180.99 ± 0.4376.92 ± 0.9586.37 ± 0.3897.85% ± 0.16 pp

Swap-GRPO achieves higher average performance retention in all three tested seeds.

Q3. Why are the gains smaller on LLaVA?

This is an insightful question. We agree that the gains are architecture-dependent and not uniform across benchmarks: at 90% pruning, our LLaVA GQA score is 58.31 versus 60.39 for VLM-Pruner. The aggregate improvements on LLaVA remain positive but small—0.37, 0.06, and 0.48 percentage points at the three pruning ratios.

The stronger retention on LLaVA is also observed for several baselines. At 90% pruning, the original tables give the following average retention relative to each model's own unpruned performance:

MethodLLaVA-1.5-7BQwen3-VL-8B
ZOO-Prune95.04%83.98%
OTPrune92.07%84.87%
VLM-Pruner95.09%89.61%
Ours95.57%92.04%

To investigate this pattern, we analyze an outcome-independent subset of 96 images from all benchmarks. For each model and image, we extract the projected main visual stream before pruning and LLM processing, then select 256 evenly spaced token positions. After L2 normalization, we measure whether each token has another sampled token from the same image with cosine similarity strictly above 0.95, excluding self-matches.

Probe statisticLLaVA-1.5-7BQwen3-VL-8B
Tokens with a high-similarity sampled neighbor39.58%0.52%
Mean nearest-neighbor cosine similarity0.90550.7623

The probe indicates a much higher incidence of near-parallel projected features on LLaVA under this protocol. Related work also studies architecture-dependent sparsity and redundancy in projected visual representations [1].

A plausible interpretation is that strong pruning baselines may already exploit more of the available redundancy on LLaVA, leaving less room for gains from finer-grained selection. We treat this as a hypothesis supported by the diagnostic pattern, and we believe this direction bears significant research merit for future exploration.

[1] Fan et al. What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models.

Dear Reviewers,

As the discussion phase deadline approaches, we kindly check if our submitted responses and revised manuscript have addressed your concerns, or if you have any remaining questions we could clarify before the end of the discussion period.

Best regards, The Authors

Reviewer_LDwaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DiffScorer, a query-conditioned scorer for visual token pruning. It distills text–image attention from a pretrained diffusion model, then uses Swap-GRPO to refine token selection through budget-preserving exchanges and answer rewards from a frozen VLM. Experiments on two VLM architectures and multiple benchmarks demonstrate its effectiveness.

Critical strengths
  1. The paper introduces a useful external supervision source for visual token selection. Distilling diffusion attention into a small scorer allows the method to benefit from text–image alignment.
  2. Swap-GRPO provides a reasonable way to refine a strong initial selection. Local token exchanges preserve the exact budget, allowing alternatives to be compared without changing the image, question, or retained token count.
  3. The evaluation covers multiple benchmarks and token budgets, with supervision-source comparisons, RL ablations, and end-to-end throughput measurements. These experiments provide evidence for the method’s effectiveness.
Critical weaknesses
  1. The specific advantage of diffusion supervision needs stronger validation. The Box+SAM control uses the original SAM to construct supervision targets. Would replacing it with a stronger model such as SAM 3 improve this baseline and reduce the gap, under the same data and training settings?
  2. Lack of experimental comparisons with several related methods, such as VisionZip [1], PyramidDrop [2], FlowCut [3], HoloV [4], and DART [5].
  3. Adaptation across model architectures and sizes is insufficiently clarified. DiffScorer uses a fixed 4096-dimensional input, and both evaluated backbones share this feature width. For models with different projected feature dimensions, does adaptation require retraining the entire scorer? The paper should also clarify whether the reported cross-architecture results reuse trained weights or involve separate training. Moreover, experiments on different model sizes would further help assess the method’s applicability.
  4. A native resolution-reduction baseline is missing. For the Qwen-VL family, visual token counts can already be reduced by lowering max_pixels. Under the same visual token budget entering the LLM, how does DiffScorer compare with this native approach in accuracy and end-to-end latency?

[1] VisionZip: Longer is Better but Not Necessary in Vision Language Models (CVPR 2025)

[2] PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction (CVPR 2025)

[3] FlowCut: Rethinking Redundancy via Information Flow for Efficient Vision-Language Models (NeurIPS 2025)

[4] Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retention (NeurIPS 2025)

[5] Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters More (EMNLP 2025)

Questions for authors

see weakness

8 comments
AuthorsOpenReview ↗

Response to Reviewer LDwa — Reply 1/4

We thank the reviewer for the constructive suggestions. We provide a stronger SAM 3 teacher control, additional pruning baselines, results on a different model size and feature width, and a token-budget-matched resolution-reduction comparison.

Q1. Stronger segmentation supervision with SAM 3

We replace the original SAM with SAM 3, prompting it with the extracted semantic evidence keywords and bounding boxes. The resulting masks are converted into supervision targets using the same procedure as the original Box+SAM control. We keep the training data, scorer architecture, SFT objective, optimization settings, and training budget identical to diffusion distillation. Both variants use SFT only, without RL.

Results on Qwen3-VL-8B are shown below.

Teacher / trainingGQAMMBMMEPOPESQATextVQASEED-ImageA-OKVQAAvg.
Unpruned Qwen3-VL-8B64.1289.74237488.9194.4983.3078.8488.03100.00%
Retain 30% tokens (70% pruning)—————————
Keywords+Box+SAM 3 (SFT)58.3486.12219487.4689.3475.4575.5683.4994.19%
Diffusion (SFT)62.0087.13229288.3790.0879.9777.0785.4196.98%
Retain 20% tokens (80% pruning)—————————
Keywords+Box+SAM 3 (SFT)56.1684.08216687.0386.9668.2472.5882.1091.21%
Diffusion (SFT)60.5585.59222487.7288.4577.5275.0984.1094.95%
Retain 10% tokens (90% pruning)—————————
Keywords+Box+SAM 3 (SFT)51.1777.75193885.8384.8862.7467.3375.8185.16%
Diffusion (SFT)57.3182.63209586.8184.4372.4572.2979.8390.76%

Diffusion supervision retains an advantage of 2.79, 3.74, and 5.60 percentage points at 70%, 80%, and 90% pruning, respectively. Thus, the advantage persists with a stronger, semantically prompted segmentation teacher under matched data and training, rather than depending on the original SAM baseline.

AuthorsOpenReview ↗

Response to Reviewer LDwa — Reply 2/4

Q2. Additional comparisons with related pruning methods

Among the five suggested methods, we add three complementary representatives under a unified Qwen3-VL evaluation protocol:

  • VisionZip: question-independent compression on the vision side.
  • PyramidDrop: progressive pruning within the decoder.
  • DART: token selection based on representation redundancy.

Following PyramidDrop's progressive pruning scheme, we prune at Layers 5, 10, and 15, with equal multiplicative reductions to reach the target final retention ratio rr:

N→Nr1/3→Nr2/3→Nr.N \rightarrow N r^{1/3} \rightarrow N r^{2/3} \rightarrow Nr.

All methods use the same evaluation protocol and final token budgets.

MethodGQAMMBMMEPOPESQATextVQASEED-ImageA-OKVQAAvg.
Unpruned Qwen3-VL-8B64.1289.74237488.9194.4983.3078.8488.03100.00%
Retain 30% tokens (70% pruning)—————————
VisionZip60.9786.60219388.1688.7077.5176.3384.8095.40%
PyramidDrop58.0384.96201787.1186.2272.1573.8881.5791.54%
DART58.9086.95212486.5489.1465.5473.7485.2492.37%
Ours (SFT+RL)62.8688.61230488.9891.1381.4877.4786.2098.04%
Retain 20% tokens (80% pruning)—————————
VisionZip58.2984.73199786.5487.2669.4174.8181.9291.30%
PyramidDrop54.6781.45190883.4281.5663.4670.5578.5286.43%
DART55.8284.52196884.0387.9556.9370.6082.0187.85%
Ours (SFT+RL)61.0987.41223388.5188.0175.7275.6684.6395.30%
Retain 10% tokens (90% pruning)—————————
VisionZip53.5480.53179580.7782.0043.5670.8075.8181.84%
PyramidDrop47.9675.68168473.4878.0449.3560.0271.0076.42%
DART50.4076.97174679.3184.3842.6266.1878.3480.07%
Ours (SFT+RL)58.5283.76216987.8385.0271.8873.5880.9692.04%

Our method exceeds the strongest of these additional baselines by 2.64, 4.00, and 10.20 percentage points in average retention across the three budgets. The advantage is largest under aggressive pruning.

AuthorsOpenReview ↗

Response to Reviewer LDwa — Reply 3/4

Q3. Adaptation across feature dimensions, architectures, and model sizes

The cross-architecture results use separately trained scorers, not reused weights. The 4096-dimensional input is specific to the backbones originally evaluated, not a fixed requirement of DiffScorer. For another feature width, we adjust the input projection and train the entire lightweight scorer, while keeping the VLM frozen. This is backbone-specific training, as in the learned pruning baselines evaluated here, rather than zero-shot scorer transfer.

We additionally evaluate Qwen3-VL-4B with a 2560-dimensional scorer input. These results use SFT only, without RL, and are normalized against the unpruned 4B model.

MethodGQAMMBMMEPOPESQATextVQASEED-ImageA-OKVQAAvg.
Unpruned Qwen3-VL-4B64.8088.45234389.7992.4182.0878.8785.24100.00%
Retain 30% tokens (70% pruning)—————————
DART61.0286.00214088.5886.9164.8674.7383.8493.45%
VisionZip61.5183.58204488.2286.8170.2275.8882.9793.49%
VLM-Pruner63.7186.49224988.9187.0176.7775.7883.7696.64%
PyramidDrop56.3082.88194484.3483.0466.2471.0778.5288.78%
Ours (SFT)62.7787.39221989.6388.1579.9777.8883.4997.46%
Retain 20% tokens (80% pruning)—————————
DART58.4883.69199385.2784.6855.2872.1681.8388.92%
VisionZip58.3283.58184784.2283.7457.4273.3880.7088.18%
VLM-Pruner62.4385.40215087.6185.3273.9774.0783.1494.52%
PyramidDrop52.5479.28175579.6280.4755.6765.4173.9782.36%
Ours (SFT)61.6886.72218988.7387.6173.0376.7382.0195.34%
Retain 10% tokens (90% pruning)—————————
DART52.1978.84175980.0281.5142.9366.6377.9081.28%
VisionZip54.5579.95170478.7979.0336.9170.4775.2080.39%
VLM-Pruner57.6081.24193282.1980.2664.3770.3180.2687.91%
PyramidDrop45.8070.59141870.7676.2039.1855.4167.2571.14%
Ours (SFT)59.4784.66197587.8678.6965.1374.1978.6890.06%

DiffScorer achieves 97.46%, 95.34%, and 90.06% average retention, leading the compared methods at all three budgets despite using neither the 4096-dimensional input nor RL refinement.

Adaptation cost is modest. As a reference, the measured training times for the 8B model on 8 × A100 80GB GPUs are:

StageWithout cached visual featuresWith cached visual features
SFTApproximately 60 minApproximately 20 min
RLApproximately 40 minApproximately 15 min
TotalApproximately 100 minApproximately 35 min
Reviewer_kCZtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DiffScorer, a lightweight scorer that distills text–image attention from a pretrained diffusion model to prune visual tokens at LLM layer 0. The scorer is subsequently trained with Swap-GRPO, which optimizes local token replacements under a fixed token budget using downstream answer rewards. The VLM remains frozen throughout training. Experiments on 2 VLMs, multiple benchmarks, and several pruning budgets show acceleration while preserving the answer quality.

Critical strengths
  1. Using diffusion-based guidance is a promising and interesting direction. The teacher is used offline to train the scorer and is not used during VLM inference.

  2. Results span 2 VLMs, diverse vision-language benchmarks, and multiple retention ratios, demonstrating generally good performance.

  3. The ablation study provides useful evidence to explain the contribution of individual components.

Critical weaknesses
  1. It is unclear that the advantage of diffusion-based supervision is significant compared with other supervision signals. The paper lacks a controlled experiment that replaces diffusion attention with target-VLM attention while keeping the scorer, training data, and optimization procedure fixed. The Box+SAM comparison supports an advantage over the geometric supervision, but does not show advantage over supervision from the VLM itself. Given that the cited MAP method already distills middle layer attention, this comparison is particularly relevant to support the main contribution.

  2. I understand that adding a little noise (3 step adding noise) for diffusion and performing one step of forward denoising are aligned with intuition, but it is unclear that the paper doesn’t have any study on different levels of noise and scheduler configurations. In another word, experiments across noise levels are missing, particularly for tasks involving text, small objects, and fine visual details.

  3. The efficiency evaluation in Table 4 and 13 only reports high resolution results, leaving the benefits of standard and low resolution tasks unclear. Under these settings, the scorer and post-processing overhead may be more significant.

  4. The paper only reports the efficiency advantage compared with training-free methods and lack on learned methods in Table 5 and 11.

Minor Weakness

  1. Figure 1 is not readable since several labels are difficult to read at normal size and the two radar charts lack legends to identify each method.

  2. Table 1 is labeled as a Layer-0 comparison, whereas Appendix C.4 states that VLM-Pruner prunes at Layer 2. Please clarify whether its original pruning position was preserved and revise the table title or description.

  3. Table 10 shows similar performance for both directions. Since the preference for Text-Q/Vision-KV is motivated by lower scoring-head overhead, a direct comparison of scorer latency, FLOPs, or memory is needed.

Questions for authors
  1. How does target-VLM attention or MAP-style supervision perform with the same data, scorer and training budget?

  2. How do different noise levels affect the supervision maps and downstream performance?

  3. Are separate DiffScorers trained for different VLMs? If so, what adaptation steps and training costs are needed for a new VLM?

  4. Could the authors provide matched runtime comparisons at standard and lower resolutions and against learned pruning baselines? End-to-end latency, time to first token, and scorer+post-processing overhead would also be useful.

The key comparison needed to establish the advantage of the diffusion teacher is missing, and the scope of the efficiency benefits remains insufficiently characterized. These are the main reasons for my current weak rejection. Convincing teacher controls and more complete runtime comparisons could move my assessment toward weak acceptance.

7 comments
AuthorsOpenReview ↗

Response to Reviewer kCZt — Reply 1/3

We thank the reviewer for the constructive feedback and for identifying two important aspects requiring stronger evidence: isolating the contribution of diffusion supervision and characterizing efficiency beyond high-resolution inputs. We address these concerns with controlled teacher comparisons, noise-level ablations, and matched runtime measurements covering both learned and training-free pruning methods.

Q1. Controlled comparisons with target-VLM and MAP-style supervision

We compare four teachers using the same scorer, training data, SFT objective, optimization schedule, and training budget. All teacher controls use SFT only.

  • Target-VLM attention: We use Layer 16 (middle) and Layer 28 (deep) of Qwen3-VL-8B. Given the image, question, and ground-truth answer, we extract question-to-vision attention for the evidence-keyword spans, average over heads and tokens within each span, and apply the same normalization and spatial-confidence-weighted keyword merging as for diffusion supervision.
  • MAP-style attention: We pair each image with the actual question and the generic question, “What is shown in this image?” We extract head-averaged, normalized text-to-vision distributions from all 36 layers, compute the Jensen–Shannon divergence between the two distributions at each layer, and select the layer with the largest divergence. The remaining supervision processing and scorer training are unchanged.
Teacher / trainingGQAMMBMMEPOPESQATextVQASEED-ImageA-OKVQAAvg. retention
Unpruned Qwen3-VL-8B64.1289.74237488.9194.4983.3078.8488.03100.00%
Retain 30% tokens (70% pruning)—————————
VLM-L28 (SFT)56.3084.66197582.1882.1560.9069.9680.5287.25%
VLM-L16 (SFT)60.7886.32206089.3889.2479.1275.7685.1595.07%
MAP-style (SFT)58.2785.59204888.6386.3777.3974.3683.6793.24%
Diffusion (SFT)62.0087.13229288.3790.0879.9777.0785.4196.98%
Retain 20% tokens (80% pruning)—————————
VLM-L28 (SFT)54.0981.82187279.1180.5251.7067.8877.8283.14%
VLM-L16 (SFT)58.1984.43203788.5987.3171.5572.3083.2391.85%
MAP-style (SFT)55.2083.78195687.6684.8369.1470.1782.4589.48%
Diffusion (SFT)60.5585.59222487.7288.4577.5275.0984.1094.95%
Retain 10% tokens (90% pruning)—————————
VLM-L28 (SFT)50.3075.26161774.7375.7637.8863.6472.4075.39%
VLM-L16 (SFT)56.0381.27195588.4485.4267.1269.8379.8388.75%
MAP-style (SFT)51.4780.30191187.0182.3064.9067.6779.3986.14%
Diffusion (SFT)57.3182.63209586.8184.4372.4572.2979.8390.76%

Diffusion-SFT exceeds the strongest tested VLM attention teacher, Layer 16, by 1.91, 3.10, and 2.01 percentage points in average retention at 70%, 80%, and 90% pruning, respectively. Its consistent advantage over both fixed-layer and MAP-style teachers supports the diffusion prior independently of RL refinement.

AuthorsOpenReview ↗

Response to Reviewer kCZt — Reply 2/3

Q2. Sensitivity to diffusion noise levels

We evaluate 1, 3, 10, and 20 noising steps at 70%, 80%, and 90% pruning, using SFT only. Every setting uses one denoising forward pass to extract teacher attention; the step count specifies the input corruption level, not the number of denoising passes.

Representative results at 90% pruning are shown below. General Avg. covers the eight general benchmarks; the remaining columns report accuracy.

Noise levelGeneral Avg.TextVQAHR-Bench 4KHR-Bench 8KV*
1 step90.83%71.8580.3869.5079.58
3 steps90.76%72.4578.1372.1379.06
10 steps90.68%72.4974.6365.0066.49
20 steps84.05%50.7666.2559.2548.17
  • 1 vs. 3 steps: Aggregate performance is comparable across pruning budgets, supporting a low-noise regime rather than a uniquely optimal timestep.
  • 10 steps: General-task performance remains similar, but fine-grained understanding deteriorates. At 90% pruning, HR-Bench 4K, HR-Bench 8K, and V* drop by 3.50, 7.13, and 12.57 accuracy points relative to three steps.
  • 20 steps: Both general and fine-grained performance decline substantially, including TextVQA. This is consistent with excessive corruption weakening the teacher's alignment with the original image.

Q3. Backbone-specific adaptation and training cost

Yes, each target VLM has a separately trained DiffScorer. Adaptation consists of extracting frozen Layer-0 features, aligning teacher maps to the visual-token layout, matching the scorer's input interface, and running SFT followed by Swap-GRPO. Only the approximately 26.25M-parameter scorer is updated; the VLM and diffusion teacher remain frozen. Backbone-specific adaptation is also required by the learned pruning baselines evaluated here.

For the 8B model on 8 × A100 80GB GPUs, caching frozen visual features substantially reduces training time:

StageWithout cached featuresWith cached features
SFTApproximately 60 minApproximately 20 min
RLApproximately 40 minApproximately 15 min
TotalApproximately 100 minApproximately 35 min
AuthorsOpenReview ↗

Response to Reviewer kCZt — Reply 3/3

Q4. Matched runtime across workloads and learned baselines

All methods use Qwen3-VL-8B, 10% visual-token retention, batch size one, and matched hardware and generation settings. Workload groups are based on benchmark-level median visual-token counts: low (<256; MMBench and ScienceQA), standard (the remaining conventional VQA group), and high (HR-Bench 4K).

E2E, TTFT, and prefill are speedups over the unpruned model; scorer and post-processing costs are in milliseconds. E2E includes all inference components from visual encoding through decoding, excluding model/data loading.

RegimeMethodE2E ↑TTFT ↑Prefill ↑Scorer (ms) ↓Post-process (ms) ↓
LowVLM-Pruner1.041×1.042×1.068×—1.79
LowOTPrune1.055×1.045×1.121×—1.31
LowVisionSelector1.105×1.094×1.156×0.140.45
LowDynamic-LLaVA1.005×1.055×1.084×0.850.57
LowTwigVLM1.055×1.019×1.030×3.020.47
LowOurs1.080×1.073×1.094×0.950.70
StandardVLM-Pruner1.068×1.268×1.727×—3.87
StandardOTPrune1.082×1.287×1.788×—5.04
StandardVisionSelector1.092×1.356×2.051×0.210.43
StandardDynamic-LLaVA1.087×1.308×1.862×0.820.55
StandardTwigVLM1.062×1.237×1.572×7.460.46
StandardOurs1.100×1.311×1.907×0.931.15
HighVLM-Pruner1.372×1.344×4.885×—125.63
HighOTPrune1.323×1.302×3.515×—370.22
HighVisionSelector1.451×1.424×12.164×8.100.65
HighDynamic-LLaVA1.410×1.384×7.113×8.941.11
HighTwigVLM1.251×1.226×2.326×521.660.73
HighOurs1.443×1.414×10.482×12.6618.62

Efficiency gains depend on workload. Our E2E speedups are 1.080×, 1.100×, and 1.443×, with scorer-plus-post-processing costs of 1.65, 2.08, and 31.28 ms. Low-token inputs offer less computation to remove, so the benefit is smaller. VisionSelector is faster on several metrics, but at 10% retention its average general-benchmark performance retention is 85.97%, versus 90.76% for diffusion-SFT and 92.04% for our complete method.

The component differences follow from pruning position and implementation cost:

  • Dynamic-LLaVA: Like VLM-Pruner, it retains its original Layer-2 pruning position, paying for two full-token LLM blocks. Our Layer-0 pruning avoids that cost, explaining why its scorer/post-processing can be cheaper while its prefill is slower.
  • OTPrune: The evaluated implementation constructs an N×NN\times N similarity matrix and multiplies matrices to form the kernel, introducing an O(N3)O(N^3) term, followed by greedy selection and repeated CPU–GPU synchronization.
  • VLM-Pruner: Post-processing constructs N2N^2-scale semantic-similarity and spatial-distance matrices, then performs buffered selection, sorting, and discarded-token feature aggregation.
  • TwigVLM: Its scorer is a full decoder block over visual and text tokens, with quadratic self-attention cost plus FFN and projection overhead, rather than a simple per-token scoring head.

We will extend Appendix C.4 to distinguish these selection costs from prefill savings and end-to-end speedups.

Minor comments

Figure 1 readability

We will enlarge the labels, improve the layout at normal viewing size, and add explicit method legends to both radar charts.

Table 1 and VLM-Pruner's pruning position

VLM-Pruner preserves its original Layer-2 pruning position. We will replace “Layer-0 comparison” with “Comparison of one-shot visual token pruning before or within early LLM layers”, and explicitly identify each method's pruning position in the caption.

Cost of the two scorer directions

We profiled the two implementations in Table 10. Representative results with 32 text tokens are:

Visual tokensDirectionGFLOPs ↓Incremental peak memory (MiB) ↓Scorer latency (ms) ↓
256Text-Q / Vision-KV5.509.560.974 ± 0.076
256Vision-Q / Text-KV9.1910.690.626 ± 0.043
2,048Text-Q / Vision-KV36.0429.320.945 ± 0.028
2,048Vision-Q / Text-KV69.8049.190.662 ± 0.117

At 2,048 visual tokens, Text-Q/Vision-KV uses 48.37% fewer FLOPs and 40.39% less incremental peak memory, but its measured latency is not lower. We will therefore state the advantage as lower arithmetic and memory costs.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall Author Rebuttal

Dear Area Chair and all Reviewers,

We sincerely thank all reviewers for their constructive feedback, which has greatly helped us improve the manuscript and strengthen empirical evidence. Below we summarize recognized strengths, core concerns, and corresponding revisions; full point-by-point responses follow.


1. Summary of Recognized Strengths

All reviewers acknowledge the value of this work, with consistent recognition on:

  • Novel technical direction: kCZt, LDwa, and JJxw all agree that using pretrained diffusion cross-modal attention as external supervision for Layer-0 visual token pruning is a promising, well-motivated direction for efficient VLM inference.
  • Sound Swap-GRPO design: LDwa and JJxw highlight that counterfactual Swap-GRPO enables elegant budget-preserving local token swaps for controlled policy refinement under fixed token budgets.
  • Comprehensive evaluation: kCZt and LDwa note that experiments cover multiple VLM architectures, benchmarks, and retention ratios, with informative ablations validating each component.
  • Strong high-resolution performance: JJxw specifically points out the favorable accuracy-efficiency trade-off: 1.50× end-to-end throughput with 99.5% accuracy retention on HR-Bench 8K.

2. Summary of Concerns & Corresponding Revisions

We systematically organize all concerns into four categories, with targeted experiments and revisions summarized below.

Category 1: Diffusion supervision validation (kCZt, LDwa, JJxw)

All three reviewers request stronger controlled comparisons to isolate the diffusion prior’s unique contribution.

  • VLM-internal / MAP-style controls (kCZt): With matched scorer, data, and training settings, diffusion supervision outperforms VLM-L16, VLM-L28, and MAP-style attention by 1.91–3.10 pp in average retention across all pruning ratios.
  • Stronger segmentation baseline (LDwa, JJxw): We add Keywords+Box+SAM 3 as a query-conditioned grounding teacher under matched settings. Diffusion supervision still holds a 2.79–5.60 pp advantage, confirming its unique value beyond strong segmentation models.
  • Manuscript location: Section 4.3, Appendix B.1
Category 2: Robustness & cross-architecture adaptation (kCZt, LDwa, JJxw)
  • Noise-level ablation (kCZt): We test 1/3/10/20 noising steps across all ratios. Low noise yields best fine-grained performance; excessive noise substantially degrades text and high-resolution task accuracy.
  • Multi-seed RL stability (JJxw): Across 3 training seeds (42/43/44), Swap-GRPO consistently outperforms GRIP-VLM-style rollouts, with a mean 0.51 pp advantage.
  • Cross-architecture adaptation (LDwa, JJxw): Each VLM uses a separately trained lightweight scorer (≈26.25M params), with only the input projection adjusted. We add Qwen3-VL-4B (2560-dim) results, confirming consistency across model sizes.
  • Smaller LLaVA gains (JJxw): Representation diagnostics show LLaVA has higher intrinsic visual token redundancy than Qwen3-VL, reducing headroom for fine-grained selection improvements.
  • Manuscript location: Section 4.3, Appendices B.5, B.6, C.7
Category 3: Baseline & efficiency completeness (kCZt, LDwa)
  • Additional baselines (LDwa): We add VisionZip, PyramidDrop, HoloV, FlowCut and DART under unified protocols. Our method leads the strongest baseline by 2.64–10.20 pp, with gaps widening at aggressive pruning ratios.
  • Resolution-reduction baseline (LDwa): At 90% pruning, DiffScorer outperforms Resolution+Uniform by 13.47 pp on TextVQA and 26.70 pp on V*. We also match both methods at a fixed 64 visual tokens, which native resizing reaches at the configured minimum resolution. DiffScorer improves mean performance retention over this resize-only baseline by 13.51 pp on fine-grained benchmarks.
  • Full efficiency breakdown (kCZt): We report E2E throughput, TTFT, prefill speedup, and scorer+post-processing latency across low/standard/high-token workloads, covering both training-free and learned methods.
  • Manuscript location: Section 4.2, 4.4, Appendices C.4, C.6
Category 4: Presentation & clarity (kCZt)
  • Improved Figure 1 readability with larger labels, optimized layout, and explicit radar chart legends.
  • Revised Table 1 title/caption to clarify each method’s pruning position, noting VLM-Pruner retains its native Layer-2 position.
  • Added FLOPs, memory, and latency comparisons for the two scorer attention directions in Appendix B.4.

All revisions are marked in blue in the updated manuscript. We provide point-by-point responses to each reviewer's comments below, and we hope our revisions and responses fully address all concerns, and we look forward to further discussing the paper with you. Should you request additional experiments or analyses, please do not hesitate to let us know. We will do our best to deliver results promptly.

Best regards, The Authors

Dear Reviewers,

The authors have posted their rebuttals and updated their manuscript/responses. Please carefully read the authors' responses and engage in the discussion to clarify any remaining concerns.

In light of the rebuttal, please let the authors know whether your concerns have been adequately addressed, and update your review comments and scores if appropriate. Active participation is essential for us to reach a well-calibrated decision.

Please note that the author-reviewer discussion phase ends on November 18 at 11:59 PM AoE, after which no further responses can be posted. Thank you for your continued time and effort.

Best regards,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.