- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a method named DiffScorer, which uses diffusion attention to supervise Layer-0 visual token pruning. The proposed method then uses Swap-GRPO to improve token selection based on answer rewards. Experiments are conducted on Qwen3-VL and LLaVA to evaluate the performance of DiffScorer compared to baselines. The results show that the method retains more of the original accuracy after pruning and improves throughput on high-resolution inputs.
Critical strengths
-
The idea of using diffusion attention to supervise early visual token selection is well-motivated and interesting. The proposed scorer outperforms several existing methods even without RL training.
-
The proposed method shows good performance on high-resolution inputs. On HR-Bench 8K, it achieves 1.50 times the end-to-end throughput while retaining 99.5% of the accuracy of the model without pruning.
Critical weaknesses
-
It is not fully clear if the performance gains come from using diffusion models for supervision. The Box+SAM baseline uses the same training data and scorer architecture. However, its supervision differs from diffusion attention in how much contextual information it includes and how it uses the query. Therefore, the improvement may come from more information in the grounding supervision. It does not clearly show the contribution of using diffusion models.
-
The performance gains from Swap-GRPO are small. Table 5 shows 98.04% average performance retention, compared to 97.40% for GRIP-VLM-style rollouts. The paper does not show results with different training seeds.
-
The performance gains do not consistently hold across different VLM architectures. On LLaVA-1.5-7B, the average performance retention only improves by 0.06 to 0.48 percentage points over VLM-Pruner across the three pruning ratios. At 90% pruning, DiffScorer also performs worse on GQA, with a score of 58.31 compared to 60.39 for VLM-Pruner.
Questions for authors
-
Can the authors compare diffusion supervision with a strong query-conditioned grounding model using the same scorer and training data? This would help show the contribution of using diffusion models for supervision.
-
Can the authors provide results with different training seeds to show if Swap-GRPO consistently outperforms GRIP-VLM-style rollouts? It would be helpful to show how much the results change for the full method and the two RL ablations.
-
Why are the performance gains over VLM-Pruner much smaller on LLaVA than on Qwen3-VL? Can the authors clarify if diffusion supervision works differently with different visual encoders or token representations?
5 comments
# Response to Reviewer JJxw — Reply 2/2
Q2. Multi-seed stability of Swap-GRPO
We repeat the rollout comparison with training seeds 42, 43, and 44 on Qwen3-VL-8B at 70% pruning, keeping the scorer architecture, data, and training budget matched. Full per-benchmark results are shown below; the final two rows report mean ± sample standard deviation across the three seeds.
| Seed | Rollout strategy | GQA | MMB | MME | POPE | SQA | TextVQA | SEED-Image | A-OKVQA | Avg. retention |
|---|---|---|---|---|---|---|---|---|---|---|
| — | Unpruned Qwen3-VL-8B | 64.12 | 89.74 | 2374 | 88.91 | 94.49 | 83.30 | 78.84 | 88.03 | 100.00% |
| 42 | GRIP-VLM-style | 62.37 | 88.08 | 2290 | 88.63 | 90.13 | 80.72 | 77.03 | 85.94 | 97.40% |
| 42 | Swap-GRPO (full) | 62.86 | 88.61 | 2304 | 88.98 | 91.13 | 81.48 | 77.47 | 86.20 | 98.04% |
| 43 | GRIP-VLM-style | 61.83 | 88.10 | 2294.24 | 88.42 | 90.63 | 80.97 | 76.74 | 85.15 | 97.23% |
| 43 | Swap-GRPO (full) | 62.92 | 88.47 | 2319.43 | 88.56 | 91.92 | 80.71 | 75.82 | 86.11 | 97.77% |
| 44 | GRIP-VLM-style | 62.95 | 87.94 | 2294.06 | 88.78 | 89.39 | 80.80 | 77.28 | 85.33 | 97.40% |
| 44 | Swap-GRPO (full) | 62.25 | 87.94 | 2287.71 | 88.96 | 91.32 | 80.77 | 77.46 | 86.81 | 97.75% |
| Mean ± SD | GRIP-VLM-style | 62.38 ± 0.56 | 88.04 ± 0.09 | 2292.77 ± 2.40 | 88.61 ± 0.18 | 90.05 ± 0.62 | 80.83 ± 0.13 | 77.02 ± 0.27 | 85.47 ± 0.41 | 97.34% ± 0.10 pp |
| Mean ± SD | Swap-GRPO (full) | 62.68 ± 0.37 | 88.34 ± 0.35 | 2303.71 ± 15.86 | 88.83 ± 0.24 | 91.46 ± 0.41 | 80.99 ± 0.43 | 76.92 ± 0.95 | 86.37 ± 0.38 | 97.85% ± 0.16 pp |
Swap-GRPO achieves higher average performance retention in all three tested seeds.
Q3. Why are the gains smaller on LLaVA?
This is an insightful question. We agree that the gains are architecture-dependent and not uniform across benchmarks: at 90% pruning, our LLaVA GQA score is 58.31 versus 60.39 for VLM-Pruner. The aggregate improvements on LLaVA remain positive but small—0.37, 0.06, and 0.48 percentage points at the three pruning ratios.
The stronger retention on LLaVA is also observed for several baselines. At 90% pruning, the original tables give the following average retention relative to each model's own unpruned performance:
| Method | LLaVA-1.5-7B | Qwen3-VL-8B |
|---|---|---|
| ZOO-Prune | 95.04% | 83.98% |
| OTPrune | 92.07% | 84.87% |
| VLM-Pruner | 95.09% | 89.61% |
| Ours | 95.57% | 92.04% |
To investigate this pattern, we analyze an outcome-independent subset of 96 images from all benchmarks. For each model and image, we extract the projected main visual stream before pruning and LLM processing, then select 256 evenly spaced token positions. After L2 normalization, we measure whether each token has another sampled token from the same image with cosine similarity strictly above 0.95, excluding self-matches.
| Probe statistic | LLaVA-1.5-7B | Qwen3-VL-8B |
|---|---|---|
| Tokens with a high-similarity sampled neighbor | 39.58% | 0.52% |
| Mean nearest-neighbor cosine similarity | 0.9055 | 0.7623 |
The probe indicates a much higher incidence of near-parallel projected features on LLaVA under this protocol. Related work also studies architecture-dependent sparsity and redundancy in projected visual representations [1].
A plausible interpretation is that strong pruning baselines may already exploit more of the available redundancy on LLaVA, leaving less room for gains from finer-grained selection. We treat this as a hypothesis supported by the diagnostic pattern, and we believe this direction bears significant research merit for future exploration.
[1] Fan et al. What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models.
Dear Reviewers,
As the discussion phase deadline approaches, we kindly check if our submitted responses and revised manuscript have addressed your concerns, or if you have any remaining questions we could clarify before the end of the discussion period.
Best regards, The Authors
Response to Reviewer JJxw — Reply 1/2
We thank the reviewer for the constructive feedback. We address the supervision control with language-guided SAM 3, evaluate rollout strategies across three training seeds, and examine the smaller gains on LLaVA through a representation-level diagnostic.
Q1. Comparison with a strong query-conditioned grounding teacher
We add SAM 3 as a strong language-guided grounding teacher, using evidence keywords extracted from the same question–answer pairs, together with their corresponding bounding boxes, to generate masks. These are converted into teacher targets using the same procedure as the original Box+SAM control.
We keep the training data, scorer architecture, SFT objective, optimization settings, and training budget fixed. Both variants use SFT only, without RL.
Diffusion supervision exceeds this stronger control by 2.79, 3.74, and 5.60 percentage points in average retention. Its advantage therefore persists against language-guided, query-derived grounding under matched downstream training. This strengthens the evidence beyond the original geometric control.