- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies limitations of existing visual token pruning evaluation protocols. It argues that aggregate accuracy and retained token count do not reliably measure capability preservation, token-selection effectiveness, or practical acceleration. The authors introduce PAA-Bench, consisting of dense-model correctness preservation, Random-Pass@k based selection difficulty analysis, and Preservation-Constrained Speedup (PCS). Experiments across multiple pruning methods, VLM backbones, and benchmarks show that preservation-aware evaluation can change method rankings and that token compression does not necessarily translate into higher end-to-end speedup.
Critical strengths
-
This paper addresses an important but underexplored issue in visual token pruning: whether current evaluation protocols actually measure what pruning methods are designed to optimize. The separation between preservation, selection sensitivity, and realized speedup provides a more complete evaluation perspective.
-
The empirical analysis is extensive. Evaluating multiple pruning methods, VLM backbones, and benchmarks reveals non-trivial ranking changes under preservation-aware metrics and demonstrates that final token count is an unreliable efficiency indicator.
Critical weaknesses
-
The paper introduces new evaluation metrics but does not sufficiently validate that these metrics better reflect practical pruning quality. For example, it would be valuable to study whether rankings produced by PAA-Bench correlate better with downstream deployment objectives or user-relevant efficiency/performance trade-offs compared with conventional accuracy-based evaluation.
-
Random-Pass@k is used as a measure of selection difficulty, but the connection between random pruning sensitivity and the actual difficulty of token selection remains unclear. Random masks may have a substantially different distribution from learned or heuristic pruning strategies, especially for models with structured visual tokens such as high-resolution or multi-scale inputs.
-
PCS relies on selecting the best evaluated configuration for each method under a preservation constraint. Since the metric uses a maximum over available configurations, rankings may depend on how densely each method is evaluated over token budgets. The paper should clarify this effect or evaluate methods on a standardized configuration grid.
-
The comparison with previous evaluation frameworks is limited. Since the main contribution is an evaluation methodology rather than a pruning algorithm, a direct comparison showing when PAA-Bench provides more informative conclusions than existing benchmarks would strengthen the contribution.
-
The preservation metric conditions on examples that are correctly solved by the dense model. While this avoids confounding from wrong-to-correct transitions, it does not evaluate whether pruning preserves the model's ability on originally incorrect examples or changes failure modes. A broader analysis of prediction transitions would provide a more complete view of pruning behavior.
Minor comments:
- Some claims about “evaluation failure” are somewhat strong because aggregate accuracy remains a valid metric for measuring final task performance, even if it is insufficient for analyzing pruning mechanisms.
- Figure 4 contains many subplots and small annotations, making detailed comparison difficult.
Questions for authors
pls see weakness
4 comments
Response to Reviewer 9Fs5 [2/3]
W2. Random-Pass@k is used as a measure of selection difficulty, but the connection between random pruning sensitivity and the actual difficulty of token selection remains unclear. Random masks may have a substantially different distribution from learned or heuristic pruning strategies, especially for models with structured visual tokens such as high-resolution or multi-scale inputs.
A. Thank you for your insightful questions. Our theoretical and empirical analyses support Random-Pass@k as a meaningful diagnostic of token-selection difficulty, including for multi-view inputs. A shared random reference provides a consistent basis for comparing methods with different selection priors, without needing to match any individual selector’s distribution. We elaborate on these two points below.
- Can Random-Pass@k capture token selection difficulty?
Random-Pass@k identifies examples whose correctness is more sensitive to which tokens are retained. At a fixed budget, it measures how often each example remains correct under different random masks. Under the simplified evidence model in Proposition 1, a lower random success probability leaves more room for targeted selection to improve on random selection. Figure 3 supports this diagnostic interpretation across the evaluated models and budgets, with larger method gaps on harder examples.
The multi-view results directly address the concern about structured visual tokens. For LLaVA-NeXT at 320 tokens, the range across the three pruning methods grows from 1.7 percentage points in Easy to 13.4 in Medium and 25.7 in Hard, with a corresponding increase in the standard deviation. The strata thus identify examples where effective token selection matters more even under multi-view inputs.
Table II. Preservation for LLaVA-NeXT at 320 tokens (%). Range and population standard deviation are computed across the three pruning methods, excluding Random.
| Method | Easy | Medium | Hard |
|---|---|---|---|
| Random | 98.0 | 56.2 | 13.8 |
| DART | 96.8 | 82.7 | 65.0 |
| DivPrune | 97.1 | 73.3 | 39.3 |
| VisionZip | 95.4 | 69.3 | 46.4 |
| Range (excluding Random) | 1.7 | 13.4 | 25.7 |
| Population SD (excluding Random) | 0.7 | 5.6 | 10.8 |
- What are the advantages of Random-Pass@k?
The shared reference need not match any individual selector’s distribution: independence from a particular selection prior supports a fair comparison across methods. Attention-based and diversity-based methods use different retention criteria, so matching one selector would tie the stratification to its prior. Uniform random sampling provides a common basis for assessing how effectively different strategies handle examples whose correctness is sensitive to token choice. The protocol is also readily reusable across existing VQA benchmarks, requiring only random-mask predictions and existing correctness labels, without additional difficulty annotations.
W3. PCS relies on selecting the best evaluated configuration for each method under a preservation constraint. Since the metric uses a maximum over available configurations, rankings may depend on how densely each method is evaluated over token budgets. The paper should clarify this effect or evaluate methods on a standardized configuration grid.
A. Thank you for this important question. To ensure a fair comparison, we apply the same search protocol and precision to every method, together with a shared timing protocol. Specifically:
- On search range and precision. For each preservation target, we fix each method's non-budget settings according to its official implementation and binary-search from its minimum valid token budget to the full budget. The search stops when the interval contains no other valid budget or reaches the shared budget resolution. Search precision is therefore governed by the same rule for every method. PCS reports each method's fastest evaluated configuration that meets the target. Table 10 lists the selected budgets.
- On timing protocol. To make latency measurements comparable, we use single-GPU inference on an NVIDIA RTX PRO 6000 Blackwell with FP16, batch size 1, greedy decoding, and
max_new_tokens=128. In each round, both dense and pruned pipelines process 50 warm-up examples before timing. All methods use the same synchronization procedure and timing boundaries, and end-to-end latency includes token selection overhead.
We will document the search and timing details in the revision and clarify that the reported speedup rankings apply to this evaluation setting.
Response to Reviewer 9Fs5 [3/3]
W5. The preservation metric conditions on examples that are correctly solved by the dense model. While this avoids confounding from wrong-to-correct transitions, it does not evaluate whether pruning preserves the model's ability on originally incorrect examples or changes failure modes. A broader analysis of prediction transitions would provide a more complete view of pruning behavior.
A. Thank you for your helpful questions. Following your suggestion, we analyze how pruning affects predictions on examples the dense model originally answered incorrectly.
For each benchmark, we measure the shares of these examples whose answers change or remain unchanged after pruning. Table III reports their averages across nine benchmarks.
Table III. Answer changes on dense-incorrect examples for LLaVA-1.5-7B at 128 and 64 tokens. Percentages are averaged equally across nine benchmarks. Changed answers include corrections and transitions between incorrect answers.
| Method | Changed (128 tokens) | Unchanged (128 tokens) | Changed (64 tokens) | Unchanged (64 tokens) |
|---|---|---|---|---|
| Random | 34.00% | 66.00% | 40.74% | 59.26% |
| CDPruner | 26.09% | 73.91% | 30.75% | 69.25% |
| DivPrune | 27.53% | 72.47% | 32.43% | 67.57% |
| FastV | 28.02% | 71.98% | 38.51% | 61.49% |
| DART | 24.09% | 75.91% | 33.66% | 66.34% |
| SparseVLM | 21.48% | 78.52% | 24.74% | 75.26% |
Pruning changes a substantial share of answers on dense-incorrect examples, and this share increases for all six methods when the budget is reduced from 128 to 64 tokens. For CDPruner, it rises from 26.09% to 30.75%.
Whether an answer changes or stays the same provides limited evidence about the model's visual grounding or image understanding. We therefore base our main preservation metric on examples the dense model originally answered correctly.
Minor comment 1. Some claims about “evaluation failure” are somewhat strong because aggregate accuracy remains a valid metric for measuring final task performance, even if it is insufficient for analyzing pruning mechanisms.
A. Thank you for constructive question. Counting wrong-to-correct transitions of uncertain origin toward preservation would make it harder to detect losses of the dense model’s existing capabilities.
In Figure 2(b), 76.3% of stable wrong-to-correct cases under random pruning for LLaVA-1.5 at 16 tokens are also answered correctly with text-only input. At 64 tokens on LLaVA-1.5-7B, Random reproduces 80.56% of CDPruner’s wrong-to-correct cases within five trials. Please refer to our response to Reviewer J39d (Q3 and Discussion 2) for detailed results.
In these comparisons, most of the corresponding wrong-to-correct cases can also be obtained with text-only input or Random selection. Such correct outcomes alone do not establish that a pruning method improves visual understanding through better token selection.
We will qualify “evaluation failure” to state that aggregate accuracy alone is insufficient for analyzing pruning mechanisms, while retaining its role in reporting net task performance.
Minor comment 2. Figure 4 contains many subplots and small annotations, making detailed comparison difficult.
A. Thank you for this suggestion. We have adjusted Figure 4's layout and font sizes to improve readability.
[1] Liao et al. Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methods.
[2] Wang et al. EffiVLM-Bench: A Comprehensive Benchmark for Evaluating Training-Free Acceleration in Large Vision-Language Models.
[3] Zhuang et al. VASparse: Towards Efficient Visual Hallucination Mitigation for Large Vision-Language Model via Visual-Aware Sparsification.
Dear Reviewer 9Fs5,
We would like to thank you again for your valuable time and effort in reviewing our submission. Based on your previous feedback, we have provided point-to-point responses. We are wondering if our response has addressed your questions appropriately and can improve your opinion of our work.
As the discussion period is coming to its end, please kindly let us know if you might have further comments, and we would be glad to address them.
Best regards,
The Authors
Response to Reviewer 9Fs5 [1/3]
Thank you for recognizing the importance of our evaluation perspective and the breadth of our empirical analysis, and for your thoughtful suggestions. We provide further clarification and evidence on the practical value of PAA-Bench, the rationale for Random-Pass@k, and the fairness of PCS evaluation to address your concerns below.
A. Thank you for these thoughtful questions. In practice, the goal of pruning is to maximize actual inference speedup while meeting the required level of capability preservation. PAA-Bench aligns method comparisons with this objective, providing a basis for selecting methods that meet practical preservation and efficiency requirements. Its three complementary components address this objective as follows:
Table I. Evaluation features of PAA-Bench and prior frameworks. [1, 2]
Random-Pass@k makes these diagnostics plug-and-play across existing VQA benchmarks by constructing difficulty strata from model predictions and correctness labels.