acceptodds
Under review as a conference paper at ICLR 2027

Are We Evaluating VLM Token Pruning Right? Looking Beyond Accuracy and Token Count

Abstract

Visual token pruning aims to reduce inference cost while preserving the capabilities of the original VLM. Yet existing evaluations rely on aggregate task accuracy and retained token count as proxies for pruning quality and actual efficiency. We reveal two limitations: 1) Aggregate accuracy can obscure whether a pruning method truly identifies critical visual tokens—many examples remain solvable under arbitrary token removal and models can exploit textual shortcuts when visual evidence is degraded. 2) Retained token count is an unreliable proxy for efficiency, as identical token budgets can incur different end-to-end costs depending on where and how pruning is performed. Consequently, comparisons based on aggregate accuracy and retained token counts fail to recover the global performance–efficiency Pareto frontier across methods. We introduce PAA-Bench, a ***plug-and-play*** evaluation framework for existing benchmarks, to evaluate whether pruning succeeds through meaningful token selection and whether this success translates into real speedup. To determine when token choice actually matters, we propose *Random-Pass@* to measure how often an example remains correct under random pruning, enabling comprehensive evaluation across different levels of selection difficulty. To compare efficiency fairly, we introduce *Preservation-Constrained Speedup* (PCS) to report the maximum end-to-end speedup each method achieves under a shared preservation target. Experiments across *8* methods, *3* VLM backbones, and *9* benchmarks show that stronger token compression does not necessarily yield greater speedup, method differences widen on examples where token selection matters, and efficiency rankings change with the preservation target.

Primary area:
datasets and benchmarks
Keywords:
Vision-Language Models, Visual Token Pruning, Efficient Inference, Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9Fs5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies limitations of existing visual token pruning evaluation protocols. It argues that aggregate accuracy and retained token count do not reliably measure capability preservation, token-selection effectiveness, or practical acceleration. The authors introduce PAA-Bench, consisting of dense-model correctness preservation, Random-Pass@k based selection difficulty analysis, and Preservation-Constrained Speedup (PCS). Experiments across multiple pruning methods, VLM backbones, and benchmarks show that preservation-aware evaluation can change method rankings and that token compression does not necessarily translate into higher end-to-end speedup.

Critical strengths
  1. This paper addresses an important but underexplored issue in visual token pruning: whether current evaluation protocols actually measure what pruning methods are designed to optimize. The separation between preservation, selection sensitivity, and realized speedup provides a more complete evaluation perspective.

  2. The empirical analysis is extensive. Evaluating multiple pruning methods, VLM backbones, and benchmarks reveals non-trivial ranking changes under preservation-aware metrics and demonstrates that final token count is an unreliable efficiency indicator.

Critical weaknesses
  1. The paper introduces new evaluation metrics but does not sufficiently validate that these metrics better reflect practical pruning quality. For example, it would be valuable to study whether rankings produced by PAA-Bench correlate better with downstream deployment objectives or user-relevant efficiency/performance trade-offs compared with conventional accuracy-based evaluation.

  2. Random-Pass@k is used as a measure of selection difficulty, but the connection between random pruning sensitivity and the actual difficulty of token selection remains unclear. Random masks may have a substantially different distribution from learned or heuristic pruning strategies, especially for models with structured visual tokens such as high-resolution or multi-scale inputs.

  3. PCS relies on selecting the best evaluated configuration for each method under a preservation constraint. Since the metric uses a maximum over available configurations, rankings may depend on how densely each method is evaluated over token budgets. The paper should clarify this effect or evaluate methods on a standardized configuration grid.

  4. The comparison with previous evaluation frameworks is limited. Since the main contribution is an evaluation methodology rather than a pruning algorithm, a direct comparison showing when PAA-Bench provides more informative conclusions than existing benchmarks would strengthen the contribution.

  5. The preservation metric conditions on examples that are correctly solved by the dense model. While this avoids confounding from wrong-to-correct transitions, it does not evaluate whether pruning preserves the model's ability on originally incorrect examples or changes failure modes. A broader analysis of prediction transitions would provide a more complete view of pruning behavior.

Minor comments:

  • Some claims about “evaluation failure” are somewhat strong because aggregate accuracy remains a valid metric for measuring final task performance, even if it is insufficient for analyzing pruning mechanisms.
  • Figure 4 contains many subplots and small annotations, making detailed comparison difficult.
Questions for authors

pls see weakness

4 comments
AuthorsOpenReview ↗

Response to Reviewer 9Fs5 [1/3]

Thank you for recognizing the importance of our evaluation perspective and the breadth of our empirical analysis, and for your thoughtful suggestions. We provide further clarification and evidence on the practical value of PAA-Bench, the rationale for Random-Pass@k, and the fairness of PCS evaluation to address your concerns below.

W1. The paper introduces new evaluation metrics but does not sufficiently validate that these metrics better reflect practical pruning quality. For example, it would be valuable to study whether rankings produced by PAA-Bench correlate better with downstream deployment objectives or user-relevant efficiency/performance trade-offs compared with conventional accuracy-based evaluation.

W4. The comparison with previous evaluation frameworks is limited. Since the main contribution is an evaluation methodology rather than a pruning algorithm, a direct comparison showing when PAA-Bench provides more informative conclusions than existing benchmarks would strengthen the contribution.

A. Thank you for these thoughtful questions. In practice, the goal of pruning is to maximize actual inference speedup while meeting the required level of capability preservation. PAA-Bench aligns method comparisons with this objective, providing a basis for selecting methods that meet practical preservation and efficiency requirements. Its three complementary components address this objective as follows:

  • Preservation changes which method is preferred. Unlike evaluations that summarize post-pruning performance with aggregate accuracy, PAA-Bench measures how many of the dense model's correct predictions remain correct. For LLaVA-1.5-7B at 64 tokens, CDPruner ranks above SparseVLM in full-set binary accuracy, while SparseVLM ranks above CDPruner in preservation. This distinction matters when choosing a method to retain the capabilities of an existing model.
  • Difficulty stratification reveals larger differences between selectors. Unlike evaluations that pool all examples into a single score, PAA-Bench uses Random-Pass@k to group examples by their sensitivity to token selection for each model and budget. For LLaVA-1.5 at 64 tokens, the range across methods, excluding Random, grows from 6.9 percentage points in Easy to 18.8 in Medium and 27.6 in Hard. These strata expose examples where the choice of selector has a greater effect on preservation.
  • PCS compares speed at the required preservation level. Unlike evaluations that hold the token budget fixed, PCS compares measured end-to-end speedups at a common preservation target and includes token selection overhead. It therefore answers which method provides greater acceleration while meeting the required level of preservation.

Table I. Evaluation features of PAA-Bench and prior frameworks. [1, 2]

Evaluation FeatureVTC-BenchEffiVLM-BenchPAA-Bench
Dense-Correct Evaluation✓—✓
Per-Model Difficulty Grouping——✓
Measured Performance–Efficiency Trade-off—✓✓

Random-Pass@k makes these diagnostics plug-and-play across existing VQA benchmarks by constructing difficulty strata from model predictions and correctness labels.

AuthorsOpenReview ↗

Response to Reviewer 9Fs5 [2/3]

W2. Random-Pass@k is used as a measure of selection difficulty, but the connection between random pruning sensitivity and the actual difficulty of token selection remains unclear. Random masks may have a substantially different distribution from learned or heuristic pruning strategies, especially for models with structured visual tokens such as high-resolution or multi-scale inputs.

A. Thank you for your insightful questions. Our theoretical and empirical analyses support Random-Pass@k as a meaningful diagnostic of token-selection difficulty, including for multi-view inputs. A shared random reference provides a consistent basis for comparing methods with different selection priors, without needing to match any individual selector’s distribution. We elaborate on these two points below.

  • Can Random-Pass@k capture token selection difficulty?

Random-Pass@k identifies examples whose correctness is more sensitive to which tokens are retained. At a fixed budget, it measures how often each example remains correct under different random masks. Under the simplified evidence model in Proposition 1, a lower random success probability leaves more room for targeted selection to improve on random selection. Figure 3 supports this diagnostic interpretation across the evaluated models and budgets, with larger method gaps on harder examples.

The multi-view results directly address the concern about structured visual tokens. For LLaVA-NeXT at 320 tokens, the range across the three pruning methods grows from 1.7 percentage points in Easy to 13.4 in Medium and 25.7 in Hard, with a corresponding increase in the standard deviation. The strata thus identify examples where effective token selection matters more even under multi-view inputs.

Table II. Preservation for LLaVA-NeXT at 320 tokens (%). Range and population standard deviation are computed across the three pruning methods, excluding Random.

MethodEasyMediumHard
Random98.056.213.8
DART96.882.765.0
DivPrune97.173.339.3
VisionZip95.469.346.4
Range (excluding Random)1.713.425.7
Population SD (excluding Random)0.75.610.8
  • What are the advantages of Random-Pass@k?

The shared reference need not match any individual selector’s distribution: independence from a particular selection prior supports a fair comparison across methods. Attention-based and diversity-based methods use different retention criteria, so matching one selector would tie the stratification to its prior. Uniform random sampling provides a common basis for assessing how effectively different strategies handle examples whose correctness is sensitive to token choice. The protocol is also readily reusable across existing VQA benchmarks, requiring only random-mask predictions and existing correctness labels, without additional difficulty annotations.

W3. PCS relies on selecting the best evaluated configuration for each method under a preservation constraint. Since the metric uses a maximum over available configurations, rankings may depend on how densely each method is evaluated over token budgets. The paper should clarify this effect or evaluate methods on a standardized configuration grid.

A. Thank you for this important question. To ensure a fair comparison, we apply the same search protocol and precision to every method, together with a shared timing protocol. Specifically:

  • On search range and precision. For each preservation target, we fix each method's non-budget settings according to its official implementation and binary-search from its minimum valid token budget to the full budget. The search stops when the interval contains no other valid budget or reaches the shared budget resolution. Search precision is therefore governed by the same rule for every method. PCS reports each method's fastest evaluated configuration that meets the target. Table 10 lists the selected budgets.
  • On timing protocol. To make latency measurements comparable, we use single-GPU inference on an NVIDIA RTX PRO 6000 Blackwell with FP16, batch size 1, greedy decoding, and max_new_tokens=128. In each round, both dense and pruned pipelines process 50 warm-up examples before timing. All methods use the same synchronization procedure and timing boundaries, and end-to-end latency includes token selection overhead.

We will document the search and timing details in the revision and clarify that the reported speedup rankings apply to this evaluation setting.

AuthorsOpenReview ↗

Response to Reviewer 9Fs5 [3/3]

W5. The preservation metric conditions on examples that are correctly solved by the dense model. While this avoids confounding from wrong-to-correct transitions, it does not evaluate whether pruning preserves the model's ability on originally incorrect examples or changes failure modes. A broader analysis of prediction transitions would provide a more complete view of pruning behavior.

A. Thank you for your helpful questions. Following your suggestion, we analyze how pruning affects predictions on examples the dense model originally answered incorrectly.

For each benchmark, we measure the shares of these examples whose answers change or remain unchanged after pruning. Table III reports their averages across nine benchmarks.

Table III. Answer changes on dense-incorrect examples for LLaVA-1.5-7B at 128 and 64 tokens. Percentages are averaged equally across nine benchmarks. Changed answers include corrections and transitions between incorrect answers.

MethodChanged (128 tokens)Unchanged (128 tokens)Changed (64 tokens)Unchanged (64 tokens)
Random34.00%66.00%40.74%59.26%
CDPruner26.09%73.91%30.75%69.25%
DivPrune27.53%72.47%32.43%67.57%
FastV28.02%71.98%38.51%61.49%
DART24.09%75.91%33.66%66.34%
SparseVLM21.48%78.52%24.74%75.26%

Pruning changes a substantial share of answers on dense-incorrect examples, and this share increases for all six methods when the budget is reduced from 128 to 64 tokens. For CDPruner, it rises from 26.09% to 30.75%.

Whether an answer changes or stays the same provides limited evidence about the model's visual grounding or image understanding. We therefore base our main preservation metric on examples the dense model originally answered correctly.

Minor comment 1. Some claims about “evaluation failure” are somewhat strong because aggregate accuracy remains a valid metric for measuring final task performance, even if it is insufficient for analyzing pruning mechanisms.

A. Thank you for constructive question. Counting wrong-to-correct transitions of uncertain origin toward preservation would make it harder to detect losses of the dense model’s existing capabilities.

In Figure 2(b), 76.3% of stable wrong-to-correct cases under random pruning for LLaVA-1.5 at 16 tokens are also answered correctly with text-only input. At 64 tokens on LLaVA-1.5-7B, Random reproduces 80.56% of CDPruner’s wrong-to-correct cases within five trials. Please refer to our response to Reviewer J39d (Q3 and Discussion 2) for detailed results.

In these comparisons, most of the corresponding wrong-to-correct cases can also be obtained with text-only input or Random selection. Such correct outcomes alone do not establish that a pruning method improves visual understanding through better token selection.

We will qualify “evaluation failure” to state that aggregate accuracy alone is insufficient for analyzing pruning mechanisms, while retaining its role in reporting net task performance.

Minor comment 2. Figure 4 contains many subplots and small annotations, making detailed comparison difficult.

A. Thank you for this suggestion. We have adjusted Figure 4's layout and font sizes to improve readability.

[1] Liao et al. Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methods.

[2] Wang et al. EffiVLM-Bench: A Comprehensive Benchmark for Evaluating Training-Free Acceleration in Large Vision-Language Models.

[3] Zhuang et al. VASparse: Towards Efficient Visual Hallucination Mitigation for Large Vision-Language Model via Visual-Aware Sparsification.

Dear Reviewer 9Fs5,

We would like to thank you again for your valuable time and effort in reviewing our submission. Based on your previous feedback, we have provided point-to-point responses. We are wondering if our response has addressed your questions appropriately and can improve your opinion of our work.

As the discussion period is coming to its end, please kindly let us know if you might have further comments, and we would be glad to address them.

Best regards,

The Authors

Reviewer_Bxy8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether evaluations of visual token pruning accurately measure capability preservation and actual inference speedup. The authors point out that overall task scores combine two types of changes. Some originally correct predictions become incorrect after pruning. Some originally incorrect predictions become correct. The number of retained tokens also fails to fully capture the costs of pruning location and token selection.

The paper proposes PAA-Bench with three evaluation components. First, it measures correctness preservation on examples that the original model answers correctly. Second, it groups examples into Easy, Medium, and Hard using Random-Pass@10, which counts correct answers across ten random pruning trials. Third, it introduces Preservation-Constrained Speedup, or PCS. This metric reports the maximum measured speedup across tested configurations that meet a shared correctness preservation target. The speedup covers the full inference pipeline.

Critical strengths
  1. Separating different evaluation goals has practical research value.

Preserving the original model's correct predictions, achieving a high final task score, and obtaining actual speedup are different goals. The paper uses prediction transition analysis and runtime examples to show why accuracy and token count alone can lead to misleading conclusions. In particular, Figure 4 and Appendix E show cases where retaining fewer tokens results in slower inference. These findings have direct practical relevance.

  1. Random pruning stratification reveals differences hidden by overall averages.

For LLaVA-1.5 with 64 retained tokens, the gap between the highest and lowest scores among pruning methods increases with selection difficulty. Excluding Random, the gap rises from 6.9 percentage points on Easy examples to 18.8 on Medium examples and 27.6 on Hard examples, as shown on page 8 and in Figure 3.

The authors also examine the effect of the number of random trials. The groups assigned using the original ten trials agree with those assigned using forty additional independent trials in an average of 89.1% of cases, as reported in Table 14. This provides a stability check beyond a single set of random trials.

Critical weaknesses
  1. The ranking reversal analysis does not fully separate correction gains from changes in scoring rules.

Section 5.1 compares Rel., the relative score under each benchmark's original metric, with W-Avg, which measures binary correctness preservation. This comparison is used to explain changes in method rankings. However, Appendix B states that native metrics such as VQAv2 allow partial credit, while MME uses a composite score. These metrics differ from binary correctness.

According to Equation (9) in the appendix, relative accuracy equals the preservation rate plus the contribution from originally incorrect predictions that become correct. This relation holds when the samples, predictions, and binary evaluator are identical. Relative accuracy therefore cannot be lower than the preservation rate under these conditions.

Nonbinary benchmark scores may explain differences between the reported metrics. However, the comparison does not isolate how much of the ranking reversal comes from correction gains. This affects the central explanation for RQ1.

  1. The key efficiency rankings lack sufficient information for verification.

PCS takes the maximum speedup over each method's tested configurations. Its value therefore depends on the candidate configurations, search density, preservation estimates, and timing procedure. Appendix C.4 mainly describes what the timing boundary should include. It does not clearly provide the complete configuration sets, the actual preservation rates at the selected points in Tables 10 to 12, or uncertainty in timing and preservation estimates.

These omissions make it difficult to assess whether all methods receive equally thorough searches and whether small speed differences are stable. The paper also does not show whether method rankings remain stable after examples are assigned to new difficulty groups. Sample counts for each group are missing.

The experiments support the reported observations under the tested conditions. More evidence is needed to establish a reliable protocol for comparing methods.

Questions for authors
  1. Could you report full-set accuracy, relative accuracy, all four prediction transition counts, and correctness preservation using identical samples, predictions, and a shared binary evaluator? Please present these results alongside the native benchmark scores.

  2. How many of the ranking reversals in Table 1 remain under this shared evaluator? To what extent can they be explained by originally incorrect predictions becoming correct?

  3. Please specify the rule used to convert each benchmark's scores into binary correctness labels.

  4. Please report each method's candidate configurations, search procedure, and configuration selection process. Please also provide the actual preservation rates and timing uncertainty at the selected points in Tables 10 to 12. Were configurations selected directly on the final evaluation set? If so, please clarify that the results describe the frontier observed on that set. Can the same preservation targets and main rankings be maintained on a test set after independent calibration?

  5. At which pruning location is the shared Random stratification performed, and which tokens are eligible for removal?

  6. Different methods prune at different stages of visual information mixing. Do random controls matched to each method's pruning location still support the main conclusions?

5 comments
AuthorsOpenReview ↗

Response to Reviewer Bxy8 [1/4]

Thank you for recognizing the practical value of separating evaluation goals and the diagnostic value of Random-Pass stratification, and for your thoughtful suggestions. We provide further clarification and evidence on ranking reversals, PCS validation, and the robustness of difficulty stratification to address your concerns below.

W1. The ranking reversal analysis does not fully separate correction gains from changes in scoring rules. […] Section 5.1 compares Rel., the relative score under each benchmark's original metric, with W-Avg, which measures binary correctness preservation. […] However, the comparison does not isolate how much of the ranking reversal comes from correction gains.

Q2. How many of the ranking reversals in Table 1 remain under this shared evaluator? To what extent can they be explained by originally incorrect predictions becoming correct?

A. Thank you for these important questions. Ranking reversals persist under a shared binary evaluator, supporting our original conclusion. Following your suggestion, we recomputed RelAcc using the same binary scoring rules as preservation, keeping the evaluated samples and predictions unchanged. The results show that:

At 128 tokens, CDPruner has higher RelAcc than SparseVLM (97.9% vs. 96.5%) but lower preservation (92.1% vs. 94.4%). The ranking reversal therefore persists under shared scoring. CDPruner corrects more of the dense model’s errors but also loses more of its originally correct predictions, illustrating why higher overall accuracy need not mean better preservation.

Table I. LLaVA-1.5-7B at 128 tokens. Avg, RelAcc, and Preservation (%) are averaged equally across benchmarks. Transition counts cover the same 255,663 examples, and parenthetical percentages use this total. Preservation is computed on each benchmark's dense-correct subset before averaging.

MethodAvgRelAccPreservationCorrect → correctCorrect → wrongWrong → correctWrong → wrong
Baseline70.5100.00100.00206,591 (80.81%)0049,072
CDPruner69.097.992.1195,444 (76.45%)11,1477,13341,939
DivPrune68.797.491.2193,875 (75.83%)12,7166,63742,435
DART68.196.693.0193,768 (75.79%)12,8236,40742,665
FastV6896.590.9186,492 (72.94%)20,0997,66341,409
SparseVLM6896.594.4196,697 (76.94%)9,8944,42344,649
Random65.492.885.5185,184 (72.43%)21,4077,71741,355

Q1. Could you report full-set accuracy, relative accuracy, all four prediction transition counts, and correctness preservation using identical samples, predictions, and a shared binary evaluator? Please present these results alongside the native benchmark scores.

A. Thank you for your insightful question. The added transition analysis shows a substantial number of wrong-to-correct transitions, which can offset the effect of correct-to-wrong transitions on RelAcc and affect how well it reflects preservation of the dense model’s capabilities. Table I reports full-set accuracy, relative accuracy, preservation, and all four transition counts from the same underlying samples and predictions, using a shared binary evaluator for the dense model, all pruning methods, and Random. For comparison, Table 1 in the main paper reports relative scores under the native benchmark evaluators.

AuthorsOpenReview ↗

Response to Reviewer Bxy8 [2/4]

W2-1. The key efficiency rankings lack sufficient information for verification. […] It does not clearly provide the complete configuration sets, the actual preservation rates at the selected points in Tables 10 to 12, or uncertainty in timing and preservation estimates. […] These omissions make it difficult to assess whether all methods receive equally thorough searches and whether small speed differences are stable.

Q4-1. Please report each method's candidate configurations, search procedure, and configuration selection process. Please also provide the actual preservation rates and timing uncertainty at the selected points in Tables 10 to 12.

A. Thank you for these thoughtful questions. To ensure a fair comparison, we apply the same search protocol and precision to every method, together with a shared end-to-end timing protocol. We clarify both parts below and report the selected configurations' preservation and timing variability in Table III.

  • On search range and precision. We fix each method's non-budget settings according to its official implementation and search only the retained-token budget for each common preservation target. Binary search starts at the method's minimum valid budget and the full token budget. It stops when the interval contains no other valid budget or reaches the shared budget resolution. Search precision is therefore governed by the same rule across methods while accommodating their valid budgets. PCS reports each method's fastest evaluated configuration that meets the target. Table 10 lists the selected token budgets.
  • On timing protocol. We use a single NVIDIA RTX PRO 6000 Blackwell GPU (96 GB), FP16, batch size 1, greedy decoding, and max_new_tokens=128. In each round, the dense and pruned configurations each process 50 warm-up examples, which are excluded from timing. All methods use the same synchronization procedure and timing boundaries, and end-to-end latency includes token-selection overhead. We will document these search and timing details in the revision.

Repeated timing shows stable speedup estimates, and all selected pruning configurations meet the common 95% preservation target. Preservation rates range from 95.07% to 95.27% across the four benchmarks. Speedup standard deviations over three paired timing runs range from 0.0003 to 0.0059.

Table III. LLaVA-1.5-7B at a 95% preservation target. MMB denotes MMBench-EN. SP. is speedup (×), reported as the mean ± sample standard deviation across three paired timing runs over the full dataset. Pre. is preservation (%). Dense is the reference for both metrics.

MethodMMB SP.MMB Pre.TextVQA SP.TextVQA Pre.POPE SP.POPE Pre.GQA SP.GQA Pre.
Dense1.0000100.00%1.0000100.00%1.0000100.00%1.0000100.00%
Random1.4080 ± 0.000395.12%1.0775 ± 0.001195.26%1.7117 ± 0.002695.24%1.1902 ± 0.002595.07%
VisionZip1.5563 ± 0.001095.09%1.3178 ± 0.002495.11%1.6959 ± 0.001295.27%1.1843 ± 0.001895.15%
FastV1.7115 ± 0.001395.15%1.4062 ± 0.002195.14%1.3293 ± 0.001595.11%1.2339 ± 0.003095.23%
SparseVLM1.7404 ± 0.001695.21%1.3275 ± 0.000895.26%1.7357 ± 0.002495.09%1.2205 ± 0.002395.25%
CDPruner1.3490 ± 0.002295.15%1.0401 ± 0.002395.23%1.9771 ± 0.005995.25%0.9039 ± 0.001395.20%
DivPrune1.5323 ± 0.001695.18%1.1964 ± 0.001995.17%2.0796 ± 0.005695.27%1.1810 ± 0.000595.19%
DART1.7217 ± 0.001695.24%1.3292 ± 0.002995.07%1.5850 ± 0.000395.27%1.2615 ± 0.001795.27%
AuthorsOpenReview ↗

Response to Reviewer Bxy8 [3/4]

Q3. Please specify the rule used to convert each benchmark's scores into binary correctness labels.

A. Thank you for this insightful question. We assign one binary correctness label to each example using the benchmark-specific rules in Table II. The same rules apply to the dense model, all pruning methods, and Random.

Table II. Rules for converting benchmark predictions into item-level binary correctness labels.

BenchmarkBinary scoring rule
VQAv2, TextVQA, VizWizNormalize answers using EvalAI and compute a soft agreement score against the 10 reference answers. A score of at least 0.3 counts as correct.
GQAThe normalized answer must exactly match the reference answer.
ScienceQA-IMG, MMBench-EN/CNExtract the predicted option letter and check it against the correct option.
POPE, MMEParse the yes/no answer using the benchmark's rules and compare it with the reference answer.

We use item-level binary scoring because some benchmarks’ native metrics do not directly support evaluation on the dense-correct subset. For example, MME’s native score includes joint correctness for question pairs, but retaining only dense-correct questions can leave these pairs incomplete.

Q4-2. Were configurations selected directly on the final evaluation set? If so, please clarify that the results describe the frontier observed on that set. Can the same preservation targets and main rankings be maintained on a test set after independent calibration?

A. Thank you for your constructive questions. As described in our response to W2-1, all methods follow the same search protocol: for each method, we select the configuration with the highest measured speedup among those evaluated that meet the common preservation target. Since this selection is performed directly on the final evaluation set, the reported PCS values characterize the preservation–speedup frontier observed among the evaluated configurations on that set. Independent calibration further shows that configurations selected under this target maintain similar preservation on held-out examples, supporting the stability of preservation beyond the data used for configuration selection.

Following your suggestion, we evaluate LLaVA-1.5-7B on TextVQA and MMBench-EN using three 50/50 calibration–test splits per dataset. For each split, we select a token budget meeting the 95% preservation target on the calibration set, then freeze the configuration for test-set evaluation. Table IV shows a mean absolute difference of 0.17 percentage points between the three-split calibration and test means, with maxima of 0.26 on TextVQA and 0.25 on MMBench-EN. These small differences indicate that preservation at the selected operating points remains largely stable on unseen examples, providing further empirical support for comparing efficiency under a common preservation target in PCS.

Table IV. Calibration and test preservation rates (%) on TextVQA and MMBench-EN. Each value is the mean across three 50/50 splits.

MethodTextVQA calibrationTextVQA testMMBench-EN calibrationMMBench-EN test
Random95.0395.2495.2795.34
VisionZip95.0995.3595.2194.96
FastV95.2294.9795.1395.09
SparseVLM95.1194.9195.0295.13
CDPruner95.2895.0595.1795.27
DivPrune95.1695.3195.0894.90
DART95.0795.2795.2595.42
Reviewer_J39dOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PAA-Bench evaluates visual-token pruning by separating preservation of dense-model correct predictions, sensitivity to token selection, and measured inference speed. Random-Pass@k counts successful random masks for each dense-correct example and stratifies selection difficulty. Preservation-Constrained Speedup (PCS) selects the fastest evaluated configuration meeting a common preservation target. Experiments across multiple pruning methods, three primary VLM backbones, and nine benchmark variants show that aggregate accuracy can mask regressions, that method differences grow on selection-sensitive examples, and that fewer retained tokens need not imply greater speedup. Additional analyses examine random-trial stability and further model sizes.

Critical strengths
  1. The evaluation separates quantities that answer genuinely different practical questions. Dense-correct preservation prevents wrong-to-correct transitions from hiding regressions, while PCS explicitly includes selection overhead and matches preservation rather than token count. The concrete ranking reversals make the argument more useful than a general reminder to measure latency. For example, Table 1 reverses CDPruner/SparseVLM ordering between relative task score and preservation at 64 tokens.

  2. Random-mask sensitivity provides a broadly reusable diagnostic. Stratification exposes examples on which token choice matters without requiring new question annotation. The breadth of backbones and tasks, separate sample-weighted and stratum-weighted aggregates, and Appendix G.2's independent 40-trial comparison make the framework more credible. The contribution is the combined diagnostic protocol and its empirical consequences, rather than the simple headroom inequality alone.

Critical weaknesses
  1. The random reference needs a stricter separation between stratum construction and performance estimation. Figure 3 reports Random as the mean of ten trials, while the strata are defined by ten-trial success counts. If these are the same trials, Random's low score in Hard and high score in Easy are partly imposed by construction, and informed-selector gains on Hard are measured against a conditionally depressed random estimate. Appendix G.2 tests label stability with independent trials, which is helpful, but does not report the full method-versus-Random results on fresh masks within frozen strata. This limits the strength of the claimed selection advantage, although it does not explain away differences among deterministic selectors.

  2. PCS needs more precise statistical and deployment semantics. Maximizing measured speed over configurations that meet a sample preservation threshold can select an optimistic boundary point. Appendix C.4 describes desired timing boundaries but does not fully specify repeated timing, synchronization/warmup, the search grid, or validation-versus-reporting separation. The single-device, batch-one, short-output setting also limits the scope of runtime rankings. Separately, preservation should complement net task utility: text-only overlap does not establish that every correction is spurious, as Appendix C.3 itself recognizes. These issues call for clearer calibration and narrower interpretation, rather than abandonment of the framework.

Questions for authors
  1. Protocol clarification and diagnostic: Were the masks used to report Random in Figure 3 also used to define its difficulty strata? With strata frozen from the first ten trials, what are Random and informed-selector scores on independent masks, including confidence intervals?
  2. PCS validation: What exact configuration grid and timing procedure were used, and do the selected PCS operating points meet the preservation target on held-out examples? Please report uncertainty around both preservation and speedup near the 90%/95% thresholds.
  3. Interpretation clarification: Can results jointly show preservation, correction gains, and net task performance, and distinguish the empirically supported diagnostic contribution from a universal claim that pruning-induced corrections are undesirable?

Discussions:

  1. Given the same pruning rate, do you believe the performance of different pruning methods can scale appropriately with the input image resolution?
  2. A more detailed analysis of whether there are cases where "wrong" needs to be "corrected"; as well as insights into how pruning techniques can help models learn to make correct predictions—both of which may prove highly valuable.
5 comments
AuthorsOpenReview ↗

Response to Reviewer J39d [1/4]

Thank you for recognizing our preservation-aware evaluation and the reusable Random-Pass diagnostic, and for your thoughtful suggestions. We provide further clarification and evidence on Random evaluation, PCS validation, and correction gains to address your concerns below.

W1. The random reference needs a stricter separation between stratum construction and performance estimation. […] Appendix G.2 tests label stability with independent trials, which is helpful, but does not report the full method-versus-Random results on fresh masks within frozen strata.

Q1. Protocol clarification and diagnostic: Were the masks used to report Random in Figure 3 also used to define its difficulty strata? With strata frozen from the first ten trials, what are Random and informed-selector scores on independent masks, including confidence intervals?

A. Thank you for your questions. In the main paper, Figure 3 uses the same ten masks to construct the strata and to score Random. Following your suggestion, we freeze these strata and then re-evaluate methods on independent masks, which further confirms that informed selectors consistently outperform Random and that their advantage grows as selection difficulty increases. Specifically:

  • Informed selectors are unaffected. With the examples, token budget, and inference settings fixed, deterministic selectors such as CDPruner and SparseVLM select the same tokens on every run. Their scores within frozen strata therefore do not depend on the masks used to evaluate Random.
  • Random remains well below informed selectors. Over 20 independent runs with fresh masks, Random's Easy and Medium scores change by at most 1.91 percentage points, while its Hard scores rise by about 6 points (13.40 → 19.40 at 128 tokens; 12.70 → 18.34 at 64 tokens). Despite this increase, the margin of informed selectors over Random still grows from 1.3–1.7 points on Easy examples to 26.9–52.3 points on Hard examples.

Table I. Preservation within frozen strata for LLaVA-1.5-7B (%). Strata are defined by the original 10 trials. S-Avg and W-Avg follow Table 5 in the main paper.

Method / evaluationTokensEasyMediumHardS-AvgW-Avg
Random (original masks)12897.9856.8213.4056.0785.61
6497.5956.6112.7055.6479.94
Random (20 new independent masks)12896.9258.7319.4058.3585.54
6496.3258.0818.3457.5879.96
CDPruner12898.1778.2946.3474.2792.10
6497.6180.7251.4776.6089.33
SparseVLM12898.4682.8264.8882.0594.43
6498.0187.1270.6585.2693.26
AuthorsOpenReview ↗

Response to Reviewer J39d [2/4]

W2-1. PCS needs more precise statistical and deployment semantics. Maximizing measured speed over configurations that meet a sample preservation threshold can select an optimistic boundary point. […] The single-device, batch-one, short-output setting also limits the scope of runtime rankings.

Q2-1. PCS validation: What exact configuration grid and timing procedure were used, […] Please report uncertainty around both preservation and speedup near the 90%/95% thresholds.

A. Thank you for your thoughtful questions. To ensure a fair comparison, we compare all methods under a shared timing protocol and a shared search procedure, as follows:

  • On search range and precision. For each method and preservation target, we binary-search the retained-token budget between 1 token and the full token budget, so the granularity is determined by bisection rather than a predefined grid. Because retaining fewer tokens generally lowers preservation and reduces runtime, this binary-search procedure identifies the smallest budget that meets the target, which yields the fastest qualifying configuration. Table 10 in the appendix lists the selected budgets.
  • On timing protocol. We adopt the evaluation setting commonly used in prior pruning work, so that our measurements remain directly comparable with existing reports: single-GPU inference (an RTX PRO 6000 in our case) with FP16, batch size 1, greedy decoding, and max_new_tokens=128. Each method uses the default hyperparameters of its official implementation, with the token budget as the only variable searched by PCS. For both dense and pruned pipelines, we exclude 50 warm-up examples, apply the same synchronization, and measure end-to-end latency including token-selection overhead. We will clarify in the revision that the reported speedup rankings refer to this adopted setting.

As in the table below, repeated measurements show that these estimates are stable. Speedup standard deviations range from only 0.0003 to 0.0059 over three paired timing runs, well below the margins between methods. Preservation also shows no variance for deterministic methods, which select the same tokens on every run, and varies by at most 0.32 points for Random.

Table II. LLaVA-1.5-7B speedup (×) at the 95% and 90% preservation targets. Values are means ± sample standard deviations over three paired timing runs on the full dataset. The DART result on MMBench-EN at 90%, previously unreported in Table 10, is newly added.

MethodMMB 95%MMB 90%TextVQA 95%TextVQA 90%POPE 95%POPE 90%GQA 95%GQA 90%
Dense1.00001.00001.00001.00001.00001.00001.00001.0000
Random1.4080 ± 0.00031.8248 ± 0.00261.0775 ± 0.00111.2407 ± 0.00171.7117 ± 0.00261.8916 ± 0.00371.1902 ± 0.00251.4895 ± 0.0016
VisionZip1.5563 ± 0.00101.9409 ± 0.00231.3178 ± 0.00241.5271 ± 0.00331.6959 ± 0.00121.8441 ± 0.00071.1843 ± 0.00181.4617 ± 0.0016
FastV1.7115 ± 0.00131.8283 ± 0.00171.4062 ± 0.00211.5083 ± 0.00211.3293 ± 0.00151.6424 ± 0.00341.2339 ± 0.00301.4188 ± 0.0015
SparseVLM1.7404 ± 0.00161.8290 ± 0.00291.3275 ± 0.00081.4535 ± 0.00191.7357 ± 0.00241.8367 ± 0.00101.2205 ± 0.00231.5233 ± 0.0020
CDPruner1.3490 ± 0.00221.9114 ± 0.00251.0401 ± 0.00231.3543 ± 0.00281.9771 ± 0.00592.0299 ± 0.00590.9039 ± 0.00131.4201 ± 0.0013
DivPrune1.5323 ± 0.00161.9509 ± 0.00231.1964 ± 0.00191.5258 ± 0.00342.0796 ± 0.00562.1279 ± 0.00101.1810 ± 0.00051.6140 ± 0.0021
DART1.7217 ± 0.00161.7938 ± 0.00241.3292 ± 0.00291.4651 ± 0.00391.5850 ± 0.00031.7471 ± 0.00091.2615 ± 0.00171.5796 ± 0.0021
AuthorsOpenReview ↗

Response to Reviewer J39d [3/4]

Q2-2. […] do the selected PCS operating points meet the preservation target on held-out examples?

A. Thank you for your insightful question. The selected operating points largely meet the targets on held-out examples, with no systematic optimistic bias. Following your suggestion, we construct three random 50/50 splits of MMBench-EN for LLaVA-1.5-7B. In each split, we select the smallest budget that meets each target on the calibration half and then evaluate this fixed configuration on the test half. As shown in the table below, the results reveal that:

  • Held-out preservation closely matches calibration. Across all methods and both targets, calibration and test preservation differ by 0.15 points on average and by at most 0.26 points. In this sense, 11 of the 14 configurations meet their targets on the test half, and the remaining three fall short by only 0.03–0.10 points.
  • Selection introduces no systematic optimistic bias. Shifts from calibration to test occur in both directions, with averages of only −0.003 and +0.04 points at the 95% and 90% targets, respectively.

We will clarify in the revision that the reported PCS values describe the frontier observed on each evaluation set, and that this frontier largely transfers to held-out data.

Table III. Calibration and test preservation (%) on MMBench-EN for LLaVA-1.5-7B. Values are means across three 50/50 splits.

Method95% calibration95% test90% calibration90% test
Random95.2795.3490.0290.28
VisionZip95.2194.9690.0889.97
FastV95.1395.0990.1590.03
SparseVLM95.0295.1390.2790.09
CDPruner95.1795.2790.3390.26
DivPrune95.0894.9090.2590.49
DART95.2595.4290.1190.34

Discussion 1. Given the same pruning rate, do you believe the performance of different pruning methods can scale appropriately with the input image resolution?

A. Thank you for this insightful question. At a fixed pruning ratio, we expect performance to generally improve with resolution, since finer-grained tokens separate task-relevant details from the background, allowing selectors to choose them more accurately. However, the gains may differ across methods depending on how well each selector identifies these details. Even at higher resolutions, PAA-Bench can help examine such differences, as it compares methods on examples where token selection truly matters through Random-Pass@k.

Comments on the paper

1 comment

Dear Reviewers, ACs, SACs, and PCs,

We sincerely appreciate the valuable feedback provided by all reviewers and the additional efforts made by the ACs, SACs, and PCs. As the discussion period is drawing to a close, we would like to provide a summary of our rebuttal.

Reviewer 9Fs5 recognizes that our framework evaluates preservation, selection sensitivity, and actual speedup separately (Strength 1) and that our experiments across methods and backbones show that a method with higher overall accuracy can preserve a smaller share of the dense model’s correct predictions and that a method retaining fewer tokens can still run more slowly (Strength 2). Reviewer 9Fs5 is mainly concerned that difficulty groups defined by random masks may not reflect actual selection difficulty because these masks differ in distribution from those produced by individual selectors (W2). Across the models and token budgets we evaluated, gaps in preservation between pruning methods generally widened from Easy to Hard. Our plug-and-play Random-Pass@k defines shared difficulty groups for each model independently of any selector, so comparing methods on the same examples within each group makes differences in their ability to preserve the dense model’s correct predictions clearer (W1, W2, W4).

Reviewer Bxy8 recognizes that our evaluation distinguishes retaining correct predictions from achieving high final accuracy while measuring end-to-end speedup directly (Strength 1) and that our Random-Pass@k stratification reveals differences between selectors that aggregate scores obscure (Strength 2). Reviewer Bxy8 is primarily concerned that differences in scoring rules might explain why relative accuracy and preservation rank methods differently (W1, Q2). Under shared binary scoring within each benchmark (Q3), we found that wrong-to-correct transitions were frequent enough to reverse the ranking (Q1). Reviewer Bxy8 also asks whether group sizes and within-group method rankings remain stable when examples are regrouped using independent random trials (W2). Our within-group comparisons remained stable under independent regrouping, which yielded similar group sizes and preserved nearly all pairwise rankings of the pruning methods. We show that difficulty assignments (Q5, Q6) were also stable across the different pruning locations, with Random-Pass@k yielding similar groups when applied to visual tokens before the LLM and at several layers within it.

Reviewer J39d recognizes that our preservation metric exposes losses in the dense model’s correct predictions that overall accuracy can mask (Strength 1) and that our Random-Pass@k diagnostic identifies selection-sensitive examples without new difficulty annotations (Strength 2). Reviewer J39d is mainly concerned that reusing the same masks to define the difficulty groups and score Random may underestimate its performance on Hard examples and overstate the advantage of pruning methods (W1, Q1). When we rescored Random using independent masks, its scores changed little overall, and it still performed substantially worse than the pruning methods, particularly on Hard examples. Reviewer J39d also asks how PCS selects configurations and measures speedup to ensure a fair comparison across methods (W2, Q2). We compare all methods at the same preservation target, use the same binary search procedure and stopping rule to select each method’s token budget, and measure end-to-end speedup under identical timing conditions.

Overall, we introduce PAA-Bench, a plug-and-play framework for existing VQA benchmarks, to evaluate how well visual token pruning preserves model capabilities and accelerates inference. To prevent wrong-to-correct transitions from masking losses, we measure the fraction of the dense model’s correct predictions that remain correct after pruning. We use Random-Pass@k to group examples by how often the model answers them correctly under random pruning. We find that pruning-sensitive examples reveal larger differences in preservation across methods. Preservation-Constrained Speedup (PCS) captures costs that token counts overlook by comparing the highest measured end-to-end speedup each method achieves at a common preservation target. Because speed rankings can change with that target, practitioners can use PAA-Bench to identify the fastest evaluated method that meets their preservation requirement.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.