acceptodds
Under review as a conference paper at ICLR 2027

Beyond Attention and Similarity: Performance-Grounded Visual Token Pruning

Abstract

In this paper, we propose a performance-grounded visual token pruning framework, termed PerfPruner, which estimates token importance directly from the likelihood change of correct responses. By explicitly measuring each token's contribution to generating the target response, PerfPruner eliminates the objective mismatch inherent in prior proxy-based pruning methods, including attention-based and feature similarity-based approaches. To enable efficient inference, we first annotate a small training set with likelihood-based supervision and then train a lightweight predictor to estimate token importance scores at test time. We further introduce a score-constrained determinantal point process (SDPP) to reduce redundancy among retained tokens and improve pruning efficiency. By aligning token scoring with the pruning objective, our method can remove a broader range of irrelevant tokens, including redundant tokens, unique distractors, and out-of-distribution visual noise, thus providing a more reliable criterion for token selection across diverse image resolutions, pruning ratios, and tasks. Extensive experiments on 12 benchmarks spanning six tasks—including safety, hallucination, general understanding, reasoning VQA, text-rich/OCR, and chart/diagram understanding—demonstrate that PerfPruner consistently outperforms existing pruning methods across various pruning ratios and image resolutions, establishing new state-of-the-art performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual Token Pruning, VLM

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rzFqOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces PerfPruner, a visual token pruning framework for multimodal large language models that scores tokens by measuring their effect on the likelihood of generating correct responses. It uses a small annotated dataset with likelihood-based supervision to train a lightweight predictor for test-time token importance estimation, and applies a score-constrained determinantal point process (SDPP) to reduce redundancy among retained tokens. The method is evaluated on 12 benchmarks spanning six tasks—safety, hallucination, general understanding, reasoning VQA, text-rich/OCR, and chart/diagram understanding—and is compared with prior attention-based and feature-similarity-based pruning approaches across various pruning ratios and image resolutions. The paper reports improvements over these baselines, including gains on Qwen2-VL-7B under large-resolution and native-resolution settings.

Critical strengths

S1: The authors propose PerfPruner, which ranks and prunes tokens according to their contribution to generating correct responses.

S2: The authors propose a score-constrained DPP to improve pruning efficiency, preserving token importance while ensuring token diversity and reducing redundant tokens.

S3: The authors conduct relatively comprehensive experiments to validate their conclusions.

Critical weaknesses

W1: Flawed Figures. a) Mismatch between figure content and subcaptions: The contents of Figure 1(a) and (b) appear to be swapped relative to the descriptions in the captions. b) Rendering errors in figure components: Some components in Figures 2 and 3 appear to have failed to render, appearing as a hollow rectangle with a question mark

W2: The stability of the importance prediction module requires further confirmation. The authors propose using a lightweight module to predict the importance of each token and train this module on a small subset. A natural concern is whether a module trained on such a limited dataset can correctly reflect the importance of each token across a wide range of test data. Table 8 ablates the size of the subset; however, it remains unclear how this training set is constructed: is a 12k training subset sampled from each dataset separately, or is a single 12k subset constructed jointly across all datasets? In addition, cross-training–evaluation validation could further verify the generalization of this module—for example, when testing on a given dataset, excluding its samples from the training set.

Questions for authors

Q.1: In Lines 184–185, is the probability in zj=p(y∣q,X(m(j)))z_j = p(y \mid q, X^{(m^{(j)})}) obtained by multiplying the values corresponding to the classification head for the ground truth?

Q.2: What is the role of β0\beta_0 in Eq. (4)?

Q.3: In the Empirical Analysis subsection of Sec. 3.2 (Lines 242–248), it is mentioned that pruning visual tokens using importance scores computed from ground-truth responses yields performance that even exceeds the full-token baseline. How can this gain be explained? Is there any risk of data leakage in this process?

4 comments
AuthorsOpenReview ↗

Response to Reviewer rzFq (1/2): W1, W2

We thank the reviewer for the careful reading and helpful comments.

W1: Flawed Figures. a) Mismatch between figure content and subcaptions: The contents of Figure 1(a) and (b) appear to be swapped relative to the descriptions in the captions. b) Rendering errors in figure components: Some components in Figures 2 and 3 appear to have failed to render, appearing as a hollow rectangle with a question mark

W1(a) (Figure 1). Thank you for pointing out the ambiguity. The panels of Figure 1(a) and (b) are in the intended order, but the caption for (a) and (b) can be imprecise. Figure 1(a) reports "Rel." for different pruning ratios using large-resolution image inputs. Figure 1(b) reports "Rel." averaged over four pruning ratios for large- and native (small)-resolution inputs. We will revise the caption accordingly in the revision.

W1(b) (Figures 2 and 3). Thank you for pointing this out. The affected components are the mathematical symbols in the two figures, which were stored with an embedded math font that some PDF viewers do not display correctly. We will convert all text in both figures to vector outlines in the revision, so that they no longer depend on any font and render identically in every viewer.

W2: The stability of the importance prediction module requires further confirmation. The authors propose using a lightweight module to predict the importance of each token and train this module on a small subset. A natural concern is whether a module trained on such a limited dataset can correctly reflect the importance of each token across a wide range of test data. Table 8 ablates the size of the subset; however, it remains unclear how this training set is constructed: is a 12k training subset sampled from each dataset separately, or is a single 12k subset constructed jointly across all datasets? In addition, cross-training–evaluation validation could further verify the generalization of this module—for example, when testing on a given dataset, excluding its samples from the training set.

W2 (training set construction and generalization). We would like to clarify how the training set is built, and we will make this explicit in Appendix B.2. It is a single 12K subset, constructed once and used to train a single predictor for evaluating all benchmarks and token budgets; it is not sampled from the evaluation datasets and has no overlap with the evaluation datasets.

The "OCRVQA" data in the 12K subset is from Infinity-MM, but it only shares its name with the OCRVQA benchmark and is not a split of that dataset; we will rename it as "OCRVQA-Infinity-MM" in the revision. We checked the complete 12K set against every reported evaluation set with image-level and question-level matching and found no overlap.

Therefore, the cross-training–evaluation setting suggested by the reviewer is already the setting of all our results: every benchmark is evaluated with a predictor that has seen none of its samples. Under this setting, the predictor generalizes well across all task categories as shown in Table R14.

Table R14: Generalization across task categories. Rel. (%) on Qwen2-VL-7B, averaged over the four token budgets (512 / 256 / 128 / 64). FastV is the strongest baseline in the main results of the paper at every budget.

MethodHallucinationGeneralReasoning VQAText-rich / OCRChart / Diagram
FastV82.1492.9192.3792.6977.56
PerfPruner (ours)87.6296.7895.8596.1389.73
Gain+5.48+3.87+3.48+3.44+12.17
AuthorsOpenReview ↗

Response to Reviewer rzFq (2/2): Q1, Q2, Q3

Q1: In Lines 184–185, is the probability in zj=p(y∣q,X(m(j)))z_{j}=p(y \mid q, X^{(\mathbf{m}^{(j)})}) obtained by multiplying the values corresponding to the classification head for the ground truth?

Q1 (how zjz_j is computed). Yes. Under teacher forcing, we take the softmax output of the language-model head at each response position, read off the probability of the ground-truth token, and multiply these probabilities over all TT response tokens (computed as a sum of log-probabilities for numerical stability):

p(y∣q,X(m(j)))=∏t=1Tp(yt∣y<t,q,X(m(j))).p\big(y \mid q, X^{(\mathbf{m}^{(j)})}\big) = \prod_{t=1}^{T} p\big(y_t \mid y_{ < t}, q, X^{(\mathbf{m}^{(j)})}\big).

Because this product is extremely small for long responses, the value regressed by Lasso is its clipped log-odds, a monotone transform of the same probability:

zj=clip⁡(logit⁡p(y∣q,X(m(j))),−50,50),logit⁡(p)=log⁡p1−p.z_j = \operatorname{clip}\Big(\operatorname{logit} p\big(y \mid q, X^{(\mathbf{m}^{(j)})}\big), -50, 50\Big), \qquad \operatorname{logit}(p) = \log\frac{p}{1-p}.

We will state this explicitly in Sec. 3.1.

Q2: What is the role of β0\beta_0 in Eq. (4)?

Q2 (role of β0\beta_0). β0\beta_0 is the intercept of the Lasso regression. Eq. 4 fits the additive model

z^(m)=β0+∑i=1Nβimi.\hat{z}(\mathbf{m}) = \beta_0 + \sum_{i=1}^{N} \beta_i m_i .

When no visual token is retained, the model gives z^(0)=β0\hat{z}(\mathbf{0}) = \beta_0. So β0\beta_0 absorbs the baseline level of the target that is common to all masks and cannot be attributed to any individual visual token, e.g., how likely the response already is given the question and the language prior. For any token xix_i, with the other tokens m−i\mathbf{m}_{-i} fixed,

z^(mi=1,m−i)−z^(mi=0,m−i)=βi,\hat{z}(m_i{=}1, \mathbf{m}_{-i}) - \hat{z}(m_i{=}0, \mathbf{m}_{-i}) = \beta_i ,

so β0\beta_0 cancels and βi\beta_i measures only the change caused by retaining xix_i.

Q3: In the Empirical Analysis subsection of Sec. 3.2 (Lines 242–248), it is mentioned that pruning visual tokens using importance scores computed from ground-truth responses yields performance that even exceeds the full-token baseline. How can this gain be explained? Is there any risk of data leakage in this process?

Q3 (why the oracle exceeds the full-token model; is there data leakage).

Why can oracle pruning improve performance? The oracle uses each evaluated sample's ground-truth response to estimate token importance. Within the fitted additive approximation, positive and negative coefficients indicate contributions that increase and decrease the target-response likelihood, respectively. Target-informed selection can therefore favor supportive tokens and suppress distracting ones, potentially correcting errors made with the full input. This explains why full-token performance is not a ceiling for oracle pruning. Improvements also occur without evaluation-answer access at inference: for example, PerfPruner improves CCBench from 63.53 to 64.51 at 512 retained tokens.

Does this introduce data leakage? Table 1 deliberately uses evaluation ground-truth responses for its oracle analysis. Its results illustrate the potential of target-informed pruning; they do not estimate deployable performance or establish a guaranteed optimum.

For practical PerfPruner evaluation, neither these sample-specific oracle scores nor the ground-truth responses are provided to the predictor. The predictor takes only the question and visual tokens and is trained on a separate 12K set, for which image-level and question-level checks detected no overlap with the evaluation sets. Table 1's oracle scores are not used as training labels for this predictor.

We will explicitly identify the oracle's access to evaluation responses and distinguish this diagnostic setting from practical inference in Sec. 3.

Thanks for your rebuttal. I have no further concerns.

Thank you for your positive feedback and for confirming that your concerns have been resolved. We truly appreciate your careful reading and constructive suggestions, which have helped us improve the clarity and quality of our work.

Reviewer_xK7xOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposed a training-based method for visual token pruning. By annotating the importance of tokens using likelihood and optimize a lightweight model to approximate the distribution of importance mask, the scoring criterion is directly aligned with the downstream objective of maintaining model performance under token pruning. Experiments on Qwen2-VL and LLaVA-1.5 on 10 benchmarks show that the proposed PerfPruner consistently outperforms baseline methods.

Critical strengths
  1. The paper explores an important limitation of existing visual token pruning methods: attention scores and feature similarities do not necessarily reflect the contribution of individual tokens to downstream predictions. The proposed likelihood-based supervision provides a more task-oriented approach to estimating token importance.
  2. The paper evaluates PerfPruner across 12 benchmarks, and the consistency of the improvements across different input resolutions supports the effectiveness of the proposed scoring approach.
Critical weaknesses
  1. No latency or FLOPs comparison.
  2. This paper is a training-based method, yet the comparison methods are all training-free. It would be better to compare with some training-based pruning methods, such as IF-Prune[1] and MQT [2].
  3. The analysis in Table 1 is interesting and insightful, but this observation cannot directly map to PerfPruner since the model used for likelihood measurement has a big gap: one is a 7B Transformer model, and another one is a small module.
  4. The likelihood-based pseudo-label generation is identical to the training objective defined in Section 3.1, which can be considered an unsupervised learning process. It would be reasonable to show an ablation study of only using Equation 1.

[1] Guohao Sun, Yufei Wang, Sizhuo Ma, Yuege Xie, Yuting Cheng, Zhiqiang Tao, and Jian Wang. If-prune: Information-flow guided token pruning for efficient vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2026.

[2] Hu, W., Dou, Z. Y., Li, L. H., Kamath, A., Peng, N., & Chang, K. W. (2024). Matryoshka query transformer for large vision-language models. Advances in Neural Information Processing Systems, 37, 50168-50188.

Questions for authors
  1. Is the training data annotator the same as the model used for token pruning? If the training data are model-specific, I’m wondering if scaling to different model architectures of different sizes will be difficult.
5 comments
AuthorsOpenReview ↗

Response to Reviewer xK7x (1/3): W1, W2

W1: No latency or FLOPs comparison.

W1 (latency / FLOPs). We thank the reviewer for the constructive feedback. We have already reported latency comparisons. Table 4 of the paper reports measured end-to-end latency, prefill time, KV-cache size and peak memory on Qwen2-VL-7B. We report measured costs rather than theoretical FLOPs, since they reflect the actual deployment overhead, including that of the predictor. Table R8 reproduces the results at 256 retained tokens.

Table R8: Efficiency on Qwen2-VL-7B at 256 retained tokens (from Table 4 of the paper). E2E latency is measured on the complete evaluation set, with the speedup over the full model in parentheses; KV cache is relative to the full model.

MethodMMEE2E latency (speedup)PrefillKV cache (%)Memory (GB)
Full (1296 tokens)2347.0216m21.9s (1.00×)10m16.5s100.015.94
DART2236.8813m42.1s (1.19×)7m03.0s28.316.27
CDPruner2166.4812m18.7s (1.33×)1m40.4s22.815.71
PerfPruner (ours)2339.2012m58.2s (1.26×)1m40.6s22.815.90

As shown in Table R8, PerfPruner has the same prefill time and KV cache as CDPruner and the highest accuracy. The overhead of the predictor and SDPP is included in the end-to-end latency, where CDPruner is slightly faster but much less accurate.

W2: This paper is a training-based method, yet the comparison methods are all training-free. It would be better to compare with some training-based pruning methods, such as IF-Prune[1] and MQT [2].

W2 (training-based baselines). We have already included comparisons with training-based method EvoComp. New comparison with IF-Prune is added.

We ran a new comparison with IF-Prune on Qwen2-VL-7B. Since its original training set (about 1.26M samples) and checkpoints are not public, we trained it with the official code and hyperparameters on our 12K training set, so both methods use the same data. Table R9 reports Rel. over the 10 benchmarks.

Table R9: Comparison with the training-based IF-Prune. Rel. (%) over 10 benchmarks on Qwen2-VL-7B; both methods are trained on our 12K set. "Pruning module" is the parameter count of the module that scores tokens.

Retained tokensPruning module51225612864
IF-Prune2.21B (Qwen2-VL-2B)98.6995.5591.2082.96
PerfPruner (ours)24.15M99.0596.5893.8287.11
Gain91× smaller+0.36+1.03+2.62+4.15

As shown in Table R9, PerfPruner is better at every budget with a pruning module about 91× smaller. Appendix F.2 of the paper additionally compares with the training-based EvoComp on LLaVA-1.5-7B; these results are reproduced in Table R10. MQT requires retraining the MLLM itself, whereas PerfPruner is a plug-in module for a frozen MLLM, so the two settings are not directly comparable; we will discuss it in the related work.

Table R10: Comparison with the training-based EvoComp on LLaVA-1.5-7B (Appendix F.2 of the paper). Rel. (%) over four benchmarks.

Retained tokens12864
EvoComp95.5292.19
PerfPruner (ours)96.6093.58
AuthorsOpenReview ↗

Response to Reviewer xK7x (2/3): W3, W4

W3: The analysis in Table 1 is interesting and insightful, but this observation cannot directly map to PerfPruner since the model used for likelihood measurement has a big gap: one is a 7B Transformer model, and another one is a small module.

W3 (Table 1 vs. the small module). We would like to clarify the roles of the two components, and we will state them more explicitly in Sec. 3. In PerfPruner, response likelihood is always measured by the MLLM being pruned (the 7B model); the small module never measures likelihood. The 7B model produces the token-importance labels offline (Sec. 3.1), and the small module is only trained to predict these labels from the question and visual tokens (Sec. 3.2), avoiding ground-truth responses and additional likelihood-evaluation passes at inference.

Table 1 therefore evaluates oracle pruning using MLLM-derived scores, whereas the main results evaluate practical pruning using the learned predictor. We agree that the oracle results alone do not establish how accurately the small module approximates these scores. It shows the great potential for measuring the importance of tokens with our proposed method.

W4: The likelihood-based pseudo-label generation is identical to the training objective defined in Section 3.1, which can be considered an unsupervised learning process. It would be reasonable to show an ablation study of only using Equation 1.

W4. We address the two possible readings of this suggestion.

(a) Use the likelihood-based score of Sec. 3.1 directly, without the predictor. This is the oracle setting in Table 1 of the paper, whose results are reproduced in Table R11. It needs the ground-truth answer and 256 forward passes per sample, so it cannot be used at inference; the predictor exists to remove both requirements.

Table R11: Using the likelihood-based scores directly, without the predictor (oracle setting; Table 1 of the paper). Rel. (%) over five benchmarks.

Pruning ratio60%80%90%
Qwen2-VL-7B112.59110.62105.69
LLaVA-1.5-7B138.76140.23136.25

(b) Train the predictor end-to-end on the response likelihood, without the Lasso labels. This would require back-propagating through the 7B MLLM at every training step and relaxing the discrete token selection into a differentiable one, which introduces a mismatch between soft selection in training and hard pruning at inference. Our two-stage design avoids both: the labels are computed once offline with forward passes only, and the predictor is then trained without the MLLM in the loop. The labels are also budget-agnostic rankings, so a single predictor serves all token budgets.

AuthorsOpenReview ↗

Response to Reviewer xK7x (3/3): Q1

Q1: Is the training data annotator the same as the model used for token pruning? If the training data are model-specific, I'm wondering if scaling to different model architectures of different sizes will be difficult.

Q1 (model-specific labels and scaling). Yes. The labels are generated by the same MLLM that is pruned, since token importance is defined by that model's response likelihood. Nevertheless, scaling to other models is not difficult.

Zero-shot transferability of the learned predictor. A predictor trained with a small model can be reused for a larger model of the same series. We applied the Qwen2-VL-7B predictor directly to Qwen2-VL-72B, without retraining it and without generating any labels with the 72B model. Table R12 reports Rel. over the 10 benchmarks, relative to the full-token 72B model.

Table R12: Transfer to a larger model: Qwen2-VL-72B. Rel. (%) over 10 benchmarks with large-resolution inputs, relative to the full-token Qwen2-VL-72B. PerfPruner uses the predictor trained with Qwen2-VL-7B, with no retraining and no labels from the 72B model. The last row is the same method on Qwen2-VL-7B (Table 2 of the paper), for reference.

Retained tokens51225612864
FastV92.4785.9477.9267.77
DART98.1094.7086.9276.01
PerfPruner (7B → 72B)99.3097.3593.4586.98
Gain over the best baseline+1.20+2.65+6.53+10.97
Reference: PerfPruner on Qwen2-VL-7B99.0596.5893.8287.11

As shown in Table R12, the transferred predictor is the best at every budget, and its margin grows from 1.20 to 10.97 points as the budget shrinks. It also preserves as much of the 72B model's performance as it does of the 7B model it was trained with. The label generation on a large model can therefore be avoided.

Generation across different architectures. For a model with a different architecture, we train a model-specific predictor. The pipeline is model-agnostic and cheap: label generation only needs forward passes of the target MLLM, the predictor is trained without the MLLM in the loop, and the whole process takes about 25 GPU-hours for a 7B model, once per model. Besides Qwen2-VL-7B and LLaVA-1.5-7B in the paper, we ran a new experiment on Qwen2.5-VL-7B, training a predictor on Qwen2.5-VL-7B labels with the same pipeline. Table R13 reports Rel. over the 10 benchmarks.

Table R13: Results on Qwen2.5-VL-7B. Rel. (%) over 10 benchmarks with large-resolution inputs. The predictor is trained on Qwen2.5-VL-7B labels. Avg. is the mean over the four budgets.

Retained tokens51225612864Avg.
FastV97.5190.5280.9670.8484.96
DART95.3189.9784.7377.3886.85
DivPrune97.8091.9683.9274.8387.13
CDPruner94.4087.9580.8473.3184.13
MMTok96.7293.4884.1573.2786.91
PerfPruner (ours)97.9095.4691.4785.4992.58

As shown in Table R13, PerfPruner is the best at every budget, and its margin over the strongest baseline grows from 0.10 to 8.11 points as the budget shrinks, which matches the trend on Qwen2-VL-7B and LLaVA-1.5-7B. We will add these results to the revision.

Reviewer_4NfzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PerfPruner learns to select visual tokens using supervision derived from target-response likelihoods. It fits a sparse linear model to randomized token-masking evaluations, distills the resulting importance rankings into a lightweight predictor, and applies score-constrained DPP for diverse subset selection. Experiments cover two MLLM architectures, multiple resolutions and token budgets, and twelve benchmarks. The reported gains are strongest under aggressive pruning, particularly on OCR and chart understanding tasks.

Critical strengths
  1. Strong empirical results under aggressive compression. On Qwen2-VL with 64 retained tokens, PerfPruner preserves 87.11% of average full-model performance, versus 75.84% for the strongest baseline in Table 2. Results across resolutions and a second architecture provide useful evidence beyond a single configuration.

  2. The ablations demonstrate contributions from both scoring and selection. Table 10 holds top-K selection fixed and shows substantial improvements from the learned scores; Tables 7 and 10 show further benefits from SDPP. The appendix also includes VisionZip and a training-based comparator, EvoComp, strengthening the experimental coverage.

Critical weaknesses
  1. The model-selection protocol leaves evaluation independence unclear. Appendix B.3 states that two benchmarks are used to select checkpoints but does not identify them or their splits. If reported evaluation sets inform checkpoint selection, the comparison with training-free baselines may be optimistic. Appendix B.2 explicitly excludes OCRVQA sample overlap but does not provide a corresponding cross-benchmark overlap analysis for the complete training mixture. These omissions prevent a confident assessment of generalization.

  2. The central claim of eliminating objective mismatch is stronger than the evidence. The method approximates response likelihood with an additive Lasso model at one masking ratio, then replaces coefficients with truncated rank-based supervision. This does not directly optimize generated-answer accuracy or account for token interactions. No ablation establishes the reliability of these labels or separates their benefit from learning on OCR-focused data using another supervision signal.

Questions for authors
  1. Evaluation independence: Which two benchmarks and splits were used for checkpoint selection? Were any reported evaluation examples or images used for training, validation, or hyperparameter selection? Please provide results selected using a disjoint validation set.

  2. Label construction and reliability: Is the regression target sequence probability, log-likelihood, or length-normalized log-likelihood? Does masking physically remove tokens or zero their embeddings as Eq. 2 suggests? How stable are the rankings and downstream results across mask seeds, sampling budgets, and retention ratios?

  3. Contribution and cost of supervision: How does the same predictor perform when trained on the same data using attention- or similarity-derived labels? Please also report pseudo-label generation GPU-hours, predictor parameter count, and training cost; the stated setup requires approximately 3.07 million masked-response evaluations.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 4Nfz (1/4): W1 / Q1

W1: The model-selection protocol leaves evaluation independence unclear. Appendix B.3 states that two benchmarks are used to select checkpoints but does not identify them or their splits. If reported evaluation sets inform checkpoint selection, the comparison with training-free baselines may be optimistic. Appendix B.2 explicitly excludes OCRVQA sample overlap but does not provide a corresponding cross-benchmark overlap analysis for the complete training mixture. These omissions prevent a confident assessment of generalization.

Q1: Evaluation independence: Which two benchmarks and splits were used for checkpoint selection? Were any reported evaluation examples or images used for training, validation, or hyperparameter selection? Please provide results selected using a disjoint validation set.

Re W1/Q1: Thank you for the careful reading. We answer each part of Q1 in turn.

Were evaluation examples or images used for training? No. The "OCRVQA" data we train on is an OCR-oriented subset of Infinity-MM (Stage 4) that only shares its name with the OCRVQA benchmark (Mishra et al., 2019); it is not a split of that dataset. We also checked the complete training set against every reported evaluation set with image-level and question-level matching and found no overlap. In the revision we will rename this subset as "OCRVQA-Infinity-MM" and add the overlap analysis to Appendix B.2.

Which benchmarks were used for checkpoint selection? HallBench and MME. We will state this explicitly in Appendix B.3. To remove any dependence on evaluation data, we re-selected the checkpoint on a disjoint validation set: 229 samples randomly drawn from the Infinity-MM subset, which is disjoint from evaluation benchmarks and the training data. Table R1 reports Rel. on Qwen2-VL-7B (large resolution) for both selections, over all 10 benchmarks and over the 8 benchmarks that played no role in either selection.

Table R1: Checkpoint selection on a disjoint validation set. Rel. (%) on Qwen2-VL-7B with large-resolution inputs. "Original selection" uses HallBench and MME; "disjoint-validation selection" uses 229 held-out samples drawn from the same source as the training data. FastV is the strongest baseline in the main results of the paper at every budget.

Retained tokens51225612864
All 10 benchmarks
FastV98.2294.3186.0775.84
PerfPruner, original selection99.0596.5893.8287.11
PerfPruner, disjoint-validation selection99.2297.1693.4787.83
8 benchmarks (excluding HallBench and MME)
FastV98.3394.5986.2576.75
PerfPruner, original selection99.1996.9793.7688.51
PerfPruner, disjoint-validation selection99.3097.5693.7788.41

Re-selection changes Rel. by only −0.35 to +0.72 points with no consistent sign, while the margin over FastV at 128 and 64 tokens remains 7.4 and 12.0 points. We will report these results and state the protocol in the revision.

Were evaluation examples used for hyperparameter selection? The only inference-time hyperparameter is the SDPP expansion factor α\alpha. In the paper, it was chosen from the ablation in Sec. 4.5 on the MME benchmark. We also repeated the selection on the new disjoint validation set, as reported in Table R2. As shown in Table R2, the disjoint validation set selects the same value, α=1.5\alpha = 1.5, so all reported results are unchanged.

Table R2: Selecting the SDPP expansion factor α\alpha on the disjoint validation set. Accuracy (%) on the 229 validation samples at three keep ratios.

α\alphaKeep 40%Keep 20%Keep 10%Mean
1.0 (w/o SDPP)85.1578.1765.5076.27
1.5 (default)87.3481.2266.3878.31
2.084.7280.3567.2577.44
3.083.8479.4865.0776.13
AuthorsOpenReview ↗

Response to Reviewer 4Nfz (2/4): W2, Q2 (label construction)

W2: The central claim of eliminating objective mismatch is stronger than the evidence. The method approximates response likelihood with an additive Lasso model at one masking ratio, then replaces coefficients with truncated rank-based supervision. This does not directly optimize generated-answer accuracy or account for token interactions. No ablation establishes the reliability of these labels or separates their benefit from learning on OCR-focused data using another supervision signal.

About claim of elimination of objective mismatch. Thanks for the constructive suggestion. We will revise "eliminates" to "mitigates" in the abstract and introduction, and state the approximation chain explicitly: response likelihood is a continuous surrogate for answer accuracy, and the Lasso model estimates each token's contribution to the response likelihood.

Regarding token interactions. Let z(m)=p(y∣q,X(m))z(\mathbf{m}) = p\big(y \mid q, X^{(\mathbf{m})}\big) denote the regression target, i.e., the likelihood of the target response yy when only the visual tokens selected by mask m\mathbf{m} are retained (Sec. 3.1). Because the masks are sampled independently, mi∼Bernoulli(r)m_i \sim \mathrm{Bernoulli}(r), the population least-squares solution of our regression has a closed form (up to the ℓ1\ell_1 shrinkage):

βi=Em−i[z(mi=1,m−i)−z(mi=0,m−i)],\beta_i = \mathbb{E}_{\mathbf{m}_{-i}}\left[ z(m_i{=}1, \mathbf{m}_{-i}) - z(m_i{=}0, \mathbf{m}_{-i}) \right],

where m−i\mathbf{m}_{-i} is the mask on all other tokens. That is, βi\beta_i is the marginal effect of retaining token xix_i averaged over random subsets of the other tokens, rather than a leave-one-out effect in the full context. Interactions with other tokens therefore enter the score through this average, although they are not modeled as explicit pairwise terms.

Q2: Label construction and reliability: Is the regression target sequence probability, log-likelihood, or length-normalized log-likelihood? Does masking physically remove tokens or zero their embeddings as Eq. 2 suggests? How stable are the rankings and downstream results across mask seeds, sampling budgets, and retention ratios?

(I) Label construction. For a mask m\mathbf{m}, we compute the teacher-forced log-likelihood of the complete ground-truth response,

ℓ(m)=∑t=1Tlog⁡p(yt∣y<t,q,X(m))=log⁡p(y∣q,X(m)),\ell(\mathbf{m}) = \sum_{t=1}^{T} \log p\big(y_t \mid y_{ < t}, q, X^{(\mathbf{m})}\big) = \log p\big(y \mid q, X^{(\mathbf{m})}\big),

summed over all TT response tokens, without length normalization. The Lasso target is its log-odds, clipped for numerical stability:

z(m)=clip⁡(logit⁡p(y∣q,X(m)),−50,50),logit⁡(p)=log⁡p1−p.z(\mathbf{m}) = \operatorname{clip}\Big(\operatorname{logit} p\big(y \mid q, X^{(\mathbf{m})}\big), -50, 50\Big), \qquad \operatorname{logit}(p) = \log\frac{p}{1-p}.

This is a monotone transform of the response likelihood p(y∣q,X(m))p(y \mid q, X^{(\mathbf{m})}) in Sec. 3.1. We use it because raw sequence probabilities are extremely small and compressed near zero, especially for long responses; the log-odds gives a well-scaled regression target across masks, and clipping prevents extreme values from destabilizing the regression. We will state this definition in Sec. 3.1.

(II) Masked tokens are physically removed from the sequence, exactly as at inference. We will revise Eq. 2, which currently reads as zeroing, to reflect this.

(III) Label stability. Please see the next comment (3/4).

AuthorsOpenReview ↗

Response to Reviewer 4Nfz (3/4): Q2 (label stability)

(III) Label stability (Q2, continued). All stability experiments follow the oracle setting of Table 1 in the paper, i.e., tokens are pruned directly with the label scores. We regenerated the labels with three mask seeds and report downstream results in Table R3, and ranking agreement and per-question outcome agreement in Table R4. Outcome agreement is the fraction of MME questions on which the pruned model is either correct under both seeds or incorrect under both.

Table R3: Stability of downstream results across mask seeds. Oracle setting on Qwen2-VL-7B (M=256M = 256, r=0.3r = 0.3): MME score and ChartQA accuracy at three keep ratios.

SeedMME (keep 10%)ChartQA (keep 10%)MME (keep 20%)ChartQA (keep 20%)MME (keep 40%)ChartQA (keep 40%)
Seed1 (default)2666.1278.442694.1985.802687.2788.32
Seed22668.5078.082707.5785.122663.8488.64
Seed32668.1778.162703.1686.082659.5988.40

Table R4: Agreement between mask seeds on MME. Spearman ρ\rho and top-kk overlap of the token rankings, and per-question outcome agreement of the pruned model. Chance level is the expected value for two random rankings.

MME, seed pairSpearman ρ\rhoTop-10% token overlapTop-20% token overlapOutcome agreement (keep 10%)Outcome agreement (keep 20%)Outcome agreement (keep 40%)
Seed1 vs. Seed20.31640.6%40.3%98.69%99.24%99.12%
Seed1 vs. Seed30.31940.6%40.3%98.44%98.95%99.16%
Seed2 vs. Seed30.31440.6%40.1%98.40%98.95%99.12%
Chance level010%20%–––

Downstream results are stable: scores vary by at most 1.1% across seeds, and the per-question outcome is identical on 98.4–99.2% of questions.

Token-level rankings agree moderately: the top-10% sets overlap by 40.6%, four times the chance level, and Spearman ρ\rho is 0.32. Together, these show that different mask seeds select partly different tokens that are functionally equivalent, which is consistent with the high redundancy among visual tokens. Moreover, the predictor is trained across 12K samples rather than on any single ranking, which further reduces the effect of per-sample variation.

Stability across the number of masks and the mask retention ratio. With seed1, we varied the number of masks MM and the mask retention ratio rr around the defaults (M=256M = 256, r=0.3r = 0.3); the results are reported in Table R5.

Table R5: Stability across the number of masks MM and the mask retention ratio rr. Oracle setting on Qwen2-VL-7B; one factor is varied at a time.

SettingMME (keep 10%)ChartQA (keep 10%)MME (keep 20%)ChartQA (keep 20%)MME (keep 40%)ChartQA (keep 40%)
Default (M=256M = 256, r=0.3r = 0.3)2666.1278.442694.1985.802687.2788.32
M=128M = 1282641.0276.602665.1984.802640.9787.16
M=512M = 5122655.0278.802730.6986.042709.5788.28
r=0.2r = 0.22681.2278.962674.4984.642650.1787.76
r=0.4r = 0.42609.3277.602641.0985.722645.3789.64

Results vary by at most 2.8% across MM and at most 2.7% across rr, at all three keep ratios. For MM, 128 masks give the lowest score in every column, while doubling from 256 to 512 changes the results by only −0.4% to +1.4%, so M=256M = 256 captures most of the benefit at half the cost. For rr, no value is best in all settings (r=0.2r = 0.2 at keep 10%, r=0.3r = 0.3 at keep 20%, mixed at keep 40%), so the default r=0.3r = 0.3 is not a sensitive choice.

Comments on the paper

2 comments

Dear Reviewers,

The authors have posted their rebuttals and updated their manuscript/responses. Please carefully read the authors' responses and engage in the discussion to clarify any remaining concerns.

In light of the rebuttal, please let the authors know whether your concerns have been adequately addressed, and update your review comments and scores if appropriate. Active participation is essential for us to reach a well-calibrated decision.

Please note that the author-reviewer discussion phase ends on November 18 at 11:59 PM AoE, after which no further responses can be posted. Thank you for your continued time and effort.

Best regards,

Area Chair

AuthorsOpenReview ↗

Overall Response

Dear Area Chair and Reviewers,

Thank you for your support throughout the discussion phase! We deeply appreciate the time and effort you have dedicated to reviewing and discussing our submission. We are glad that all three reviewers found their concerns addressed, and that Reviewer xK7x raised the score.

As the discussion phase comes to an end, we would like to summarize the efforts we have made during this period. In response to your suggestions, we conducted extensive additional experiments, including:

  • a new training-based baseline (IF-Prune);
  • two additional MLLMs (Qwen2.5-VL-7B, and zero-shot transfer to Qwen2-VL-72B);
  • an ablation that replaces our likelihood labels with attention-derived labels;
  • label-stability analysis and checkpoint re-selection on a disjoint validation set;

We are pleased that these experiments further validate our central claim: proxy signals such as attention and feature similarity are misaligned with the pruning objective, and grounding token importance in response likelihood leads to more reliable pruning. The gains hold across models, scales, and task categories, and grow as the token budget shrinks.

Once again, we sincerely appreciate the constructive discussions and the opportunity to refine our work. All new results and clarifications will be incorporated into the revised paper.

Best regards,

Authors of Paper 10581

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.