- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces PerfPruner, a visual token pruning framework for multimodal large language models that scores tokens by measuring their effect on the likelihood of generating correct responses. It uses a small annotated dataset with likelihood-based supervision to train a lightweight predictor for test-time token importance estimation, and applies a score-constrained determinantal point process (SDPP) to reduce redundancy among retained tokens. The method is evaluated on 12 benchmarks spanning six tasks—safety, hallucination, general understanding, reasoning VQA, text-rich/OCR, and chart/diagram understanding—and is compared with prior attention-based and feature-similarity-based pruning approaches across various pruning ratios and image resolutions. The paper reports improvements over these baselines, including gains on Qwen2-VL-7B under large-resolution and native-resolution settings.
Critical strengths
S1: The authors propose PerfPruner, which ranks and prunes tokens according to their contribution to generating correct responses.
S2: The authors propose a score-constrained DPP to improve pruning efficiency, preserving token importance while ensuring token diversity and reducing redundant tokens.
S3: The authors conduct relatively comprehensive experiments to validate their conclusions.
Critical weaknesses
W1: Flawed Figures. a) Mismatch between figure content and subcaptions: The contents of Figure 1(a) and (b) appear to be swapped relative to the descriptions in the captions. b) Rendering errors in figure components: Some components in Figures 2 and 3 appear to have failed to render, appearing as a hollow rectangle with a question mark
W2: The stability of the importance prediction module requires further confirmation. The authors propose using a lightweight module to predict the importance of each token and train this module on a small subset. A natural concern is whether a module trained on such a limited dataset can correctly reflect the importance of each token across a wide range of test data. Table 8 ablates the size of the subset; however, it remains unclear how this training set is constructed: is a 12k training subset sampled from each dataset separately, or is a single 12k subset constructed jointly across all datasets? In addition, cross-training–evaluation validation could further verify the generalization of this module—for example, when testing on a given dataset, excluding its samples from the training set.
Questions for authors
Q.1: In Lines 184–185, is the probability in obtained by multiplying the values corresponding to the classification head for the ground truth?
Q.2: What is the role of in Eq. (4)?
Q.3: In the Empirical Analysis subsection of Sec. 3.2 (Lines 242–248), it is mentioned that pruning visual tokens using importance scores computed from ground-truth responses yields performance that even exceeds the full-token baseline. How can this gain be explained? Is there any risk of data leakage in this process?
4 comments
Response to Reviewer rzFq (2/2): Q1, Q2, Q3
Q1: In Lines 184–185, is the probability in obtained by multiplying the values corresponding to the classification head for the ground truth?
Q1 (how is computed). Yes. Under teacher forcing, we take the softmax output of the language-model head at each response position, read off the probability of the ground-truth token, and multiply these probabilities over all response tokens (computed as a sum of log-probabilities for numerical stability):
Because this product is extremely small for long responses, the value regressed by Lasso is its clipped log-odds, a monotone transform of the same probability:
We will state this explicitly in Sec. 3.1.
Q2: What is the role of in Eq. (4)?
Q2 (role of ). is the intercept of the Lasso regression. Eq. 4 fits the additive model
When no visual token is retained, the model gives . So absorbs the baseline level of the target that is common to all masks and cannot be attributed to any individual visual token, e.g., how likely the response already is given the question and the language prior. For any token , with the other tokens fixed,
so cancels and measures only the change caused by retaining .
Q3: In the Empirical Analysis subsection of Sec. 3.2 (Lines 242–248), it is mentioned that pruning visual tokens using importance scores computed from ground-truth responses yields performance that even exceeds the full-token baseline. How can this gain be explained? Is there any risk of data leakage in this process?
Q3 (why the oracle exceeds the full-token model; is there data leakage).
Why can oracle pruning improve performance? The oracle uses each evaluated sample's ground-truth response to estimate token importance. Within the fitted additive approximation, positive and negative coefficients indicate contributions that increase and decrease the target-response likelihood, respectively. Target-informed selection can therefore favor supportive tokens and suppress distracting ones, potentially correcting errors made with the full input. This explains why full-token performance is not a ceiling for oracle pruning. Improvements also occur without evaluation-answer access at inference: for example, PerfPruner improves CCBench from 63.53 to 64.51 at 512 retained tokens.
Does this introduce data leakage? Table 1 deliberately uses evaluation ground-truth responses for its oracle analysis. Its results illustrate the potential of target-informed pruning; they do not estimate deployable performance or establish a guaranteed optimum.
For practical PerfPruner evaluation, neither these sample-specific oracle scores nor the ground-truth responses are provided to the predictor. The predictor takes only the question and visual tokens and is trained on a separate 12K set, for which image-level and question-level checks detected no overlap with the evaluation sets. Table 1's oracle scores are not used as training labels for this predictor.
We will explicitly identify the oracle's access to evaluation responses and distinguish this diagnostic setting from practical inference in Sec. 3.
Thanks for your rebuttal. I have no further concerns.
Thank you for your positive feedback and for confirming that your concerns have been resolved. We truly appreciate your careful reading and constructive suggestions, which have helped us improve the clarity and quality of our work.
Response to Reviewer rzFq (1/2): W1, W2
We thank the reviewer for the careful reading and helpful comments.
W1(a) (Figure 1). Thank you for pointing out the ambiguity. The panels of Figure 1(a) and (b) are in the intended order, but the caption for (a) and (b) can be imprecise. Figure 1(a) reports "Rel." for different pruning ratios using large-resolution image inputs. Figure 1(b) reports "Rel." averaged over four pruning ratios for large- and native (small)-resolution inputs. We will revise the caption accordingly in the revision.
W1(b) (Figures 2 and 3). Thank you for pointing this out. The affected components are the mathematical symbols in the two figures, which were stored with an embedded math font that some PDF viewers do not display correctly. We will convert all text in both figures to vector outlines in the revision, so that they no longer depend on any font and render identically in every viewer.
W2 (training set construction and generalization). We would like to clarify how the training set is built, and we will make this explicit in Appendix B.2. It is a single 12K subset, constructed once and used to train a single predictor for evaluating all benchmarks and token budgets; it is not sampled from the evaluation datasets and has no overlap with the evaluation datasets.
The "OCRVQA" data in the 12K subset is from Infinity-MM, but it only shares its name with the OCRVQA benchmark and is not a split of that dataset; we will rename it as "OCRVQA-Infinity-MM" in the revision. We checked the complete 12K set against every reported evaluation set with image-level and question-level matching and found no overlap.
Therefore, the cross-training–evaluation setting suggested by the reviewer is already the setting of all our results: every benchmark is evaluated with a predictor that has seen none of its samples. Under this setting, the predictor generalizes well across all task categories as shown in Table R14.
Table R14: Generalization across task categories. Rel. (%) on Qwen2-VL-7B, averaged over the four token budgets (512 / 256 / 128 / 64). FastV is the strongest baseline in the main results of the paper at every budget.