- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
Multimodal large language models pay a large amount of visual tokens for every image, and pruning is the standard way to control inference cost, with importance scores drawn either from image-side priors or from text-to-visual attention. The paper first asks whether these scores really change with the question, and finds that the early text-to-visual attention that is nominally question-conditioned stays almost identical across different questions and favors local detail, whereas visual priors favor global coverage and attention that can truly discriminate between questions appears only in the middle-to-late decoder layers. Based on this diagnosis the paper proposes a two-stage pruning framework that first builds a conservative visual prefix reusable across questions from two complementary early signals and then refines it per question within that prefix using a calibrated combination of middle-to-late attention signals. Across several open-source vision-language models and several visual question-answering benchmarks, the framework is compared with existing pruning methods under a matched cost budget, with ablations over the two stages and the signal sources, and it is reported to achieve a better accuracy-cost trade-off at low retention rates.
Critical strengths
- The diagnostic study of existing pruning signals is consistent across models, signals and evidence types, and is the strongest part of the paper.
- The paper provides unusually concrete reproduction and comparison material: the full calibration objective (Eq. 6-13), per-backbone Stage-2 mixture weights (p.14 Table 4), the calibration-source substitution study (p.9 Figure 6b: eight single-source and two/three/four-source substitutions, maximum deviation 0.016), and a baseline reproduction audit for 14 methods (p.17 Table 7, e.g., AgilePruner TextVQA 79.80 to 84.38, FastV A-OKVQA 77.00 to 84.33).
Critical weaknesses
- The paper's formatting is very poor, with too many lines that are less than 80% filled.
- On InViG, Q-Cal at 5-10% visual-token retention is above the unpruned full-token reference on all six backbones (up to about 67% relative gain), and the paper neither explains this column nor defines how it is scored.
- At the 5%/10% final budgets, on the same hardware and batch size, what are the measured prefill and decoding latency, throughput and memory of Q-Cal relative to the strongest cost-matched baseline (e.g., SparseVILA, ID-Selection)?
- At the six backbones, eight benchmarks and the same matched-C protocol of Table 2, what are the accuracies of (i) Stage 1 only (no Stage 2) and (ii) the best single deep-attention layer replacing the calibrated mixture, and what is the gain of the calibrated mixture over them? I would consider raising my score if the authors could address these issues.
Questions for authors
Please see weakness.
Response to Reviewer bahe: layout fixes, InViG bug fix, serving measurements, and matched-cost ablations
W1: Formatting and short lines
Thank you for pointing out this shortcoming. We readjusted the layout of the paper to avoid short lines. In the main text (pages 1–9), the number of body lines filled to less than 80% dropped from 23 in the submission to 8. All of them are ordinary paragraph endings or the lead-in of a list.
W2: InViG scores above Full
Thank you for spotting this anomaly; it indeed required an implementation audit. It was caused by a code bug: when the prefix KV cache was reused across the questions of an image, the attention mask was misaligned, which suppressed the scores of Full and of the baselines; InViG was affected the most.
We fixed the bug, unified the pipeline, and reran all cells for all methods. After the correction, Q-Cal no longer exceeds Full on InViG on any backbone. The revised values are in Table 2 (p. 8, l. 378) and Table 9 (p. 19, l. 1005), with all Avg. Fid. values recomputed. We apologize for the original evaluation errors.
W3: Measured latency, throughput and memory
Thank you for the suggestion. We strengthened the efficiency analysis with measured latency, throughput and memory. Appendix H (p. 21, l. 1130) reports prefill latency, decoding latency, throughput and peak memory of Full, Q-Cal and all eleven baselines on the nine backbones (the six Qwen-VL backbones and the three new ones), on the same hardware, at fixed batch sizes B=1,4,16,64 and both budgets (Tables 10–13, p. 23, l. 1188–p. 26, l. 1350). Table R6 summarizes the comparison with the strongest cost-matched baselines.
Table R6: Serving cost at B=64, mean over six Qwen-VL backbones (from Tables 10–13; 5% / 10%).
Q-Cal has a clear efficiency advantage over the full-token model: at B=64 it decodes 3.4$\times$ faster, reaches 2.2$\times$ the throughput and uses 1.5$\times$ less memory. Compared with SparseVILA and ID-Selection, Q-Cal is the most efficient in decoding, because its decoding cache holds only the final 5% or 10% of the visual tokens, whereas the one-shot baselines keep 13–16% to match the same cost C: at 5%, ID-Selection needs 15% and SparseVILA 9% more time per decoding step. Its throughput and memory are on par with these baselines (throughput within 5% of ID-Selection at every batch size). Its prefill is slower, since Q-Cal scores every question at layer S whereas ID-Selection prunes once before the language model.
W4: Stage 1 only and best single layer at matched cost
Thank you for the suggestion. We added to the appendix a complete C-matched comparison of Stage 1 only, the best single deep-attention layer and the calibrated mixture on the nine backbones and eight benchmarks at both budgets (Appendix I, p. 27, l. 1404; Table 14, p. 27, l. 1420). Stage 1 only keeps the Stage-1 selection with the number of visual tokens whose total cost matches C; the single layer keeps Q-Cal's pruning position S and final budget but scores the tokens with the best single layer. Table R7 summarizes the results; "Wins" counts the benchmark–budget cells (16 per backbone) in which each setting is the most accurate (ties counted for each).
Table R7: Avg. Fid. (%, 5% / 10%) and wins at matched cost (from Table 14).
In average fidelity, both the best single layer and the calibrated mixture are clearly better than Stage 1 only. The two Stage-2 signals reach similar average fidelity, but the calibrated mixture wins more cells than the best single layer (84 against 72 of 144). Introducing Stage 2 therefore improves pruning at the same cost, and the best single layer can serve as an approximate substitute for the calibrated mixture.
We hope these revisions address your concerns, and we are happy to answer any further questions.