- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how visual token pruning changes the allocation of decoder attention between visual and textual tokens in vision-language models. It introduces Average Visual Token Attention (AVTA) and the Visual-to-Text Attention Ratio (VTAR), and observes that stronger pruning generally increases AVTA while decreasing VTAR. The authors also report a shift toward the no-image output distribution and cases in which downstream accuracy and teacher-forced next-token prediction loss decrease together. Motivated by these observations, they propose Causal Attention Recalibration (CAR), a training-free post-pruning framework that combines offline, task-specific Causal Attention Profiling with inference-time Plateau Attention Recalibration. Experiments include three VLMs, four pruning methods, and up to seven benchmarks. The reported gains reach 2.38 percentage points in normalized AvgAcc, with approximately 1% inference-latency overhead in the measured configuration, although improvements on LLaVA-NeXT are limited or negative.
Critical strengths
-
A well-motivated question that complements existing work. Most visual token pruning methods focus on which tokens to retain. This paper instead asks whether the decoder makes effective use of the tokens that remain, a question that has received little attention. CAR operates after token selection and adds no visual tokens, learned parameters, or KV-cache entries, with roughly 1% latency overhead in the measured configuration. It can therefore be combined with existing selectors, and it yields consistent gains across four selector families on LLaVA-v1.5 and Qwen2.5-VL.
-
Diagnostics that go beyond attention statistics. The Frozen control separates the structural effect of removing visual keys from the model's response to the shortened sequence. The KL analysis against the no-image output distribution provides behavioral evidence of reduced visual conditioning. The profiling procedure also identifies layers through interventions rather than attention magnitude, and the layer-selection ablation shows that this choice matters.
Critical weaknesses
-
The central causal claim, that pruning-induced VTAR reduction is harmful, is not established. Equation (6) shows that removing visual keys lowers VTAR when attention logits are held fixed, so a lower VTAR is largely an expected consequence of pruning. The Frozen comparison even shows that the model partially compensates. Treating the remaining gap from the full-token VTAR as an "imbalance" that should be corrected assumes that the full-token balance is the right target after removing tokens judged to be redundant, and the paper does not justify this assumption. The KL shift toward the no-image distribution and the accuracy–NTP relationship are consistent with reduced visual conditioning, but they cannot separate attention misallocation from the loss of visual information. No experiment isolates the effect of the attention ratio itself, for example by lowering VTAR with all visual tokens retained. Why this is consequential: the paper's main contribution is a diagnosis ("retained but underused"), and CAR is motivated entirely by it. If the VTAR reduction mainly reflects information loss, the diagnosis does not hold, and CAR's gains could instead come from a generic visual-attention boost of the kind existing methods already provide.
-
The empirical gains are small, concentrated in two benchmarks, and do not appear on the benchmark used to establish the phenomenon. In the headline LLaVA-v1.5 / VisionZip-32 setting, POPE (+9.61 F1, ≈1.60 points of AvgAcc) and ChartQA (+0.96, ≈0.75 points, amplified by its low normalization baseline) account for essentially all of the 2.38-point AvgAcc gain. GQA and MME decrease, and the remaining benchmarks change by at most 0.40 points. GQA accuracy decreases in 9 of the 12 LLaVA-v1.5 configurations, even though all Section 4 diagnostics are performed on GQA. Gains on LLaVA-NeXT are negligible or negative. No confidence intervals or variance across profiling sets are reported for downstream performance. No comparison is made with closely related visual-attention interventions such as PAI, which are known to be effective on POPE. Why this is consequential: the method has to stand on its empirical results. As reported, the evidence does not show a robust, broadly applicable improvement that can be attributed to correcting modality attention imbalance rather than to a benchmark-specific effect, and it does not show an advantage over existing alternatives. Together with the first weakness, this is the main reason for my recommendation of rejection.
Questions for authors
-
Is the VTAR reduction itself harmful, or does it reflect information loss? A residual gap from full-token VTAR does not by itself show that the retained evidence is underused, and the KL and accuracy–NTP analyses do not separate attention misallocation from information loss. Could you run a full-token control that lowers visual attention with layer-matched logit offsets until VTAR matches the pruned level, and compare the resulting accuracy and output-distribution shifts with those caused by pruning? Could you also report VTAR before and after CAR at a fixed retained-token set, relative to the full-token VTAR? Evidence of this kind would substantially change my assessment of the paper's central claim.
-
Where do the gains come from, and how robust are they? In the headline LLaVA-v1.5 / VisionZip-32 setting, nearly all of the 2.38-point AvgAcc gain comes from POPE (+9.61 F1) and ChartQA (+0.96). GQA and MME decrease, and GQA accuracy drops in 9 of the 12 LLaVA-v1.5 configurations, even though GQA is the primary benchmark in the Section 4 diagnostics. Why does the correction help least where the mechanism is characterized most directly? Could you report paired confidence intervals, or downstream results across independently sampled profiling sets, so that the smaller per-benchmark changes can be distinguished from noise?
-
How does CAR compare with existing visual-grounding interventions? PAI (Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs) also amplifies attention to image tokens to reduce reliance on language priors, and such methods are known to be effective on POPE, the benchmark that drives most of CAR's gain. Could you compare CAR with PAI, or at least its attention-amplification component, under the same pruning methods and token budgets? This would clarify what causal region selection and pruning-dependent scaling add beyond existing corrections.
6 comments
Answer 1: Response to Reviewer hLLG (Weakness 1 & Question 1) (2/2)
Experiment 2. Full-input attention suppression matched to pruned VTAR.
Following your suggestion, we next conduct a matched-VTAR comparison on CQA using LLaVA-v1.5-7B. For each VisionZip budget , we retain all 576 visual tokens and apply the visual-logit suppression introduced in Experiment 1, selecting to approximately match the all-layer aggregate VTAR measured under pruning. This control preserves the complete visual input while allowing us to compare attention suppression and token pruning at comparable aggregate VTAR levels.
Table A2. Full-input attention suppression matched to pruned VTAR.
| Condition | Visual Tokens | CQA Acc. (%) | All-layer VTAR | ||
|---|---|---|---|---|---|
| Full | 576 | 18.24 | 0.15 | 0.000000 | 0.667348 |
| VisionZip-128 | 128 | 16.92 | 0.14 | 0.039219 | 0.634668 |
| VisionZip-64 | 64 | 15.92 | 0.12 | 0.091000 | 0.595571 |
| VisionZip-32 | 32 | 14.52 | 0.10 | 0.161535 | 0.540809 |
| Full, target K=128 | 576 | 18.20 | 0.14 | 0.000068 | 0.660601 |
| Full, target K=64 | 576 | 18.04 | 0.12 | 0.002329 | 0.625594 |
| Full, target K=32 | 576 | 17.44 | 0.10 | 0.009600 | 0.580241 |
As shown in Tab. A2, pruning reproduces the pattern in Fig. 2: accuracy decreases, increases, and decreases. Full-input suppression at matched aggregate VTAR exhibits the same pattern, although the changes are smaller than under pruning. These results show that attenuating visual attention can itself impair performance and alter predictions without removing visual tokens. CAR targets this attention-related degradation rather than recovering lost visual information.
Experiment 3. VTAR before and after CAR at fixed retained-token sets.
Following your suggestion, we report VTAR before and after CAR on CQA while keeping the retained-token set fixed at each budget, using the full-token VTAR as a reference.
Table A3. VTAR before and after CAR at fixed retained-token sets.
| Condition | Visual Tokens | CQA Acc. (%) | All-layer VTAR | ||
|---|---|---|---|---|---|
| Full | 576 | 18.24 | 0.15 | 0.000000 | 0.667348 |
| VisionZip-128 | 128 | 16.92 | 0.14 | 0.039219 | 0.634668 |
| + CAR | 128 | 17.76 | 0.17 | 0.043976 | 0.671121 |
| VisionZip-64 | 64 | 15.92 | 0.12 | 0.091000 | 0.595571 |
| + CAR | 64 | 16.24 | 0.17 | 0.107121 | 0.656903 |
| VisionZip-32 | 32 | 14.52 | 0.10 | 0.161535 | 0.540809 |
| + CAR | 32 | 15.56 | 0.16 | 0.194036 | 0.636522 |
As shown in Tab. A3, with the retained-token set fixed, CAR restores the pruning-reduced VTAR and partially recovers the accuracy lost after pruning, with gains of 0.84, 0.32, and 1.04 percentage points at 128, 64, and 32 tokens, respectively. Meanwhile, increases at all three budgets, indicating that predictions move away from the no-image reference distribution.
(These revisions are highlighted in blue in Section 4 and Appendix E.2 and illustrated in Fig. 3 and Tabs. 28-29 of the revised manuscript.)
Answer 2: Response to Reviewer hLLG (Weakness 2 & Question 2)
We sincerely thank Reviewer hLLG for highlighting the concentration of gains, the discrepancy between our GQA diagnostics and downstream improvements, and the need for stronger evidence of robustness. We appreciate the opportunity to clarify and refine our conclusions, as detailed below.
1. [Clarification] Our diagnostic analysis is not limited to GQA
While Section 4 uses GQA for the main analysis, Appendix E.1 (Figs. 9–11) extends the diagnostics to CQA, MME, and SQA$^{\mathrm{I}}$ across LLaVA-v1.5-7B, Qwen2.5-VL-7B, and LLaVA-NeXT-7B. These analyses corroborate the core observations in Fig. 2 beyond GQA.
2. Gains are not restricted to POPE and CQA.
We agree that these two benchmarks account for most of AvgAcc. gain in the LLaVA-v1.5/VisionZip-32 setting. However, other model–task combinations also show substantial improvements. For example, on Qwen2.5-VL, CAR improves SQA$^{\mathrm{I}}$ accuracy by 1.88–3.52 percentage points across the eight pruning configurations in Tab. 2. CAR provides complementary improvements to pruning with negligible inference overhead (Tab. 5).
3. Performance depends on both the pruning configuration and the retained evidence.
The limited gains on GQA are not uniform across pruning methods or models: on Qwen2.5-VL, CAR improves GQA accuracy in seven of the eight configurations in Tab. 2. As a post-pruning intervention, CAR recalibrates the use of retained evidence rather than recovering discarded visual information. Its effectiveness therefore depends in part on the relevance and coverage of that evidence. Figs. 12–13 provide qualitative examples where relevant visual tokens remain but the pruned model answers incorrectly, and CAR restores the correct answer. We additionally provide comparative GQA examples for CDPruner and ResPrune in Figs. 6–8. Fig. 6 shows cases corrected by CAR under both methods, whereas Figs. 7–8 show cases corrected under only CDPruner or only ResPrune, respectively. For example, in Fig. 7’s first example, CDPruner retains more tokens covering the windows and the regions below them that provide cues about their height, whereas ResPrune provides more limited coverage. In Fig. 8’s first example, CDPruner retained tokens are concentrated mainly on the chair, whereas ResPrune retains tokens covering multiple cords relevant to the question. These examples illustrate how differences in retained evidence can affect recoverability: CAR can improve its use, while insufficient coverage of relevant regions may limit correction.
4. Downstream robustness across independently sampled profiling sets.
Tab. 12 already shows that layer selection is relatively stable across profiling seeds. Following your suggestion, we evaluate the layer regions obtained from the six profiling seeds in Tab. 12, keeping fixed to the paper’s setting for each task. Tab. A4 reports downstream means and standard deviations.
Table A4. Downstream performance under layer regions selected from six profiling seeds on LLaVA-v1.5-7B with VisionZip-32.
| Task | Baseline | Six-Seed Mean ± SD |
|---|---|---|
| GQA | 51.73 | 51.51 ± 0.07 |
| POPE | 68.76 | 78.47 ± 0.63 |
| MMB | 54.83 | 55.21 ± 0.03 |
| SQA$^{\mathrm{I}}$ | 68.86 | 68.39 ± 0.68 |
(These revisions are highlighted in blue in Appendix C.3 and D.3 and illustrated in Tab.15 and Figs. 6-8 of the revised manuscript.)
Answer 1: Response to Reviewer hLLG (Weakness 1 & Question 1) (1/2)
We sincerely thank Reviewer hLLG for carefully reading our paper and providing valuable feedback. We appreciate your concerns regarding our central causal claim. We acknowledge that our original analysis did not disentangle the effects of information loss and VTAR, which is indeed an important issue. Motivated by your insightful comments, we conducted three additional experiments to further substantiate the central claim of our paper.
Experiment 1. Effects of VTAR suppression with full visual input.
We use LLaVA-v1.5-7B on MME. To disentangle the effect of VTAR from visual information loss caused by pruning, we retain all visual tokens and adjust VTAR by applying a negative bias to visual attention logits: Zqj(l,h)=Zqj(l,h)−λ1[q∈Qtext]1[j∈V]. Here, Qtext and V denote textual query and visual key positions, respectively. The intervention is applied across all decoder layers.
We evaluate visual evidence reliance, task performance, and prediction distribution shifts under VTAR suppression. First, we measure JSD between the first-answer-token distributions before and after replacing 64 important visual token representations selected by VisionZip. Larger JSD indicates a stronger effect of replacement, reflecting greater reliance on the replaced visual evidence. Second, we normalize MME scores by their value at λ=0 to assess performance degradation. Finally, we compute KLfull and KLnoimg against fixed, unmodified full-image and no-image reference distributions to examine prediction shifts.
Table A1. Effects of VTAR suppression with full visual input.
As shown in Tab. A1, as VTAR decreases from 0.10141 to 0.06747, JSD decreases from 0.008873 to 0.008023, and relative MME performance drops from 100.00% to 96.49%. Meanwhile, KLfull increases while KLnoimg decreases, consistent with Fig. 2(c). These results show that reducing VTAR weakens reliance on visual evidence, degrades performance, and shifts predictions toward the no-image distribution, even with all visual tokens retained. In Section 4 of the revised manuscript, we visualize these results in Fig. 3, showing trends consistent with those observed in Fig. 2.