acceptodds
Under review as a conference paper at ICLR 2027

Retained but Underused: Modality Attention Imbalance after Visual Token Pruning

Abstract

Visual token pruning accelerates vision-language models (VLMs) by retaining only a small yet informative subset of image tokens. Most existing methods focus on token selection and assume that the decoder can effectively use the retained visual evidence. However, we find that pruning also changes the attention allocation between visual and textual modalities. As the token budget decreases, the average attention received by each retained visual token increases, while aggregate visual attention relative to textual attention decreases. We refer to this phenomenon as modality attention imbalance. Meanwhile, task accuracy and teacher-forced next-token prediction loss may even vary in the same direction. The output distribution also shifts toward the distribution produced without an image, indicating greater reliance on textual context. Simply rebalancing attention across all decoder layers fails to correct this imbalance. Through controlled causal interventions, we identify task-specific decoder regions where visual evidence is critical. These regions vary across models and tasks but transfer across pruning methods. Based on these findings, we propose Causal Attention Recalibration (CAR), a training-free post-pruning framework consisting of Causal Attention Profiling and Plateau Attention Recalibration. Causal Attention Profiling identifies decoder regions where visual evidence is causally important. Plateau Attention Recalibration applies a flat-top profile within these regions to rebalance visual and textual attention. Experiments across multiple VLMs, benchmarks, and representative pruning methods demonstrate performance gains of up to 2.38% with negligible computational overhead.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Models, Visual Token Pruning, Multimodal Attention

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_hLLGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how visual token pruning changes the allocation of decoder attention between visual and textual tokens in vision-language models. It introduces Average Visual Token Attention (AVTA) and the Visual-to-Text Attention Ratio (VTAR), and observes that stronger pruning generally increases AVTA while decreasing VTAR. The authors also report a shift toward the no-image output distribution and cases in which downstream accuracy and teacher-forced next-token prediction loss decrease together. Motivated by these observations, they propose Causal Attention Recalibration (CAR), a training-free post-pruning framework that combines offline, task-specific Causal Attention Profiling with inference-time Plateau Attention Recalibration. Experiments include three VLMs, four pruning methods, and up to seven benchmarks. The reported gains reach 2.38 percentage points in normalized AvgAcc, with approximately 1% inference-latency overhead in the measured configuration, although improvements on LLaVA-NeXT are limited or negative.

Critical strengths
  • A well-motivated question that complements existing work. Most visual token pruning methods focus on which tokens to retain. This paper instead asks whether the decoder makes effective use of the tokens that remain, a question that has received little attention. CAR operates after token selection and adds no visual tokens, learned parameters, or KV-cache entries, with roughly 1% latency overhead in the measured configuration. It can therefore be combined with existing selectors, and it yields consistent gains across four selector families on LLaVA-v1.5 and Qwen2.5-VL.

  • Diagnostics that go beyond attention statistics. The Frozen control separates the structural effect of removing visual keys from the model's response to the shortened sequence. The KL analysis against the no-image output distribution provides behavioral evidence of reduced visual conditioning. The profiling procedure also identifies layers through interventions rather than attention magnitude, and the layer-selection ablation shows that this choice matters.

Critical weaknesses
  1. The central causal claim, that pruning-induced VTAR reduction is harmful, is not established. Equation (6) shows that removing visual keys lowers VTAR when attention logits are held fixed, so a lower VTAR is largely an expected consequence of pruning. The Frozen comparison even shows that the model partially compensates. Treating the remaining gap from the full-token VTAR as an "imbalance" that should be corrected assumes that the full-token balance is the right target after removing tokens judged to be redundant, and the paper does not justify this assumption. The KL shift toward the no-image distribution and the accuracy–NTP relationship are consistent with reduced visual conditioning, but they cannot separate attention misallocation from the loss of visual information. No experiment isolates the effect of the attention ratio itself, for example by lowering VTAR with all visual tokens retained. Why this is consequential: the paper's main contribution is a diagnosis ("retained but underused"), and CAR is motivated entirely by it. If the VTAR reduction mainly reflects information loss, the diagnosis does not hold, and CAR's gains could instead come from a generic visual-attention boost of the kind existing methods already provide.

  2. The empirical gains are small, concentrated in two benchmarks, and do not appear on the benchmark used to establish the phenomenon. In the headline LLaVA-v1.5 / VisionZip-32 setting, POPE (+9.61 F1, ≈1.60 points of AvgAcc) and ChartQA (+0.96, ≈0.75 points, amplified by its low normalization baseline) account for essentially all of the 2.38-point AvgAcc gain. GQA and MME decrease, and the remaining benchmarks change by at most 0.40 points. GQA accuracy decreases in 9 of the 12 LLaVA-v1.5 configurations, even though all Section 4 diagnostics are performed on GQA. Gains on LLaVA-NeXT are negligible or negative. No confidence intervals or variance across profiling sets are reported for downstream performance. No comparison is made with closely related visual-attention interventions such as PAI, which are known to be effective on POPE. Why this is consequential: the method has to stand on its empirical results. As reported, the evidence does not show a robust, broadly applicable improvement that can be attributed to correcting modality attention imbalance rather than to a benchmark-specific effect, and it does not show an advantage over existing alternatives. Together with the first weakness, this is the main reason for my recommendation of rejection.

Questions for authors
  1. Is the VTAR reduction itself harmful, or does it reflect information loss? A residual gap from full-token VTAR does not by itself show that the retained evidence is underused, and the KL and accuracy–NTP analyses do not separate attention misallocation from information loss. Could you run a full-token control that lowers visual attention with layer-matched logit offsets until VTAR matches the pruned level, and compare the resulting accuracy and output-distribution shifts with those caused by pruning? Could you also report VTAR before and after CAR at a fixed retained-token set, relative to the full-token VTAR? Evidence of this kind would substantially change my assessment of the paper's central claim.

  2. Where do the gains come from, and how robust are they? In the headline LLaVA-v1.5 / VisionZip-32 setting, nearly all of the 2.38-point AvgAcc gain comes from POPE (+9.61 F1) and ChartQA (+0.96). GQA and MME decrease, and GQA accuracy drops in 9 of the 12 LLaVA-v1.5 configurations, even though GQA is the primary benchmark in the Section 4 diagnostics. Why does the correction help least where the mechanism is characterized most directly? Could you report paired confidence intervals, or downstream results across independently sampled profiling sets, so that the smaller per-benchmark changes can be distinguished from noise?

  3. How does CAR compare with existing visual-grounding interventions? PAI (Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs) also amplifies attention to image tokens to reduce reliance on language priors, and such methods are known to be effective on POPE, the benchmark that drives most of CAR's gain. Could you compare CAR with PAI, or at least its attention-amplification component, under the same pruning methods and token budgets? This would clarify what causal region selection and pruning-dependent scaling add beyond existing corrections.

6 comments
AuthorsOpenReview ↗

Answer 1: Response to Reviewer hLLG (Weakness 1 & Question 1) (1/2)

We sincerely thank Reviewer hLLG for carefully reading our paper and providing valuable feedback. We appreciate your concerns regarding our central causal claim. We acknowledge that our original analysis did not disentangle the effects of information loss and VTAR, which is indeed an important issue. Motivated by your insightful comments, we conducted three additional experiments to further substantiate the central claim of our paper.

Experiment 1. Effects of VTAR suppression with full visual input.

We use LLaVA-v1.5-7B on MME. To disentangle the effect of VTAR from visual information loss caused by pruning, we retain all visual tokens and adjust VTAR by applying a negative bias to visual attention logits: Z~qj(l,h)=Zqj(l,h)−λ 1[q∈Qtext]1[j∈V]\widetilde Z_{qj}^{(l,h)}=Z_{qj}^{(l,h)}-\lambda\,\mathbf 1[q\in\mathcal Q_{\mathrm{text}}]\mathbf 1[j\in\mathcal V]. Here, Qtext\mathcal Q_{\mathrm{text}} and V\mathcal V denote textual query and visual key positions, respectively. The intervention is applied across all decoder layers.

We evaluate visual evidence reliance, task performance, and prediction distribution shifts under VTAR suppression. First, we measure JSD between the first-answer-token distributions before and after replacing 64 important visual token representations selected by VisionZip. Larger JSD indicates a stronger effect of replacement, reflecting greater reliance on the replaced visual evidence. Second, we normalize MME scores by their value at λ=0\lambda=0 to assess performance degradation. Finally, we compute KLfullKL_{\mathrm{full}} and KLnoimgKL_{\mathrm{noimg}} against fixed, unmodified full-image and no-image reference distributions to examine prediction shifts.

Table A1. Effects of VTAR suppression with full visual input.

λ\lambdaVTARJSDRelative MME (%)KLfull\mathrm{KL}_{\mathrm{full}}KLnoimg\mathrm{KL}_{\mathrm{noimg}}
0.000.101410.008873100.000.0000000.399974
0.250.088880.00873499.300.0012900.383960
0.500.077600.00847597.290.0061370.364468
0.750.067470.00802396.490.0168480.341294

As shown in Tab. A1, as VTAR decreases from 0.10141 to 0.06747, JSD decreases from 0.008873 to 0.008023, and relative MME performance drops from 100.00% to 96.49%. Meanwhile, KLfullKL_{\mathrm{full}} increases while KLnoimgKL_{\mathrm{noimg}} decreases, consistent with Fig. 2(c). These results show that reducing VTAR weakens reliance on visual evidence, degrades performance, and shifts predictions toward the no-image distribution, even with all visual tokens retained. In Section 4 of the revised manuscript, we visualize these results in Fig. 3, showing trends consistent with those observed in Fig. 2.

AuthorsOpenReview ↗

Answer 1: Response to Reviewer hLLG (Weakness 1 & Question 1) (2/2)

Experiment 2. Full-input attention suppression matched to pruned VTAR.

Following your suggestion, we next conduct a matched-VTAR comparison on CQA using LLaVA-v1.5-7B. For each VisionZip budget K∈128,64,32K\in{128,64,32}, we retain all 576 visual tokens and apply the visual-logit suppression introduced in Experiment 1, selecting λ\lambda to approximately match the all-layer aggregate VTAR measured under pruning. This control preserves the complete visual input while allowing us to compare attention suppression and token pruning at comparable aggregate VTAR levels.

Table A2. Full-input attention suppression matched to pruned VTAR.

ConditionVisual TokensCQA Acc. (%)All-layer VTARKLfull\mathrm{KL}_{\mathrm{full}}KLnoimg\mathrm{KL}_{\mathrm{noimg}}
Full57618.240.150.0000000.667348
VisionZip-12812816.920.140.0392190.634668
VisionZip-646415.920.120.0910000.595571
VisionZip-323214.520.100.1615350.540809
Full, target K=12857618.200.140.0000680.660601
Full, target K=6457618.040.120.0023290.625594
Full, target K=3257617.440.100.0096000.580241

As shown in Tab. A2, pruning reproduces the pattern in Fig. 2: accuracy decreases, KLfullKL_{\mathrm{full}} increases, and KLnoimgKL_{\mathrm{noimg}} decreases. Full-input suppression at matched aggregate VTAR exhibits the same pattern, although the changes are smaller than under pruning. These results show that attenuating visual attention can itself impair performance and alter predictions without removing visual tokens. CAR targets this attention-related degradation rather than recovering lost visual information.

Experiment 3. VTAR before and after CAR at fixed retained-token sets.

Following your suggestion, we report VTAR before and after CAR on CQA while keeping the retained-token set fixed at each budget, using the full-token VTAR as a reference.

Table A3. VTAR before and after CAR at fixed retained-token sets.

ConditionVisual TokensCQA Acc. (%)All-layer VTARKLfull\mathrm{KL}_{\mathrm{full}}KLnoimg\mathrm{KL}_{\mathrm{noimg}}
Full57618.240.150.0000000.667348
VisionZip-12812816.920.140.0392190.634668
+ CAR12817.760.170.0439760.671121
VisionZip-646415.920.120.0910000.595571
+ CAR6416.240.170.1071210.656903
VisionZip-323214.520.100.1615350.540809
+ CAR3215.560.160.1940360.636522

As shown in Tab. A3, with the retained-token set fixed, CAR restores the pruning-reduced VTAR and partially recovers the accuracy lost after pruning, with gains of 0.84, 0.32, and 1.04 percentage points at 128, 64, and 32 tokens, respectively. Meanwhile, KLnoimg\mathrm{KL}_{\mathrm{noimg}} increases at all three budgets, indicating that predictions move away from the no-image reference distribution.

(These revisions are highlighted in blue in Section 4 and Appendix E.2 and illustrated in Fig. 3 and Tabs. 28-29 of the revised manuscript.)

AuthorsOpenReview ↗

Answer 2: Response to Reviewer hLLG (Weakness 2 & Question 2)

We sincerely thank Reviewer hLLG for highlighting the concentration of gains, the discrepancy between our GQA diagnostics and downstream improvements, and the need for stronger evidence of robustness. We appreciate the opportunity to clarify and refine our conclusions, as detailed below.

1. [Clarification] Our diagnostic analysis is not limited to GQA

While Section 4 uses GQA for the main analysis, Appendix E.1 (Figs. 9–11) extends the diagnostics to CQA, MME, and SQA$^{\mathrm{I}}$ across LLaVA-v1.5-7B, Qwen2.5-VL-7B, and LLaVA-NeXT-7B. These analyses corroborate the core observations in Fig. 2 beyond GQA.

2. Gains are not restricted to POPE and CQA.

We agree that these two benchmarks account for most of AvgAcc. gain in the LLaVA-v1.5/VisionZip-32 setting. However, other model–task combinations also show substantial improvements. For example, on Qwen2.5-VL, CAR improves SQA$^{\mathrm{I}}$ accuracy by 1.88–3.52 percentage points across the eight pruning configurations in Tab. 2. CAR provides complementary improvements to pruning with negligible inference overhead (Tab. 5).

3. Performance depends on both the pruning configuration and the retained evidence.

The limited gains on GQA are not uniform across pruning methods or models: on Qwen2.5-VL, CAR improves GQA accuracy in seven of the eight configurations in Tab. 2. As a post-pruning intervention, CAR recalibrates the use of retained evidence rather than recovering discarded visual information. Its effectiveness therefore depends in part on the relevance and coverage of that evidence. Figs. 12–13 provide qualitative examples where relevant visual tokens remain but the pruned model answers incorrectly, and CAR restores the correct answer. We additionally provide comparative GQA examples for CDPruner and ResPrune in Figs. 6–8. Fig. 6 shows cases corrected by CAR under both methods, whereas Figs. 7–8 show cases corrected under only CDPruner or only ResPrune, respectively. For example, in Fig. 7’s first example, CDPruner retains more tokens covering the windows and the regions below them that provide cues about their height, whereas ResPrune provides more limited coverage. In Fig. 8’s first example, CDPruner retained tokens are concentrated mainly on the chair, whereas ResPrune retains tokens covering multiple cords relevant to the question. These examples illustrate how differences in retained evidence can affect recoverability: CAR can improve its use, while insufficient coverage of relevant regions may limit correction.

4. Downstream robustness across independently sampled profiling sets.

Tab. 12 already shows that layer selection is relatively stable across profiling seeds. Following your suggestion, we evaluate the layer regions obtained from the six profiling seeds in Tab. 12, keeping β\beta fixed to the paper’s setting for each task. Tab. A4 reports downstream means and standard deviations.

Table A4. Downstream performance under layer regions selected from six profiling seeds on LLaVA-v1.5-7B with VisionZip-32.

TaskBaselineSix-Seed Mean ± SD
GQA51.7351.51 ± 0.07
POPE68.7678.47 ± 0.63
MMB54.8355.21 ± 0.03
SQA$^{\mathrm{I}}$68.8668.39 ± 0.68

(These revisions are highlighted in blue in Appendix C.3 and D.3 and illustrated in Tab.15 and Figs. 6-8 of the revised manuscript.)

Reviewer_VyywOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how visual token pruning changes the allocation of attention between visual and textual tokens. The authors observe that, although the average attention received by each retained visual token may increase, the total attention allocated to the visual modality decreases relative to text. They propose CAR, a training-free recalibration framework. CAR first uses Causal Attention Profiling (CAP) to identify the layers to recalibrate. Within these layers, it subtracts a constant determined by the pruning ratio and a parameter β from the attention logits associated with textual tokens, then reapplies softmax to increase the relative attention allocated to visual tokens. Experiments on models including LLaVA-1.5-7B, LLaVA-NeXT, and Qwen2.5-VL-7B show some performance improvements.

Critical strengths
  1. The observation in Figure 2(d) is interesting and provides a useful starting point for further investigation into model behavior after visual token pruning.

  2. The authors conduct a relatively detailed exploration of β, helping illustrate how recalibration strength affects performance.

Critical weaknesses
  1. The performance gains are limited and do not yet demonstrate a clear practical advantage; their dependence on token-selection quality also remains unclear. The paper motivates CAP by showing that uniform recalibration performs poorly, but the final improvements remain modest, particularly on Qwen2.5-VL-7B and LLaVA-1.5-7B. The reported results do not yet establish a clear and consistent advantage over pruning alone that justifies the additional layer-selection and parameter-tuning effort. If pruning fails to retain the visual tokens needed to answer a question, increasing the attention allocated to the remaining tokens may offer little benefit. It remains unclear how much of the performance degradation comes from removing essential visual information versus underusing the retained tokens.

  2. The conclusions in Figures 2(a–c) are largely apparent. These observations alone do not sufficiently explain why the proposed method is effective.

  3. The method is sensitive to β, and a broadly applicable setting remains unclear. The performance variation caused by changing β appears comparable to the gains from the method itself, making it unclear how much the final improvement depends on parameter selection. If β must be tuned separately for each dataset and model, this increases the effort required to use the method. The authors should clarify whether a single fixed β can be used across models and datasets and report the results under this setting.

Questions for authors
  1. Do the benefits of recalibration depend on retaining the relevant visual tokens? For examples where CAR provides little or no improvement, has the essential visual information already been removed by pruning? Could the authors compare methods such as CDPruner and ResPrune with and without CAR at matched token budgets, and examine representative successful and unsuccessful cases to clarify how token-selection quality affects the benefits of recalibration?

  2. Do the gains extend to text-centric tasks and other models? Could the authors provide representative results on benchmarks not already covered, such as OCRBench, OCRBench v2, or TextVQA? Evaluation on an additional suitable vision-language model from the Qwen3-VL or Qwen3.5 family would also help clarify the method’s applicability.

  3. How stable is layer selection, and what is the additional runtime cost? The selected layer regions have an IoU of approximately 0.8 across different seeds. How much does this variation affect final performance? With FlashAttention2 enabled, how does end-to-end inference latency compare between pruning alone and pruning with CAR? Please report the offline layer-selection time separately from inference latency with and without recalibration.

  4. If NTP loss does not reliably evaluate pruning quality, what should be used instead? The paper and Appendix E show that changes in NTP loss do not consistently align with changes in downstream performance. When selecting a pruning strategy or token budget, is there a more reliable proxy metric, or should evaluation rely on downstream task performance together with actual inference costs?

8 comments
AuthorsOpenReview ↗

Answer 1: Response to Reviewer Vyyw (Weakness 1 & Question 1)

We sincerely thank Reviewer Vyyw for carefully reading our paper and providing valuable feedback. W1 and Q1 raise important concerns about CAR’s practical benefits and its dependence on token-selection quality, both of which are central to our work. Motivated by your comments, we provide the following clarifications and additional experiments.

1. Benefits of CAR

Tab. 1 and Tab. 2 show that CAR improves average accuracy in most settings across different pruning methods and token budgets, with gains of 0.42%-2.38% on LLaVA-1.5 and 0.22%-1.38% on Qwen2.5. Following your suggestion, we compare CDPruner and ResPrune with and without CAR at matched token budgets, using the CAR configurations reported in the paper.

Table A1. Performance of CDPruner and ResPrune with and without CAR on LLaVA-1.5-7B at a retained visual-token budget of 32.

MethodGQAPOPEMMEMMBMMB$^{\mathrm{CN}}$SQA$^{\mathrm{I}}$CQAAvg. Acc.Δ\Delta Acc.
CDPruner57.0387.82166758.5948.6669.1614.6490.90%-
+ CAR57.0987.48165658.6648.5269.1615.5691.48%+0.58%
ResPrune56.4986.41162158.7249.8768.7714.4890.31%-
+ CAR56.5086.97162058.5250.1369.0614.8090.73%+0.42%

As shown in Tab. A1, CAR improves the aggregate score by 0.58% and 0.42%, respectively. CAR provides complementary improvements to pruning with negligible inference overhead (Tab. 5). The ablations in Tab. 4, Tab. 16 and Tab. 25 further validate the contributions of CAR’s key components. We have discussed the limited gains or performance drops on LLaVA-NeXT in Section 6.3. Moreover, for a fixed model-task pair, the selected layer range and β\beta can be reused across pruning methods without repeating the selection process. We address the associated costs in our response to Q3.

2. CAR’s dependence on token-selection quality.

CAR is a post-pruning calibration method designed to improve the utilization of retained visual tokens. It cannot recover information no longer represented after pruning, so its benefits depend on token-selection quality. Figs. 12-13 in the appendix provide qualitative examples where relevant visual tokens are retained but the model still answers incorrectly, and CAR restores the correct answer. Following your suggestion, we add comparative examples of CDPruner and ResPrune on GQA in Figs. 6-8. Fig. 6 shows cases where CAR corrects errors made by both pruning methods, while Fig. 7 and Fig. 8 show cases where it corrects only CDPruner or only ResPrune, respectively. For example, in Fig. 7’s first example, CDPruner retains more tokens covering the windows and the regions below them that provide cues about their height, whereas ResPrune provides more limited coverage. In Fig. 8’s first example, CDPruner retained tokens are concentrated mainly on the chair, whereas ResPrune retains tokens covering multiple cords relevant to the question. These examples illustrate that CAR can improve the use of retained visual information, whereas insufficient coverage of relevant regions may limit its ability to recover correct answers. These observations provide qualitative support for CAR’s dependence on retained information.

(These revisions are highlighted in blue in Appendix D.3 and illustrated in Tab. 19 and Figs. 6-8 of the revised manuscript.)

AuthorsOpenReview ↗

Answer 2: Response to Reviewer Vyyw (Weakness 2)

We thank the reviewer for pointing out that Fig. 2(a–c) alone do not sufficiently explain CAR’s effectiveness. We clarify below the rationale behind CAR and the evidence supporting it.

1. Why is recovery possible?

Performance degradation after pruning may arise from both the removal of useful visual information and the underutilization of information that remains available. Removing some visual tokens does not necessarily eliminate all evidence needed to answer a question. Even when relevant information is retained, it may contribute insufficiently to answer generation. CAR targets this recoverable component rather than reconstructing information lost through pruning.

2. What evidence supports this explanation?

To further distinguish information removal from underutilization, we conduct an additional experiment on LLaVA-v1.5-7B using MME. We retain all visual tokens and suppress visual attention by applying a negative bias to visual attention logits: Z~qj(l,h)=Zqj(l,h)−λ 1[q∈Qtext]1[j∈V]\widetilde Z_{qj}^{(l,h)}=Z_{qj}^{(l,h)}-\lambda\,\mathbf 1[q\in\mathcal Q_{\mathrm{text}}]\mathbf 1[j\in\mathcal V]. Here, Qtext\mathcal Q_{\mathrm{text}} and V\mathcal V denote textual query and visual key positions, respectively. This diagnostic intervention is applied across all decoder layers. Relative MME is normalized by the score at λ=0\lambda=0, and the KL divergences are computed against fixed, unmodified full-image and no-image reference distributions.

Table A2. Effects of VTAR suppression with full visual input on LLaVA-v1.5-7B.

λ\lambdaVTARRelative MME (%)KLfull\mathrm{KL}_{\mathrm{full}}KLnoimg\mathrm{KL}_{\mathrm{noimg}}
0.000.10141100.000.0000000.399974
0.250.0888899.300.0012900.383960
0.500.0776097.290.0061370.364468
0.750.0674796.490.0168480.341294

As λ\lambda increases, VTAR and MME performance decrease, while predictions move away from the full-input distribution and toward the no-image distribution. Thus, suppressing visual attention can impair performance even when all visual tokens remain available. In Section 4 of the revised manuscript, we visualize these results in Fig. 3, showing trends consistent with those observed in Fig. 2. This supports the distinction between information availability and information utilization, motivating CAR’s focus on improving the latter.

Importantly, this does not imply that higher VTAR is always better. Fig. 4 shows that all-layer recalibration yields inconsistent effects across tasks. The controlled ablations in Tab. 4 and Tab. 16 support the layer-selection strategy of Causal Attention Profiling, while Tab. 25 supports adapting recalibration strength to pruning severity. The qualitative recovery examples provide complementary illustrations (Figs. 6-8, Figs. 12-13). Together, these results support CAR’s design rationale.

(These revisions are highlighted in blue in Section 4 and illustrated in Fig. 3 of the revised manuscript.)

AuthorsOpenReview ↗

Answer 3: Response to Reviewer Vyyw (Weakness 3)

We sincerely thank the reviewer for highlighting the sensitivity to β\beta and the effort required for parameter selection. Tab. 24 examines this sensitivity: an insufficient β\beta may undercorrect the imbalance, whereas an excessively large β\beta may suppress textual attention too strongly and impair decoding. Following your suggestion, we additionally evaluate a fixed β=0.5\beta=0.5 across all seven benchmarks on LLaVA-v1.5-7B with VisionZip, keeping the selected layer ranges unchanged.

Table A3. Fixed versus task-specific β\beta on LLaVA-v1.5-7B with VisionZip. Scores are averaged over retained visual-token budgets of 32, 64, and 128.

βGQAPOPEMMEMMBMMB$^{\mathrm{CN}}$SQA$^{\mathrm{I}}$CQA
Fixed β=0.5\beta=0.554.7278.541686.6759.2252.0168.7816.13
Task-specific β\beta54.7281.281686.6759.2652.0169.0016.52

As shown in Tab. A3, β=0.5\beta=0.5 offers competitive performance on several tasks but is not uniformly optimal. The parameter is limited to a one-time offline selection for each model–task pair, after which β\beta is reused across token budgets.

(These revisions are highlighted in blue in Appendix D.7 and illustrated in Tab. 23 of the revised manuscript.)

Reviewer_xP4hOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates modality attention imbalance in vision-language models (VLMs) following visual token pruning. The authors observe that, although the average attention received by each retained visual token increases as the token budget decreases, the aggregate visual-to-text attention ratio (VTAR) declines, potentially causing the decoder to underutilize retained visual evidence. To address this issue, this paper proposes Causal Attention Recalibration (CAR), a training-free post-pruning framework consisting of two components: Causal Attention Profiling, which identifies task-specific decoder regions important for visual information processing through controlled interventions, and Plateau Attention Recalibration, which selectively adjusts attention logits according to pruning severity. Experiments across multiple VLM architectures, benchmarks, and pruning methods demonstrate improvements of up to 2.38 percentage points in normalized average performance with negligible inference overhead.

Critical strengths

(1) CAR is training-free and can be integrated with existing pruning methods without modifying the retained token set or introducing additional model parameters. Its lightweight attention-logit adjustment incurs minimal computational and memory overhead, making the approach practically attractive.

(2) By analyzing changes in visual attention allocation and prediction distributions, this paper studies an often-overlooked issue in visual token pruning: whether the decoder effectively utilizes the retained visual information.

Critical weaknesses

(1) Although this paper demonstrates that VTAR decreases with stronger pruning, this reduction is partly a mathematical consequence of removing visual tokens (Eq. (6)). A lower VTAR does not necessarily imply that the remaining visual evidence is inadequately utilized. Similarly, the observed shift toward the no-image output distribution may also reflect genuine information loss. While Causal Attention Profiling identifies decoder regions important for visual information, it does not directly establish that the pruning-induced VTAR reduction causes the observed performance degradation.

(2) CAR requires a labeled profiling set of 500 examples and a task-specific recalibration coefficient for each model–task pair. Although the resulting configurations transfer across pruning methods and token budgets, their transferability to unseen tasks or different data distributions remains unclear.

Questions for authors

Causal Attention Profiling identifies important decoder regions through progressively blocking visual attention from the output side. Since these interventions are cumulative, how can the authors distinguish the individual contributions of different layers from interactions among them? Would independently intervening on individual layers or layer groups lead to similar causal regions?

6 comments
AuthorsOpenReview ↗

Answer 1: Response to Reviewer xP4h (Weakness 1)

We thank Reviewer xP4h for the positive assessment and insightful suggestions. You pointed out that a lower VTAR does not necessarily imply less visual evidence use or worse performance. We agree that the paper lacks a direct examination, which is important for interpreting our findings. Motivated by your insights, we conducted the following experiment.

We use LLaVA-v1.5-7B on MME. To exclude visual information loss caused by pruning and the mathematical effect of removing visual tokens on VTAR, we retain all visual tokens and adjust VTAR by applying a negative bias to visual attention logits: Z~qj(l,h)=Zqj(l,h)−λ 1[q∈Qtext]1[j∈V]\widetilde Z_{qj}^{(l,h)}=Z_{qj}^{(l,h)}-\lambda\,\mathbf 1[q\in\mathcal Q_{\mathrm{text}}]\mathbf 1[j\in\mathcal V]. Here, Qtext\mathcal Q_{\mathrm{text}} and V\mathcal V denote textual query and visual key positions, respectively. The intervention is applied across all decoder layers.

Table A1. Effects of VTAR suppression with full visual input.

λ\lambdaVTARJSDRelative MME (%)KLfull\mathrm{KL}_{\mathrm{full}}KLnoimg\mathrm{KL}_{\mathrm{noimg}}
0.000.101410.008873100.000.0000000.399974
0.250.088880.00873499.300.0012900.383960
0.500.077600.00847597.290.0061370.364468
0.750.067470.00802396.490.0168480.341294

1. We use JSD to examine the relationship between VTAR and visual evidence use.

Specifically, we define J(λ)=1N∑iJSD⁡(PO,iλ,PS,iλ)J(\lambda)=\frac{1}{N}\sum_i\operatorname{JSD}(P_{O,i}^{\lambda},P_{S,i}^{\lambda}), where $P_{O,i}^{\lambda}$ and PS,iλP_{S,i}^{\lambda} denote the first-answer-token distributions before and after replacing the representations of 64 important visual tokens selected by VisionZip. Larger JSD indicates a stronger effect of replacement, reflecting greater reliance on the replaced visual evidence. As shown in Tab. A1, increasing λ\lambda from 0 to 0.75 reduces VTAR from 0.10141 to 0.06747, accompanied by a consistent decrease in JSD from 0.008873 to 0.008023. With all visual tokens retained, reducing VTAR weakens the effect of replacing the important visual representations. This reduced sensitivity supports a link between VTAR and reliance on visual evidence.

2. We report the MME score normalized by its value at λ=0\lambda=0 to examine the relationship between VTAR and performance degradation.

As shown in Tab. A1, despite retaining the full visual input, lowering VTAR reduces the relative MME score from 100.00% to 96.49%. This supports a link between VTAR and performance degradation.

3. We compute KLfullKL_{\mathrm{full}} and KLnoimgKL_{\mathrm{noimg}} against fixed, unmodified full-image and no-image reference distributions to test whether the shift toward the no-image distribution occurs without input information loss.

As shown in Tab. A1, decreasing VTAR leads to an increase in KLfullKL_{\mathrm{full}} but a decrease in KLnoimgKL_{\mathrm{noimg}}. This reproduces the pattern in Fig. 2(c), despite retaining all input visual tokens.

Together, these experimental results show that reducing VTAR weakens sensitivity to visual evidence, degrades task performance, and shifts predictions toward the no-image distribution, without removing any input visual tokens.

(These revisions are highlighted in blue in Section 4 and illustrated in Fig. 3 of the revised manuscript.)

AuthorsOpenReview ↗

Answer 2: Response to Reviewer xP4h (Weakness 2)

We thank Reviewer xP4h for or raising the question of transferring CAR configurations to unseen tasks or different data distributions. This is an important aspect of our study, which we clarify below.

[Clarification] CAR configurations are task-specific, and cross-task region transfer is not universally effective.

Fig. 4 shows that visual attention profiles differ across tasks, while Tab. 11 reports different layer regions identified by CAR for different tasks. Appendix D.6 and Tab. 22 directly evaluate cross-task region transfer and show that it is not universally effective. These findings motivate our task-specific profiling. Our transferability claim is therefore limited to different pruning methods and token budgets within the same model-task pair, rather than transfer to unseen tasks without additional profiling.

AuthorsOpenReview ↗

Answer 3: Response to Reviewer xP4h (Question 1)

We thank Reviewer xP4h for the question about layer interventions in Causal Attention Profiling. The manuscript did not compare cumulative interventions with independent layer-group interventions. We provide clarification and additional experimental results below.

1. [Clarification]

We progressively block text-to-visual attention from the output side and measure the additional performance drop caused by each newly blocked group. This design helps locate regions where the model relies on visual access. Blocking individual layers may produce only a small change because later layers can still retrieve visual information. Consistent with this motivation, Shi et al. [1] report that removing visual tokens at a single layer often produces changes too subtle to interpret, and therefore remove them from a selected layer onward. Similarly, Kaduri et al. [2] block visual attention from different starting layers onward to identify important stages of visual information transfer. Salazar et al. [3] further show that alternative pathways can compensate when a visual pathway is blocked. Our profiling therefore identifies regional visual dependence through conditional group contributions.

[1]Shi C, Yu Y, Yang S. Vision function layer in multimodal llms[J]. Advances in neural information processing systems, 2026, 38: 27510-27537.

[2]Kaduri O, Bagon S, Dekel T. What’s in the Imageƒ A Deep-Dive into the Vision of Vision Language Models[C]//2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025: 14549-14558.

[3]Salazar I, Frank S, Oneata D, et al. Pathways of Visual Information Flow in Vision-Language Models[J]. arXiv preprint arXiv:2607.03358, 2026.

2. [New Experiment] Independent layer-group interventions.

We independently intervene on each adjacent two-layer group Gk={L−k,L−k+1}G_k=\{L-k,L-k+1\}, leaving all other layers unchanged. We replace Eq. (11) with Cτind(k)=Pτ(0)−Pτind(k)C_\tau^{\mathrm{ind}}(k)=P_\tau(0)-P_\tau^{\mathrm{ind}}(k), where Pτind(k)P_\tau^{\mathrm{ind}}(k) is the accuracy when only GkG_k is intervened on. All other evaluation settings and region-selection rules remain unchanged.

Table A2. Comparison of regions selected by cumulative and independent interventions.

TaskCumulative InterventionIndependent InterventionIoU
GQA6–216–2388.89%
POPE8–196–1557.14%
MME6–156–15100.00%
MMB6–136–2736.36%
MMB$^{\mathrm{CN}}$6–136–13100.00%
SQA$^{\mathrm{I}}$6–116–733.33%
CQA24–296–2925.00%

As shown in Tab. A2, the selected regions overlap but also differ substantially across tasks. We further compare CAR performance on the five tasks with different selected regions using VisionZip-32, keeping all other settings fixed.

Table A3. CAR performance on tasks with different selected regions.

MethodGQAPOPEMMBSQA$^{\mathrm{I}}$CQAAvgAcc.Δ\DeltaAcc.
VisionZip51.7368.7654.8368.8614.5685.15%—
+ CAR (Cumulative)51.4678.3755.2369.0115.5288.52%+3.37%
+ CAR (Independent)51.4378.2752.4268.9113.2485.10%−0.05%

As shown in Tab. A3, independent profiling yields lower performance on all five tasks, reducing AvgAcc. from 85.15% to 85.10%. This supports cumulative profiling as a more effective approach to selecting regions for recalibration.

(These revisions are highlighted in blue in Appendix D.2 and illustrated in Tabs. 17-18 of the revised manuscript.)

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response: Summary of Revisions and Improvements

Dear Program Chairs, Senior Area Chairs, Area Chairs, and Reviewers,

We sincerely thank the Program Chairs, Senior Area Chairs, Area Chairs, and Reviewers for their time and constructive feedback. Your comments have helped us strengthen the evidence supporting our diagnosis, better justify CAR’s design, and clarify its applicability and practical costs. We appreciate the reviewers’ follow-up assessments recognizing that the additional experiments and clarifications address most of their major concerns. We have incorporated these findings into the revised manuscript, with major additions highlighted in blue. The main revisions are summarized below.

1. [New] Controlled Experiments Distinguishing Attention Effects from Visual Information Loss.

To address whether the observed VTAR reduction merely reflects token removal, we added three complementary experiments in Section 4 and Appendix E.2 (Fig. 3 and Tabs. 28–29). Together, these results support attention-related underutilization as a contributor to pruning-induced performance degradation and show that CAR can partially recover performance without restoring discarded visual tokens.

2. [New] Validation of Cumulative Causal Attention Profiling.

We added a discussion of three related studies to clarify why CAP measures conditional layer-group contributions through cumulative interventions. We additionally compared cumulative profiling with independent two-layer-group interventions in Appendix D.2 (Tabs. 17–18). These findings support cumulative profiling for selecting recalibration regions.

3. [New] Comparison with an Existing Attention-Amplification Method.

We added a comparison with PAI’s attention-amplification component (PAI-attn). The headline comparison appears in Section 6.3 (Tab. 6), with complete results in Appendix D.4 (Tab. 20). The results demonstrate CAR’s aggregate advantage over the evaluated attention-amplification baseline under pruning.

4. [New] Evaluation on Additional Pruning Methods, a Newer Model, and a Text-Centric Task.

We expanded the evaluation in Appendices D.3 and D.9: CDPruner and ResPrune (Tab. 19); Qwen3-VL-8B (Tab. 26); TextVQA (Tab. 27). These results extend the evidence for CAR’s applicability. We additionally provide comparative GQA examples in Figs. 6–8, illustrating how CAR’s effectiveness depends on the relevance and coverage of retained visual evidence.

5. [New] Downstream Stability and Shared-Parameter Evaluation. In Appendix C.3 (Tab. 13), we report downstream means and standard deviations using layer regions selected from six independently sampled profiling sets. In Appendix D.7 (Tab. 24), we additionally evaluate a fixed β=0.5\beta=0.5 across all seven benchmarks. These results support the stability of CAP and clarify the need to select recalibration strength for each model-task pair.

6. [New] Explicit Reporting of Offline and Inference Costs. We separately report the offline cost of CAP and CAR’s inference overhead with FlashAttention2 in Appendix C.4 (Tabs. 14-15). These measurements complement the efficiency results in Tab. 5 and make the practical costs of calibration and deployment more explicit.

7. [Clarification] Scope, Retained-Evidence Dependence, and Presentation. We clarified that CAR targets the underutilization of retained visual evidence and cannot reconstruct information discarded through pruning. We also distinguish configuration reuse across pruning methods and budgets from transfer to unseen tasks without calibration. The task-specific configurations and cross-task transfer analysis are documented in Appendix C.2 (Tab. 11) and Appendix D.6 (Tab. 22). We have incorporated the controlled intervention and its implementation into Section 4, added the PAI-attn comparison to the main paper, and organized the supporting experiments in the appendices. We also expanded the explanations in Section 5 to ensure that the methodology and its implementation details are clearly presented in the main paper.

Conclusion We sincerely thank the Program Chairs, Senior Area Chairs, Area Chairs, and Reviewers again for helping us improve the rigor, clarity, and empirical coverage of this work. We hope the revised manuscript and additional evidence provide a clearer account of when CAR is effective, why it helps, and what is required to apply it in practice.

Best Regards,

The Authors

Area_Chair_gpJEOpenReview ↗

ICLR 2027 Significance Reminder

Dear Reviewers,

Please review the authors' rebuttal to your comments and the other reviews and to respond.

Please remember to update your ratings and final justification if your concerns have been addressed. If ratings are not updated, clearly explain any remaining concerns in your final justification.

As a reminder, the discussion period ends on November 18th, 11:59 PM AoE.

Best, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.