acceptodds
Under review as a conference paper at ICLR 2027

CAR: Reducing Decoder Output Shift in Visual Token Pruning via Coverage Selection and Attention Reallocation

Abstract

Visual token pruning is an effective way to accelerate multimodal large language models (MLLMs). However, existing methods degrade substantially under aggressive pruning, especially on visual grounding (VG). We analyze the sources of this degradation from the perspective of the output shift that pruning causes in decoder layers, and decompose this shift in closed form into two terms, an image summary error and an attention allocation error. The summary error stems from the loss of visual information, and the allocation error from the drop in the share of attention assigned to visual tokens. To reduce both errors, we propose CAR (Coverage selection and Attention Reallocation), a token pruning framework that restores attention share while maintaining information. Specifically, to minimize the summary error, whose upper bound tightens as the retained tokens better cover all visual tokens, our coverage selection retains tokens that are both salient and non-redundant, with saliency from encoder heads chosen by a label-free probe. To minimize the allocation error, our attention reallocation reassigns the attention of the discarded tokens to the retained ones through a closed-form logit bias. On Qwen2.5-VL-7B with of the visual tokens pruned, CAR retains of the unpruned accuracy on visual question answering (VQA) and on VG, against for the best existing method on VG; the logit bias alone raises the VG accuracy of existing methods by up to points. Experiments on multiple MLLMs across VQA, VG and video understanding show that CAR generalizes well and consistently outperforms existing methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
token pruning, multimodal large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Yw16OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper decomposes the shift in the attention output caused by visual token pruning into the image summary error and the attention allocation error. To reduce errors, it proposes a saliency weighted coverage selection and a logit bias that restore the visual attention share, respectively. Experimental results on four backbones show that CAR outperforms existing methods consistently, especially on visual grounding under aggressive pruning.

Critical strengths
  • S1. This paper decomposes the shift in the attention output into the image summary error and the attention allocation error, identifying the loss of the visual attention share as a source of degradation. While the token selection and the logit bias are individually close to prior methods, this mathematical modeling provides a clear justification for each method.

  • S2. Each component of CAR targets its own error term, which Fig. 3 and Table 5 confirm. The bias reduces only the allocation error and the selection only the summary error, making the analysis and the design consistent.

Critical weaknesses

W1. Reduction of the actual output shift.

  • Fig. 3 shows the two error terms of Eq. 5 separately. Since both terms depend on vˉS\bar{v}_S , this does not guarantee a small total error ∥∣O′−O∥∣2/∥∣O∥∣2\||O'-O\||^2/\||O\||^2, which is not reported.
  • The error analysis considers only the attention output of each layer, given the input of the unpruned model, excluding the MLP and the errors accumulated over layers. The KL divergence between the output distributions over the vocabulary of the pruned and unpruned models would show whether CAR reduces the shift in the final output.

W2. Configurations separated by task.

  • CAR sets the spatial kernel and the form of the bias differently for grounding and question answering, assuming that the task family is known at inference time. The paper, however, provides no guideline for a query that asks both where an object is and what it is, or for the case where the task family is unknown.
Questions for authors
  • Q1. Nüwa (Huang et al., 2026) and RESTORE (Cho et al., 2026) take different position policies for the retained tokens. Nüwa favors the relative position mapping extension (RPME), which rescales the positions of the retained tokens to span the whole image, over keeping the original ones, while RESTORE keeps the original positions and compensates for the loss of visual attention with a calibration term. CAR follows RESTORE, so how do the authors interpret this choice for question answering and grounding, respectively?

  • Q2. CAR keeps the original positions and adds a logarithmic bias to the logits of the retained tokens, closely following the design of RESTORE. Why is this relation discussed and evaluated only in the appendix rather than in the main paper? Since it directly determines the novelty of the attention reallocation, it should be clearly stated in the main manuscript.

  • Q3. What are the total error ∥∣O′−O∥∣2/∥∣O∣∥2\||O'-O\||^2/\||O|\|^2 in Fig. 3 and the KL divergence between the output distributions over the vocabulary of the pruned and unpruned models?

  • Q4. How does CAR perform with a single configuration shared across grounding and question answering, and how does it handle a query that asks both where an object is and what it is?

  • Q5. The probe requires running the unpruned model on a probe set for each backbone and task family, which most training-free baselines do not need. Does CAR perform comparably to the probed head when the head is selected per input at inference time without calibration data?

5 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer Yw16 (1/4)

Thank you for your positive recommendation and insightful comments. Our responses to each point follow.

W1/Q3. Total output error and output KL

For every configuration of Fig. 3 (Qwen2.5-VL-7B, grounding, 9.9% retention), we report the total error ∑∥O′−O∥2/∑∥O∥2\sum\lVert O'-O\rVert^2/\sum\lVert O\rVert^2 (%) over layers 13–19, where ∥O′−O∥2=∥Esum∥2+∥Ealloc∥2+2⟨Esum,Ealloc⟩\lVert O'-O\rVert^2=\lVert\mathcal E_{\rm sum}\rVert^2+\lVert\mathcal E_{\rm alloc}\rVert^2+2\langle\mathcal E_{\rm sum},\mathcal E_{\rm alloc}\rangle is the sum of the image summary error, the attention allocation error and their cross term. We also report the KL divergence between the output distributions of the unpruned and pruned models over the vocabulary (Tables R1 and R2).

Total error. The bias lowers the total error of every method. CAR has the lowest total error both without the bias (16.7, against 18.5–24.2 for the existing methods) and with it (14.0, against 17.4–22.4). In the component ablation, the total error decreases from 16.8 for coverage selection alone to 14.0 for CAR, mainly through the bias (−1.5) and the selected head (−1.1).

Output KL. The bias also lowers the KL of every method. With the bias, CAR has the lowest KL (0.150, against 0.194–0.302 for the existing methods). In the component ablation, each component lowers the KL from 0.405 for coverage selection alone: the bias to 0.218, the saliency weights to 0.175, the selected head to 0.158, and the spatial kernel to 0.150 (CAR). Thus, CAR lowers not only the two terms of Fig. 3 but also their total and the shift in the final output.

Table R1. Total error (% of ∥O∥2\lVert O\rVert^2, layers 13–19) and output KL for the existing methods of Fig. 3 (Qwen2.5-VL-7B, grounding, 9.9% retention).

MethodTotal errorKL
w/o biasw/ biasw/o biasw/ bias
VisionZip19.117.40.3640.208
DivPrune20.217.90.3560.194
AnchorPrune19.517.50.3570.198
EADP19.217.50.3590.203
FastV21.120.60.4040.302
SparseVLM24.222.40.3250.290
HAWK21.119.80.3580.283
VScan18.717.40.3490.219
Nüwa18.517.80.3590.222
CAR16.714.00.4000.150

Table R2. Component ablation of Fig. 3 (Table 5 of the paper); setting as in Table R1.

ConfigurationTotal errorKLVG Acc.
Coverage16.80.40520.1
+ bias15.30.21850.0
+ saliency ww15.60.17554.8
+ head h∗h^*14.50.15862.5
+ spatial kernel (CAR)14.00.15066.8

We have added these results to Appendix C.16 of the revised manuscript.

AuthorsOpenReview ↗

Rebuttal for Reviewer Yw16 (2/4)

Q1. Position policy for the retained tokens

We first clarify Nüwa's position policy. In the Nüwa paper, RPME appears only in the analysis, where it is applied to VisionZip and FastV. The method itself does not remap positions: it preserves the global coordinate system through region partitioning, and its second stage follows FastV, which keeps the original positions. Its released code also passes the original position IDs to the language model. Nüwa, RESTORE, and CAR therefore all keep the original positions.

We also compare the two policies on CAR (Table R3): PESP keeps the original positions (CAR's default), and RPME stretches the retained positions along each spatial axis of Qwen2.5-VL's multimodal RoPE; selection and AR are unchanged. The two policies are within 1.0 point on all benchmarks. On grounding, PESP is slightly better (72.2 vs. 71.2): the model predicts boxes in absolute coordinates read from the RoPE row and column indices, which RPME shifts. On question answering, they are on par, since the answer depends on the content rather than its exact location. CAR therefore keeps the original positions, which need no remapping.

Table R3. Position policies on CAR (Qwen2.5-VL-7B; VG 9.9% retention, VQA 128 tokens).

Position policyRefCOCO valTextVQAChartQA
PESP (original, CAR)72.278.876.1
RPME (stretched)71.278.976.0

Q2. Relation to RESTORE in the main paper

Thank you for this suggestion. We have made this relation explicit in the main paper in two places (highlighted in blue).

  • Related Work. The new Related Work section (Section 4) includes a paragraph on attention calibration. It traces the log-count bias to ToMe and describes RESTORE, which merges the non-retained tokens into representatives and calibrates attention with group size and relative distance, as well as ERA. CAR differs in that it leaves the retained features unchanged, and its bias is set by the retention ratio or by the token counts of the coverage assignment.
  • Comparison. Section 3.2 now includes the comparison with RESTORE (Table 8), applying RESTORE's merging, its merging and bias, and our bias on top of DivPrune and WCS at the same final token budget. RESTORE's merging improves DivPrune on VQA but lowers accuracy with WCS and on grounding with both selectors, whereas our bias improves both selectors on both tasks, by 22.4 and 40.0 points on grounding. The detailed analysis remains in Appendix C.10.
AuthorsOpenReview ↗

Rebuttal for Reviewer Yw16 (3/4)

W2/Q4. Configurations separated by task

We evaluate one configuration shared by grounding and question answering, and test it on the MM-GCoT benchmark, whose queries ask both what an object is and where it is. This unified configuration fixes the spatial kernel and the capped per-cell bias: the kernel gains 4.3 on VG and costs 0.2 on VQA (Table 5 of the paper), and the capped per-cell bias is the best form on VQA and within 0.5 points of the best on VG (Table 7 of the paper). Its saliency averages the saliency of the two heads that the label-free head probe selects for VQA and VG. The probe runs once per backbone on the probe set, so no task label is needed at inference.

(1) Grounding and question answering. Against the task-specific configurations, the unified configuration loses 3.0 points on VG and 1.3 on VQA (Table R4). It still clearly outperforms the best VG baseline (63.8 vs. 22.8) and remains above the best VQA baseline (78.0 vs. 77.1).

Table R4. Single vs. task-specific configurations (Qwen2.5-VL-7B; VQA 128 tokens, VG 9.9% retention).

Method (VQA)TextVQAChartQAAI2DOCRBenchHallBenchMMEMMB-ENMMB-CNAcc.
Unpruned84.387.181.686756.22342.383.182.984.9
HAWK (best baseline)79.971.377.570548.82225.679.078.677.1
CAR, task-specific78.876.179.473149.02314.381.181.379.3
CAR, unified76.275.679.669548.52262.080.380.978.0
Method (VG)RefCOCORefCOCO+RefCOCOgAvg.
testAtestBvaltestAtestBvaltestval
Unpruned92.284.889.487.674.182.886.786.785.5
Nüwa (best baseline)27.523.726.221.817.420.523.022.022.8
CAR, task-specific75.466.772.267.255.962.367.866.566.8
CAR, unified71.962.769.464.352.259.765.664.363.8

(2) A query that asks both what and where. We use the MM-GCoT benchmark (CVPR 2026), where each question asks for the attribute or category of an object and for its box, in two settings (Table R5): the official answer-then-grounding protocol, which asks for the answer and then the box, and a single query asking for both. The unified configuration uses the saliency averaged over the heads selected by the VQA and VG probes, without any MM-GCoT data. The two task-specific configurations each favor one part of the query: the VQA configuration answers well but localizes poorly (G-Acc 25.6 and 30.9), whereas the VG configuration localizes well but answers less accurately with a single query (A-Acc 55.4 vs. 61.1). The unified configuration balances the two: its G-Acc (38.1 and 43.8) approaches that of the VG configuration (41.0 and 47.8), its single-query A-Acc (58.7) lies between the two, and its answer-box consistency is within 0.8 points of the VG configuration. All three remain well above the best baseline in localization (G-Acc 19.9 and 21.2).

We further run the head probe on 500 unlabeled questions from the MM-GCoT training set and compute the saliency from the head it selects. This variant gives the best pruned results on all three metrics under answer-then-grounding (64.5 A-Acc, 42.3 G-Acc, 39.7 Cons.) and the best G-Acc and consistency with a single query. It improves G-Acc over the unified configuration by 4.2 and 4.6 points and also exceeds the VG configuration in both settings.

Table R5. MM-GCoT (Qwen2.5-VL-7B, 9.9% retention). A-Acc: answer accuracy; G-Acc: box IoU > 0.5; Cons.: answer-box consistency. VQA/VG config.: the task-specific configurations. Bold: best pruned.

MethodAnswer, then groundingSingle query
A-AccG-AccCons.A-AccG-AccCons.
Unpruned70.963.456.166.170.356.9
VisionZip58.817.818.954.519.118.7
DivPrune63.619.920.058.821.220.9
AnchorPrune61.017.619.056.120.318.3
EADP58.516.417.652.018.516.9
Nüwa61.218.017.955.317.616.6
CAR, VQA config.63.825.624.061.130.927.3
CAR, VG config.64.341.038.155.447.838.1
CAR, unified62.138.137.358.743.837.4
CAR, unified, MM-GCoT probe64.542.339.759.048.439.6

Guideline. (i) When the task family is known, use its task-specific configuration. (ii) When it is unknown or the query is mixed, use the unified configuration. (iii) For a new task family with unlabeled inputs, a one-time probe on these inputs can further improve accuracy. We have added this guideline, the unified configuration, and Tables R4 and R5 to Appendix C.15 of the revised manuscript.

Reviewer_6HEbOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CAR, a training-free visual token pruning framework for MLLMs that addresses performance degradation at high pruning ratios, particularly in visual grounding. By analyzing Decoder Output Shift and decomposing it into Image Summary Error and Attention Allocation Error, CAR combines weighted coverage selection with attention reallocation. Experiments across multiple models and tasks demonstrate its effectiveness, achieving substantial improvements over existing methods at high pruning ratios.

Critical strengths

S1. A Somewhat Novel Analytical Perspective

Existing visual token pruning methods typically focus on token importance estimation, redundancy elimination, and token selection strategies. This paper further points out that pruning not only leads to visual information loss but also alters the attention allocation between visual and text tokens. This observation is valuable from a research perspective. In particular, at high pruning ratios, performance degradation on visual grounding tasks is often much more severe than on visual question answering tasks. This phenomenon is difficult to fully explain solely in terms of the semantic information preserved by the remaining tokens. By analyzing the proportion of visual attention across different decoder layers, the paper demonstrates how pruning affects visual attention allocation, providing a useful perspective for understanding model behavior after visual token pruning.

S2. Significant Experimental Results, Particularly on Visual Grounding

The most notable strength of this paper is its improvement in visual grounding performance under high pruning ratios. According to Table 2, when only 9.9% of visual tokens are retained, CAR achieves an average visual grounding accuracy of 66.8% on Qwen2.5-VL-7B, substantially outperforming the strongest baseline, Nüwa, which achieves 22.8%. Meanwhile, Table 1 shows that CAR is also highly competitive on visual question answering tasks. Furthermore, the authors validate the effectiveness of the attention reallocation mechanism by adding a uniform bias to existing pruning methods. This suggests that the performance gains do not rely entirely on the novel token selection strategy and provides empirical evidence supporting the authors' analysis of attention allocation error.

S3. Comprehensive Experimental Evaluation

The paper evaluates CAR on multiple multimodal large language models and covers a diverse range of tasks, including visual question answering, visual grounding, and video understanding. The authors also provide component ablations, comparisons of token selection strategies, comparisons of different bias formulations, and analyses of image summary error and attention allocation error.

Critical weaknesses

W1. Gap Between the Theoretical Coverage Bound and the Practical Optimization Objective

Proposition 1 derives a theoretical upper bound based on token similarity and attention in the decoder, whereas the actual algorithm selects tokens using feature similarity and attention from the encoder. The paper does not verify whether tokens that are similar or important in the encoder exhibit the same properties in the decoder. Moreover, the attention bias does not guarantee that the attention of pruned tokens is redistributed to the most similar retained tokens. Although Figure 3 shows that CAR reduces information loss in the decoder, direct evidence that the practical coverage objective aligns with the theoretical objective remains lacking. The authors should provide a correlation analysis between the two to validate this alignment.

W2. The Attention Mass Approximation Assumption Underlying Uniform Bias Lacks Validation

Section 2.3 assumes that the original attention mass allocated to retained tokens can be approximated by the token retention ratio k/Nk/N. However, CAR selects important tokens based on saliency and coverage, so the actual attention mass of retained tokens may be substantially higher than k/Nk/N, potentially leading to overcompensation by the uniform bias. The authors should analyze the deviation between the actual attention mass and this approximation across different decoder layers, attention heads, and input samples to validate the reasonableness of this assumption.

W3. Insufficient Generalization Evaluation on Next-Generation Multimodal Models

Although the paper evaluates CAR on relatively recent models such as Qwen3-VL-8B and InternVL3.5-8B, the detailed experiments and ablation studies are primarily conducted on Qwen2.5-VL-7B. Given the differences in visual token processing and attention mechanisms across architectures, the authors should provide comprehensive ablation studies on next-generation models such as Qwen3-VL and further evaluate the generalization capability of CAR on larger-scale models.

Questions for authors

Q1. Consistency Between the Theoretical and Practical Optimization Objectives

It would be helpful to provide correlation analyses of token similarity and attention between the encoder and decoder, and to clarify the relationship between the practical Coverage Objective and the theoretical Image Summary Error.

Q2. Validity of the Uniform Bias Approximation Assumption

How large is the deviation between the actual attention mass ρ\rho and k/Nk/N across different decoder layers, attention heads, and input samples? Further analysis of how this deviation affects the effectiveness of attention reallocation would help validate the proposed approximation.

Q3. Cross-Model Generalization

To what extent do Coverage Selection and Attention Reallocation remain effective on newer architectures such as Qwen3-VL-8B? Comprehensive component ablations on these models would provide stronger evidence of the method's generalization capability.

Q4. Does the Efficiency Evaluation Include the Full Pruning Overhead?

The paper reports CAR's prefill time, end-to-end latency, and FLOPs in the main efficiency table, while the appendix acknowledges that similarity computation and greedy selection introduce additional overhead for large inputs. Please clarify whether the reported efficiency results account for token selection, bias preparation, and associated data transfers. Reporting end-to-end latency across different input sizes and token retention ratios would further help assess the practical deployment benefits.

3 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer 6HEb (1/3)

Thank you for your positive recommendation and insightful comments. Our responses to each point follow.

W1/Q1. Consistency between the theoretical and practical optimization objectives

Proposition 1 bounds the image summary error with the similarity and attention of the decoder, whereas CAR selects tokens with those of the encoder. We relate the two on Qwen2.5-VL-7B, measuring decoder quantities in the unpruned model (300 grounding images from RefCOCO/+/g val and 300 VQA images from TextVQA and ChartQA).

Similarity. In layers 13–19, where the text attends most to the image, the Spearman correlation between the encoder cosine similarity of token pairs and their decoder similarity is 0.54/0.63/0.68 for values/keys/hidden states on grounding and 0.38/0.52/0.53 on VQA. Tokens similar in the encoder thus largely remain similar in the decoder.

Saliency. The head-averaged encoder saliency correlates moderately with the decoder attention in the shallow layers (Spearman 0.33–0.37 on grounding, 0.44 on VQA) and less in deeper layers, where attention depends more on the query. CAR does not need the saliency to match the decoder attention token by token, since the head probe selects the head by the agreement of pruned and unpruned outputs. The selected head focuses on task-relevant content: on RefCOCO/+/g, a token on the referred object receives on average 1.30× the saliency of the other tokens.

Coverage objective and summary error. We replace 10–75% of the tokens selected by CAR with random discarded tokens and measure the encoder objective FF and the image summary error in layers 13–19 (grounding, 9.9% retention; Table R1). As FF decreases, the summary error increases monotonically. The Spearman correlation between them, computed within each image, has a median of −0.52. A higher practical coverage objective thus corresponds to a lower image summary error, the quantity the bound controls.

Table R1. Coverage objective FF and image summary error as CAR's tokens are replaced with random discarded ones (Qwen2.5-VL-7B grounding, 9.9% retention).

Retained setCAR10% replaced25% replaced50% replaced75% replaced100% replaced (random)
FF0.6730.6530.6250.5760.5380.509
Summary error (%)10.2810.6211.1412.1712.7113.57

W2/Q2. Validity of the uniform bias approximation assumption

We agree that, since CAR retains salient tokens, the attention mass ρ\rho they receive before pruning exceeds k/Nk/N. We measured ρ/(k/N)\rho/(k/N) for every decoder layer, attention head and input sample on Qwen2.5-VL-7B grounding (RefCOCO val, 100 images, 9.9% retention; Table R2). ρ\rho is about twice k/Nk/N.

Table R2. ρ/(k/N)\rho/(k/N) of CAR's retained tokens in the unpruned model.

ρ/(k/N)\rho/(k/N)
Across layers1.5–3.6×
Across heads, within a layer1.3–3.4×
Across samples, within a layer1.7–2.5×
Overall mean2.2×

The concern is over-restoration: after the bias is added, the retained tokens would receive more visual attention than the whole image does in the unpruned model. Note that ρ\rho is measured with the hidden states of the unpruned model. Pruning and the bias change the inputs of every later layer, and with them the attention that the retained tokens actually receive, so the deviation of ρ\rho does not carry over unchanged to the pruned model. Appendix C.10 (Figure 5) of the paper already measures, at each layer of the pruned model, the visual attention share relative to that of the unpruned model. In layers 13–19, where the text attends most to the image, log⁡(N/k)\log(N/k) raises this ratio from 0.60 (without bias) to 0.87, below one; it exceeds one only in the shallow layers. Therefore, although ρ\rho is about twice k/Nk/N, log⁡(N/k)\log(N/k) does not over-restore the share in the layers that attend most to the image.

We also calibrate a separate bias for each layer (on 50 RefCOCO val images, solving the layers from shallow to deep so that the visual attention share of each pruned layer equals that of the unpruned model). The calibrated bias is smaller than log⁡(N/k)=2.31\log(N/k)=2.31 in the shallow layers (1.7–1.9 in layers 0–10) and larger in the deep layers (2.5–2.7 in layers 16–27), consistent with Figure 5, where the constant bias over-restores the shallow layers and under-restores the deep ones. It restores the share of every layer to within 3% of the unpruned model, yet on the eight RefCOCO splits it reaches 65.9, compared with 66.8 for log⁡(N/k)\log(N/k). Matching the share layer by layer therefore brings no gain, and the constant log⁡(N/k)\log(N/k) suffices in practice.

AuthorsOpenReview ↗

Rebuttal for Reviewer 6HEb (2/3)

W3/Q3. Cross-model generalization

Component ablation on Qwen3-VL-8B. We follow the cumulative protocol of the component ablation on Qwen3-VL-8B, adding facility-location selection, AR, saliency weighting, the probe-selected head and the spatial kernel in turn (Tables R3 and R4). Both components remain effective on Qwen3-VL: without the bias, facility-location selection improves over random selection by 5.2 points on grounding and 8.6 on VQA, and AR on top of it adds a further 2.1 and 0.8. Saliency weighting and the probe-selected head add 1.3 and 1.8 on grounding. On VQA, saliency averaged over heads does not help (−0.5), whereas the probe-selected head adds 3.6. As on Qwen2.5-VL, the spatial kernel improves grounding (+4.8) and leaves VQA unchanged (0.0), consistent with the task-specific configuration of the paper. We have added these results to Appendix C.9 of the revised manuscript.

Table R3. Component ablation on grounding with Qwen3-VL-8B (9.9% retention). FL: facility-location selection; Sal.: saliency weight; Head: probe-selected head; Sp.: spatial kernel.

FLBiasSal.HeadSp.RefCOCO testAtestBvalRefCOCO+ testAtestBvalRefCOCOg testvalAvg.Δ
Unpruned90.484.788.585.875.581.487.186.485.0–
✗ (random)✗–––39.431.534.133.525.728.329.328.631.3–
✗ (random)✓–––35.731.333.428.425.426.428.328.529.7−1.6
✓✗––✗45.534.839.039.428.233.536.235.836.5–
✓✓––✗46.339.541.538.430.234.139.339.638.6+2.1
✓✓✓✗✗46.038.743.838.732.536.441.741.940.0+1.3
✓✓✓✓✗48.241.845.639.633.737.843.644.041.8+1.8
✓✓✓✓✓ (CAR)53.748.751.444.338.941.947.046.846.6+4.8

Table R4. Component ablation on VQA with Qwen3-VL-8B (11.1% retention).

FLBiasSal.HeadSp.TextVQAChartQAAI2DOCRBenchHallBMMEMMB-ENMMB-CNAcc.Δ
Unpruned83.583.584.187457.92417.285.484.985.9–
✗ (random)✗–––48.732.569.939537.91863.071.372.858.2–
✗ (random)✓–––48.930.269.940240.71922.873.573.359.1+0.9
✓✗––✗67.938.174.751645.32115.475.575.266.8–
✓✓––✗67.438.874.452544.72137.078.177.367.5+0.8
✓✓✓✗✗68.540.773.549045.52097.677.576.367.0−0.5
✓✓✓✓✗ (CAR)72.246.875.756448.12183.778.277.970.6+3.6
✓✓✓✓✓70.050.476.355549.92136.778.277.470.60.0

Larger and newer models. We further evaluate CAR on Qwen3.5-9B and Qwen3.8-27B (Table R5).

Extending CAR to hybrid attention. Most decoder layers of both models are Gated DeltaNet (GDN) layers (24 of 32 in Qwen3.5-9B, 48 of 64 in Qwen3.8-27B). AR uses a logit bias to let retained tokens carry the attention mass of the removed tokens, which requires a softmax. We therefore apply AR only in the full-attention layers, whose softmax attention is the same as in the other backbones. A GDN layer has none: it writes the tokens one by one into a fixed-size recurrent state. After pruning, fewer visual tokens are written over a shorter timeline, so both the share of the image in the state and the proportions among its contents deviate from those of the unpruned model, which a logit bias cannot correct. We therefore extend the idea of AR to GDN layers: each representative explicitly writes on behalf of the removed tokens it covers.

GDN representative replay. Each removed visual token is represented by the retained token most similar to it, the same assignment that defines the cells of AR. Within each GDN layer, the projections are computed on the pruned sequence, expanded to the original length by filling each removed position with those of its representative, and passed through the convolution and the recurrence; only the outputs at the retained positions are then kept. Every other component, namely the full-attention layers, the MLPs and the KV cache, processes only the pruned sequence. Replay is applied only during prefill; decoding continues from the resulting GDN states without any change. The only overhead is the GDN scan over the removed positions during prefill, which on Qwen3.5-9B adds less than 2% to the FLOPs of the vision encoder and the prefill of plain deletion.

AuthorsOpenReview ↗

Rebuttal for Reviewer 6HEb (3/3)

W3/Q3 (continued).

Results. On grounding (9.9% retention), all baselines nearly collapse on both models: the strongest reaches only 7.4 on Qwen3.5-9B and 12.8 on Qwen3.8-27B, whereas CAR reaches 43.1 and 41.0, clearly outperforming all baselines. The component ablations follow the same trend on both models. The probe-selected head raises WCS from 6.6 and 15.1 to 20.3 and 23.3, GDN replay brings the largest gain, to 42.7 and 40.4, and AR adds a small further gain, reaching 43.1 and 41.0. Replay also transfers to other selection methods: on Qwen3.5-9B, combined with replay and AR, the five baselines rise from at most 7.4 to between 17.7 and 43.2. On VQA (11.1% retention), CAR outperforms the strongest baseline, EADP, by 5.0 and 4.7 points on the two models. Most of the gain comes from head selection, while replay and AR bring little further change. The full results for every split and benchmark, including all baselines and the baselines combined with replay, are in Appendix C.14 of the revised manuscript.

Table R5. CAR on Qwen3.5-9B and Qwen3.8-27B (grounding 9.9%, VQA 11.1% retention), with components added from left to right.

ModelTaskUnprunedStrongest baselineWCS (head avg.)+ Head sel.+ GDN replay+ AR (CAR)
Qwen3.5-9BGrounding87.67.4 (EADP)6.620.342.743.1
Qwen3.5-9BVQA87.064.7 (EADP)64.768.669.069.7
Qwen3.8-27BGrounding86.612.8 (DivPrune)15.123.340.441.0
Qwen3.8-27BVQA84.660.0 (EADP)60.165.365.264.7

Q4. Does the efficiency evaluation include the full pruning overhead?

The reported prefill time and end-to-end latency cover the whole forward pass and generation call, including token selection, bias preparation and the associated tensor transfers; only CPU preprocessing and the initial CPU-to-GPU input transfer are excluded, for all methods alike. The FLOPs in Table 4 count the vision encoder and the language model for all methods alike, excluding the selection step of every method; all additional steps of CAR, including the selection, add only about 0.6% (Appendix D.2).

Following the same efficiency protocol as in the paper, we evaluate the end-to-end latency of CAR across input sizes and retention ratios (Table R6). Each model uses the two retention ratios at which its accuracy is reported, and for N≥3,000N \geq 3{,}000 selection uses the incremental implementation in Appendix D. CAR is faster than the unpruned model in every setting, by 1.71–2.18× at the low ratio and by 1.30–1.57× at the high ratio, with all selection overhead included. We have added these clarifications and results to Appendix D.2 of the revised manuscript.

Table R6. End-to-end speedup of CAR over the unpruned model (NN: number of visual tokens).

ModelRatio (low / high)NUnpruned (ms)Speedup, low ratioSpeedup, high ratio
Qwen2.5-VL-7B (TextVQA)9.9% / 19.8%1,2965531.73×1.57×
Qwen2.5-VL-7B (TextVQA)9.9% / 19.8%2,9161,1621.76×1.42×
Qwen2.5-VL-7B (TextVQA)9.9% / 19.8%3,1361,2461.71×1.34×
Qwen2.5-VL-7B (TextVQA)9.9% / 19.8%4,0961,6081.73×1.34×
InternVL3.5-8B (OCRBench, 24 tiles)11.1% / 22.2%6,1442,1392.18×1.47×
Qwen3-VL-8B (Video-MME, 32 frames)9.9% / 25%11,5204,6071.95×1.30×

Thank you again for your valuable comments and suggestions. Please let us know if you have any further concerns, and we would be glad to address them.

Reviewer_qHPGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the severe performance degradation of visual token pruning in multimodal large language models (MLLMs) under aggressive pruning ratios, particularly on visual grounding tasks. The authors identify that the root cause is an output shift in the decoder's softmax attention: when visual tokens are discarded, the remaining tokens lose their original attention mass, causing their softmax weights and thus the weighted-sum output to deviate significantly from the unpruned result. To mitigate this, the paper proposes CAR (Coverage selection and Attention Reallocation), which comprises two complementary components: (1) Weighted Coverage Selection (WCS), a training-free greedy selector that retains tokens maximizing a saliency-weighted coverage objective—ensuring the retained set spans the visual information space while prioritizing salient tokens; and (2) Attention Reallocation (AR), a closed-form additive logit bias assigned to retained tokens that redistributes the attention mass lost by pruned tokens, provably minimizing the L2 distance between pruned and unpruned decoder outputs. Extensive experiments across multiple backbones (Qwen2.5-VL-7B, Qwen3-VL-8B, InternVL3.5-8B, LLaVA-1.5-7B) and tasks (VQA, visual grounding, video understanding) show that CAR consistently outperforms prior training-free pruning methods, with particularly dramatic gains on visual grounding (e.g., +40.0 absolute improvement on RefCOCO at 9.9% retention).

Critical strengths

The paper makes a compelling contribution by formally identifying decoder output shift as the root cause of performance collapse in aggressive visual token pruning. The proposed CAR method is elegant: WCS provides a principled, training-free token selection strategy, while AR offers a closed-form bias with a provable L2-optimality guarantee. The post-softmax recombination (PSR) implementation is practically efficient, adding only ~5.6ms latency. CAR achieves state-of-the-art results across four diverse backbones and multiple tasks without any training, with particularly dramatic gains on visual grounding (+40.0 absolute improvement on RefCOCO at 9.9% retention).

Critical weaknesses
  1. While CAR requires no gradient-based training, selecting the saliency head requires an exhaustive probe search over all 16 decoder heads (and encoder layers on some backbones) of the vision encoder, taking 1.4–6 hours on 7 RTX 3090 GPUs per backbone-task combination (Appendix C.3, Table 10). This is a costly search procedure that contradicts the spirit of "training-free" and limits practical deployability — especially compared to baselines like VisionZip or DivPrune that require zero such tuning.

  2. Table 25 shows wildly inconsistent gains from the AR bias: +40.0 on Qwen2.5-VL-7B (VG, 9.9% retention), +3.1 on Qwen3-VL-8B (VG), but −0.4 on InternVL3.5-8B (VG), where the authors disable the bias entirely. No diagnostic analysis is provided to explain why the bias fails on certain architectures (e.g., whether it stems from different projection layers, patch sizes, or attention patterns). This raises concerns about the generalizability of CAR and suggests the bias may need architecture-specific tuning, undermining the claim of a plug-and-play solution.

  3. The key result — that the uniform bias bⱼ = log(N/k) minimizes the L₂ output shift (Eq. 12) — assumes visual tokens are exchangeable, i.e., each independently carries 1/N of the attention mass. However, the WCS selector explicitly breaks this assumption by preferentially retaining salient tokens that carry more than average attention. The authors acknowledge this in Appendix B (Remark on E[ρ] = k/N): "The identity assumes that the visual tokens are exchangeable, which the selector does not respect, since it retains the tokens that carry the most saliency." Consequently, the bias may over-restore or under-restore attention mass, and the L₂ optimality guarantee does not rigorously hold in practice.

  4. CAR is not evaluated on newer models like Qwen3.5-9B or Qwen3.8-27B. Since these models may differ in architecture (e.g., hybrid attention mechanisms, encoder design), the method's generality on such models remains unverified.

  5. Lack of some latest relevent citations: [1] Chen, T., Tan, Z., Ye, Z., Wu, Y., Tu, X., Ren, J., ... & Ye, J. (2026). SPIDER: Multi-Layer Semantic Token Pruning and Adaptive Sub-Layer Skipping in Multimodal Large Language Models. IEEE Transactions on Pattern Analysis and Machine Intelligence. [2] Wang, T., Guo, Y., Zhang, Q., Zhang, Y., Ouyang, W., Huang, R., ... & Zhang, S. (2026). MiCo: Mutual Information Coverage Optimization through Semantic Erasure Modeling for Efficient MLLM Inference. arXiv preprint arXiv:2609.34330. [3] Li, S., Hu, Z. Y., Huang, S., Li, Y., Zhong, Y., & Wang, L. (2026). SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Models. arXiv preprint arXiv:2609.01004.

Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer qHPG (1/4)

Thank you for the detailed and constructive comments. Our responses to each point follow.

W1. Cost of the head probe

The probe set consists of 500 RefCOCO images for grounding and 500 samples from each of four benchmarks for VQA (Appendix C.3). The probe cost reported in the paper comes from an unoptimized implementation, which reloaded the model and restarted the evaluation for every candidate head. Running the same probe with the model loaded once per GPU selects the same heads and reduces the cost on Qwen2.5-VL-7B from 9.8 to 3.9 RTX 3090 GPU-hours for grounding and from 42 to 6.0 GPU-hours for VQA, i.e., about 0.65 and 1.0 hours of wall-clock time on six GPUs. We have updated the probe cost in Appendix C.3 of the revised manuscript. The cost scales linearly with the size of the probe set, and a smaller set suffices: with 100 samples per benchmark, the selected heads are on average within 0.2 points in accuracy on grounding and 0.6 points on VQA of those selected with the full set, at 0.77 and 1.2 GPU-hours, down from 3.9 and 6.0 with the full set.

A one-time offline step is common among training-free acceleration methods: VTW (AAAI 2025) selects the layer at which visual tokens are withdrawn, FitPrune (AAAI 2025) derives its pruning recipe from attention statistics, and D²Pruner (AAAI 2026) calibrates a positional bias prior. CAR's probe likewise uses only forward passes and no labels, and it runs once per backbone and task family. The selected head is reused across benchmarks and retention ratios. The component ablation (Table 5) shows that the selection is worthwhile, as the probe-selected head improves grounding accuracy by 7.7 points, and that even without both the probe and the spatial kernel, CAR with the head-averaged saliency reaches 54.8, far above the strongest grounding baseline in Table 2 (Nüwa, 22.8).

W2. Why the bias is disabled on InternVL3.5-8B

AR restores the visual attention that pruning removes. On InternVL3.5-8B, pruning mainly removes attention from the high-resolution tiles, while the thumbnail of the whole image, which the model relies on most, keeps its share on average. Since the thumbnail largely covers the content of the tiles, AR has little useful to restore. We analyze this in Appendix C.7.

InternVL3.5-8B splits an image into up to 6–24 tiles, depending on the benchmark, and appends a 256-token thumbnail as its only global view. In the unpruned model, the thumbnail holds 16% of the visual tokens but receives 36% of the attention that the text pays to the image. Following IVC-Prune (ICLR 2026), all methods therefore keep half of the thumbnail tokens and assign the rest of the budget to the tiles, which lose more than 90% of their tokens at the lowest retention ratios (9.9% on VG and 11.1% on VQA). After pruning to 11.1% on VQA, the attention share of the thumbnail rises to 1.26 times its unpruned value, while that of the tiles falls to 0.25 (Appendix C.7).

A single log⁡(N/k)\log(N/k) for all visual tokens ignores this imbalance: the thumbnail loses far fewer tokens but receives the same bias as the tiles and is over-compensated, so accuracy drops to 66.4 on VG (9.9% retention) and 71.5 on VQA (11.1% retention), against 76.2 and 74.4 without the bias. We therefore compute the bias separately for the thumbnail and the tiles, replacing N/kN/k with the ratio of total to retained tokens within each pool. This restores the share of the tiles to 0.81 but does not improve accuracy (75.8 on VG and 73.4 on VQA at the same retention ratios), so the restored tile attention adds little information. We therefore disable the bias on this backbone, and the gain of CAR on InternVL3.5 comes from the selection. We have added these results to Appendix C.7 of the revised manuscript.

AuthorsOpenReview ↗

Rebuttal for Reviewer qHPG (2/4)

W3. Exchangeability and the optimality of log⁡(N/k)\log(N/k)

We agree that, under the selector, log⁡(N/k)\log(N/k) does not restore the visual attention mass exactly. The paper treats log⁡(N/k)\log(N/k) as an approximation, not as the per-query optimum. If pruning left the inputs of every layer unchanged, the exact bias would be −log⁡ρ-\log\rho, where ρ\rho is the share of the visual attention that the retained tokens receive in the unpruned model. Because weighted coverage selection (WCS) retains salient tokens, the actual ρ\rho is larger than its value under exchangeability, k/Nk/N. The questions are how large the resulting deviation is and whether it affects performance.

We measured ρ\rho on Qwen2.5-VL-7B grounding (RefCOCO val, 100 images, 9.9% retention): on average it is about twice k/Nk/N, so with unchanged inputs log⁡(N/k)\log(N/k) would over-restore. Pruning and the bias, however, change the inputs of every later layer, so this deviation does not carry over unchanged to the pruned model. Appendix C.10 (Figure 5) of the paper measures, at each layer of the pruned model, the visual attention share relative to that of the unpruned model. In layers 13–19, where the text attends most to the image, log⁡(N/k)\log(N/k) raises this ratio from 0.60 (without bias) to 0.87, below one; it exceeds one only in the shallow layers. Therefore, log⁡(N/k)\log(N/k) does not over-restore the share in the layers that attend most to the image.

We also calibrate a separate bias for each layer (on 50 RefCOCO val images, solving the layers from shallow to deep so that the visual attention share of each pruned layer equals that of the unpruned model). The calibrated bias is smaller than log⁡(N/k)=2.31\log(N/k)=2.31 in the shallow layers (1.7–1.9 in layers 0–10) and larger in the deep layers (2.5–2.7 in layers 16–27), consistent with Figure 5, where the constant bias over-restores the shallow layers and under-restores the deep ones. It restores the share of every layer to within 3% of the unpruned model, yet on the eight RefCOCO splits it reaches 65.9, compared with 66.8 for log⁡(N/k)\log(N/k). Matching the share layer by layer therefore brings no gain, and the constant log⁡(N/k)\log(N/k) suffices in practice.

W5. Missing citations

Thank you for these suggestions. We have added all three works to the revised manuscript (highlighted in blue). In the Related Work section (Section 4), SPIDER is cited among the encoder-side methods, and MiCo and SinkPruner among the two-stage methods. Appendix A.1 describes each of them. SPIDER (TPAMI 2026) combines middle- and deep-layer visual features for token selection with adaptive decoder sub-layer skipping. MiCo (arXiv 2026) uses weighted coverage for encoder-side candidate selection and decoder-side task-aware refinement. SinkPruner (Findings of EMNLP 2026) aggregates high-norm outliers and selects informative visual tokens before text-guided decoder pruning.

AuthorsOpenReview ↗

Rebuttal for Reviewer qHPG (3/4)

W4. Evaluation on Qwen3.5-9B and Qwen3.8-27B

We additionally evaluate CAR on Qwen3.5-9B and Qwen3.8-27B (Tables R1–R4 below) and have added these results to Appendix C.14 of the revised manuscript.

Extending CAR to hybrid attention. Most decoder layers of both models are Gated DeltaNet (GDN) layers (24 of 32 in Qwen3.5-9B, 48 of 64 in Qwen3.8-27B). AR uses a logit bias to let retained tokens carry the attention mass of the removed tokens, which requires a softmax. We therefore apply AR only in the full-attention layers, whose softmax attention is the same as in the other backbones. A GDN layer has none: it writes the tokens one by one into a fixed-size recurrent state. After pruning, fewer visual tokens are written over a shorter timeline, so both the share of the image in the state and the proportions among its contents deviate from those of the unpruned model, which a logit bias cannot correct. We therefore extend the idea of AR to GDN layers: each representative explicitly writes on behalf of the removed tokens it covers.

GDN representative replay. Each removed visual token is represented by the retained token most similar to it, the same assignment that defines the cells of AR. Within each GDN layer, the projections are computed on the pruned sequence, expanded to the original length by filling each removed position with those of its representative, and passed through the convolution and the recurrence; only the outputs at the retained positions are then kept. Every other component, namely the full-attention layers, the MLPs and the KV cache, processes only the pruned sequence. Replay is applied only during prefill; decoding continues from the resulting GDN states without any change. The only overhead is the GDN scan over the removed positions during prefill, which on Qwen3.5-9B adds less than 2% to the FLOPs of the vision encoder and the prefill of plain deletion.

Results. On grounding (Tables R1 and R2, 9.9% retention), all baselines nearly collapse on both models: the strongest reaches only 7.4 on Qwen3.5-9B and 12.8 on Qwen3.8-27B, whereas CAR reaches 43.1 and 41.0, clearly outperforming all baselines. The component ablations follow the same trend on both models. The probe-selected head raises WCS from 6.6 and 15.1 to 20.3 and 23.3, GDN replay brings the largest gain, to 42.7 and 40.4, and AR adds a small further gain, reaching 43.1 and 41.0. Replay also transfers to other selection methods: on Qwen3.5-9B, combined with replay and AR, the five baselines rise from at most 7.4 to between 17.7 and 43.2, with EADP on par with CAR. On VQA (Tables R3 and R4, 11.1% retention), CAR outperforms the strongest baseline, EADP, by 5.0 and 4.7 points on the two models. Most of the gain comes from head selection, while replay and AR bring little further change.

Table R1. Grounding on Qwen3.5-9B (9.9% retention).

MethodRefCOCORefCOCO+RefCOCOgAvg.Rel.
testAtestBvaltestAtestBvaltestval
Unpruned93.1287.8190.8188.7479.0884.2888.0688.7187.58100.0%
Pruning baselines
DivPrune9.554.816.697.463.665.325.676.116.167.0%
VisionZip8.704.206.527.603.505.105.445.765.856.7%
AnchorPrune9.975.507.636.993.685.305.895.746.347.2%
EADP10.615.068.359.874.447.087.206.767.428.5%
Nüwa11.196.037.789.334.876.387.026.747.428.5%
CAR components
WCS (head avg.)9.695.467.048.474.196.045.855.686.557.5%
WCS (selected head)27.8417.4922.5523.4913.8519.1518.8119.3820.3223.2%
+ GDN replay49.6242.6546.8841.5134.4938.6343.4544.4042.7048.8%
+ GDN replay + AR (CAR)49.7443.5946.8341.9835.6339.1943.5244.5943.1449.3%
Baseline selection with GDN replay and AR
EADP + replay + AR49.0743.7546.8442.2135.2240.0943.8944.5943.2149.3%
DivPrune + replay + AR35.4429.3833.6628.5723.2427.5629.5430.0929.6833.9%
VisionZip + replay + AR33.1431.0532.6927.7525.2026.0630.0130.7829.5933.8%
AnchorPrune + replay + AR21.6019.4520.9716.5713.8915.4017.0016.7517.7020.2%
Nüwa + replay + AR37.2132.8835.0831.2126.0428.0833.1933.0332.0936.6%
AuthorsOpenReview ↗

Rebuttal for Reviewer qHPG (4/4)

W4 (continued).

Table R2. Grounding on Qwen3.8-27B (9.9% retention).

MethodRefCOCORefCOCO+RefCOCOgAvg.Rel.
testAtestBvaltestAtestBvaltestval
Unpruned90.9386.7189.8686.3177.7383.5988.7988.6886.58100.0%
Pruning baselines
DivPrune17.7510.2114.2215.658.8611.8712.1312.0512.8414.8%
VisionZip8.594.656.367.423.664.975.424.845.746.6%
AnchorPrune16.3710.8713.2513.837.9011.2811.8812.2112.2014.1%
EADP17.9610.1111.0515.377.9811.8011.8811.2512.1814.1%
Nüwa12.236.978.749.615.626.937.927.298.169.4%
CAR components
WCS (head avg.)20.5112.1516.8617.7810.4715.1013.8114.2215.1117.5%
WCS (selected head)30.5521.2225.2526.9317.4122.7420.8921.2223.2826.9%
+ GDN replay47.9439.6944.7641.4332.3837.2740.3839.3440.4046.7%
+ GDN replay + AR (CAR)47.9940.5945.8641.8132.7138.2640.6440.2641.0147.4%

Table R3. VQA on Qwen3.5-9B (11.1% retention).

MethodTextVQAChartQAAI2DOCRBenchHallBenchMMEMMB-ENMMB-CNAcc.Rel.
Unpruned84.687.488.088957.22365.485.885.887.0100.0%
Pruning baselines
DivPrune68.534.474.141143.31981.275.974.163.873.1%
VisionZip52.740.271.220241.21715.772.671.356.965.5%
AnchorPrune57.227.970.527844.31785.868.565.656.465.0%
EADP64.139.774.240242.92059.878.175.764.774.0%
Nüwa60.352.474.923345.41994.777.676.463.773.2%
CAR components
WCS (head avg.)66.841.474.338242.92007.277.676.364.774.1%
WCS (selected head)69.048.777.044846.12120.079.078.668.678.6%
+ GDN replay70.146.976.652145.32111.278.976.369.078.9%
+ GDN replay + AR (CAR)70.347.076.951346.92144.979.777.969.779.8%

Table R4. VQA on Qwen3.8-27B (11.1% retention).

MethodTextVQAChartQAAI2DOCRBenchHallBenchMMEMMB-ENMMB-CNAcc.Rel.
Unpruned83.088.288.285657.72179.983.781.084.6100.0%
Pruning baselines
DivPrune62.832.872.035836.01720.170.267.057.868.0%
VisionZip47.136.469.416536.71546.268.365.252.161.6%
AnchorPrune55.234.472.826738.61619.268.764.755.365.3%
EADP64.134.971.935939.21792.773.570.560.070.7%
Nüwa54.241.670.122339.01550.766.163.454.364.3%
CAR components
WCS (head avg.)64.039.073.332440.41737.774.170.560.170.9%
WCS (selected head)67.559.676.140542.51887.073.768.565.376.9%
+ GDN replay67.854.674.145739.11927.174.069.765.276.4%
+ GDN replay + AR (CAR)67.952.473.544441.61880.574.569.464.776.2%

Thank you again for your valuable comments and suggestions. Please let us know if you have any further concerns, and we would be glad to address them.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Contributions and Discussion

Dear Area Chair and Reviewers,

We thank all reviewers for their time and constructive feedback. We summarize our contributions, the main points of the discussion, and the revisions.

Contributions.

  • Analysis. We decompose the decoder output shift caused by visual token pruning into an image summary error and an attention allocation error, and identify the loss of the visual attention share as a source of degradation that token selection alone cannot fix. Reviewers found this "a compelling contribution" (qHPG) and "valuable from a research perspective" (6HEb).
  • Method. CAR targets each error term with one component: weighted coverage selection reduces the summary error, and attention reallocation restores the visual attention share. Reviewers described CAR as "elegant", with weighted coverage selection as "a principled, training-free token selection strategy" (qHPG), and noted that the analysis "provides a clear justification for each method" and makes "the analysis and the design consistent" (Yw16).
  • Experiments. CAR is training-free and effective across six backbones (Qwen2.5-VL-7B, Qwen3-VL-8B, InternVL3.5-8B, LLaVA-1.5-7B, and the hybrid-attention Qwen3.5-9B and Qwen3.8-27B added during the rebuttal) and three task families (VQA, visual grounding and video understanding). The gains are largest under aggressive pruning, e.g., 66.8 vs. 22.8 for the strongest baseline on grounding with Qwen2.5-VL-7B at 9.9% retention. Reviewers noted that CAR "achieves state-of-the-art results across four diverse backbones and multiple tasks without any training, with particularly dramatic gains on visual grounding" (qHPG), "outperforms existing methods consistently" (Yw16), and is supported by a "comprehensive experimental evaluation" (6HEb).
  • Consistency between theory and practice. We verify this consistency at four levels: the coverage objective (Appendix C.17), the bias approximation (Appendix C.10), the two error terms (Fig. 3), and the total output error and output KL (Appendix C.16). Together, these show that CAR reduces in practice the errors identified by our analysis.

Discussion. We summarize the main points raised by the reviewers and our responses.

  • (a) Practical cost (qHPG W1, W4; 6HEb Q4; Yw16 Q5). The head probe is a one-time offline step with forward passes only; with as few as 100 samples per benchmark, it takes about one GPU-hour with nearly the same accuracy. Head choices without calibration data, including per-input selection at inference time, are less accurate than the probed head, which justifies this step. With all selection overhead included, CAR gives 1.30x to 2.18x end-to-end speedups. On the hybrid Qwen3.5-9B, CAR with GDN replay keeps nearly the same prefill speedup as plain deletion (1.48x vs. 1.52x).
  • (b) Generality (qHPG W2, W4; 6HEb W3). On the new hybrid-attention Qwen3.5-9B and Qwen3.8-27B, CAR outperforms the previous best methods by a wide margin (43.1 and 41.0 vs. 7.4 and 12.8 on grounding at 9.9% retention). Its components also remain effective beyond Qwen2.5-VL, as the component ablations on Qwen3-VL-8B, Qwen3.5-9B and Qwen3.8-27B show, and extending attention reallocation to the GDN layers brings the largest gain on the hybrid models. Attention reallocation corrects the allocation error, so its gain depends on how strongly a backbone relies on the visual attention share.
  • (c) Theory and practice (qHPG W3; 6HEb W1, W2; Yw16 W1). (1) Encoder similarity largely agrees with decoder similarity, and a higher practical coverage objective gives a lower theoretical image summary error. (2) The retained tokens carry about 2.2 times k/N of the attention mass, but since pruning changes the inputs of later layers, log(N/k) does not over-restore the visual attention share in the layers of the pruned model where the text attends most to the image. (3) CAR has the lowest total output error and output KL, so reducing the two error terms also reduces the shift in the final output.
  • (d) Task configurations (Yw16 W2, Q4). A single configuration shared by grounding and VQA still outperforms the best baselines on both tasks, and balances answering and localization on a benchmark whose queries ask both what and where.
  • (e) Presentation (qHPG W5; Yw16 Q2). We moved the comparison with RESTORE from the appendix to the main paper and added the three suggested references.

We thank Reviewer Yw16 for confirming that their concerns have been resolved, Reviewer 6HEb for noting that theirs have mostly been addressed, and Reviewer qHPG for acknowledging part of our responses and for the detailed follow-up comments, which helped us strengthen the paper. We hope that our responses also address the remaining concerns of Reviewer qHPG.

All revisions are highlighted in blue in Sections 2.2, 3.2 and 4 and Appendices A.1, C and D.2.

Thank you for considering our responses.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.