- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper decomposes the shift in the attention output caused by visual token pruning into the image summary error and the attention allocation error. To reduce errors, it proposes a saliency weighted coverage selection and a logit bias that restore the visual attention share, respectively. Experimental results on four backbones show that CAR outperforms existing methods consistently, especially on visual grounding under aggressive pruning.
Critical strengths
-
S1. This paper decomposes the shift in the attention output into the image summary error and the attention allocation error, identifying the loss of the visual attention share as a source of degradation. While the token selection and the logit bias are individually close to prior methods, this mathematical modeling provides a clear justification for each method.
-
S2. Each component of CAR targets its own error term, which Fig. 3 and Table 5 confirm. The bias reduces only the allocation error and the selection only the summary error, making the analysis and the design consistent.
Critical weaknesses
W1. Reduction of the actual output shift.
- Fig. 3 shows the two error terms of Eq. 5 separately. Since both terms depend on , this does not guarantee a small total error , which is not reported.
- The error analysis considers only the attention output of each layer, given the input of the unpruned model, excluding the MLP and the errors accumulated over layers. The KL divergence between the output distributions over the vocabulary of the pruned and unpruned models would show whether CAR reduces the shift in the final output.
W2. Configurations separated by task.
- CAR sets the spatial kernel and the form of the bias differently for grounding and question answering, assuming that the task family is known at inference time. The paper, however, provides no guideline for a query that asks both where an object is and what it is, or for the case where the task family is unknown.
Questions for authors
-
Q1. Nüwa (Huang et al., 2026) and RESTORE (Cho et al., 2026) take different position policies for the retained tokens. Nüwa favors the relative position mapping extension (RPME), which rescales the positions of the retained tokens to span the whole image, over keeping the original ones, while RESTORE keeps the original positions and compensates for the loss of visual attention with a calibration term. CAR follows RESTORE, so how do the authors interpret this choice for question answering and grounding, respectively?
-
Q2. CAR keeps the original positions and adds a logarithmic bias to the logits of the retained tokens, closely following the design of RESTORE. Why is this relation discussed and evaluated only in the appendix rather than in the main paper? Since it directly determines the novelty of the attention reallocation, it should be clearly stated in the main manuscript.
-
Q3. What are the total error in Fig. 3 and the KL divergence between the output distributions over the vocabulary of the pruned and unpruned models?
-
Q4. How does CAR perform with a single configuration shared across grounding and question answering, and how does it handle a query that asks both where an object is and what it is?
-
Q5. The probe requires running the unpruned model on a probe set for each backbone and task family, which most training-free baselines do not need. Does CAR perform comparably to the probed head when the head is selected per input at inference time without calibration data?
5 comments
Rebuttal for Reviewer Yw16 (2/4)
Q1. Position policy for the retained tokens
We first clarify Nüwa's position policy. In the Nüwa paper, RPME appears only in the analysis, where it is applied to VisionZip and FastV. The method itself does not remap positions: it preserves the global coordinate system through region partitioning, and its second stage follows FastV, which keeps the original positions. Its released code also passes the original position IDs to the language model. Nüwa, RESTORE, and CAR therefore all keep the original positions.
We also compare the two policies on CAR (Table R3): PESP keeps the original positions (CAR's default), and RPME stretches the retained positions along each spatial axis of Qwen2.5-VL's multimodal RoPE; selection and AR are unchanged. The two policies are within 1.0 point on all benchmarks. On grounding, PESP is slightly better (72.2 vs. 71.2): the model predicts boxes in absolute coordinates read from the RoPE row and column indices, which RPME shifts. On question answering, they are on par, since the answer depends on the content rather than its exact location. CAR therefore keeps the original positions, which need no remapping.
Table R3. Position policies on CAR (Qwen2.5-VL-7B; VG 9.9% retention, VQA 128 tokens).
| Position policy | RefCOCO val | TextVQA | ChartQA |
|---|---|---|---|
| PESP (original, CAR) | 72.2 | 78.8 | 76.1 |
| RPME (stretched) | 71.2 | 78.9 | 76.0 |
Q2. Relation to RESTORE in the main paper
Thank you for this suggestion. We have made this relation explicit in the main paper in two places (highlighted in blue).
- Related Work. The new Related Work section (Section 4) includes a paragraph on attention calibration. It traces the log-count bias to ToMe and describes RESTORE, which merges the non-retained tokens into representatives and calibrates attention with group size and relative distance, as well as ERA. CAR differs in that it leaves the retained features unchanged, and its bias is set by the retention ratio or by the token counts of the coverage assignment.
- Comparison. Section 3.2 now includes the comparison with RESTORE (Table 8), applying RESTORE's merging, its merging and bias, and our bias on top of DivPrune and WCS at the same final token budget. RESTORE's merging improves DivPrune on VQA but lowers accuracy with WCS and on grounding with both selectors, whereas our bias improves both selectors on both tasks, by 22.4 and 40.0 points on grounding. The detailed analysis remains in Appendix C.10.
Rebuttal for Reviewer Yw16 (3/4)
W2/Q4. Configurations separated by task
We evaluate one configuration shared by grounding and question answering, and test it on the MM-GCoT benchmark, whose queries ask both what an object is and where it is. This unified configuration fixes the spatial kernel and the capped per-cell bias: the kernel gains 4.3 on VG and costs 0.2 on VQA (Table 5 of the paper), and the capped per-cell bias is the best form on VQA and within 0.5 points of the best on VG (Table 7 of the paper). Its saliency averages the saliency of the two heads that the label-free head probe selects for VQA and VG. The probe runs once per backbone on the probe set, so no task label is needed at inference.
(1) Grounding and question answering. Against the task-specific configurations, the unified configuration loses 3.0 points on VG and 1.3 on VQA (Table R4). It still clearly outperforms the best VG baseline (63.8 vs. 22.8) and remains above the best VQA baseline (78.0 vs. 77.1).
Table R4. Single vs. task-specific configurations (Qwen2.5-VL-7B; VQA 128 tokens, VG 9.9% retention).
| Method (VQA) | TextVQA | ChartQA | AI2D | OCRBench | HallBench | MME | MMB-EN | MMB-CN | Acc. |
|---|---|---|---|---|---|---|---|---|---|
| Unpruned | 84.3 | 87.1 | 81.6 | 867 | 56.2 | 2342.3 | 83.1 | 82.9 | 84.9 |
| HAWK (best baseline) | 79.9 | 71.3 | 77.5 | 705 | 48.8 | 2225.6 | 79.0 | 78.6 | 77.1 |
| CAR, task-specific | 78.8 | 76.1 | 79.4 | 731 | 49.0 | 2314.3 | 81.1 | 81.3 | 79.3 |
| CAR, unified | 76.2 | 75.6 | 79.6 | 695 | 48.5 | 2262.0 | 80.3 | 80.9 | 78.0 |
| Method (VG) | RefCOCO | RefCOCO+ | RefCOCOg | Avg. | |||||
|---|---|---|---|---|---|---|---|---|---|
| testA | testB | val | testA | testB | val | test | val | ||
| Unpruned | 92.2 | 84.8 | 89.4 | 87.6 | 74.1 | 82.8 | 86.7 | 86.7 | 85.5 |
| Nüwa (best baseline) | 27.5 | 23.7 | 26.2 | 21.8 | 17.4 | 20.5 | 23.0 | 22.0 | 22.8 |
| CAR, task-specific | 75.4 | 66.7 | 72.2 | 67.2 | 55.9 | 62.3 | 67.8 | 66.5 | 66.8 |
| CAR, unified | 71.9 | 62.7 | 69.4 | 64.3 | 52.2 | 59.7 | 65.6 | 64.3 | 63.8 |
(2) A query that asks both what and where. We use the MM-GCoT benchmark (CVPR 2026), where each question asks for the attribute or category of an object and for its box, in two settings (Table R5): the official answer-then-grounding protocol, which asks for the answer and then the box, and a single query asking for both. The unified configuration uses the saliency averaged over the heads selected by the VQA and VG probes, without any MM-GCoT data. The two task-specific configurations each favor one part of the query: the VQA configuration answers well but localizes poorly (G-Acc 25.6 and 30.9), whereas the VG configuration localizes well but answers less accurately with a single query (A-Acc 55.4 vs. 61.1). The unified configuration balances the two: its G-Acc (38.1 and 43.8) approaches that of the VG configuration (41.0 and 47.8), its single-query A-Acc (58.7) lies between the two, and its answer-box consistency is within 0.8 points of the VG configuration. All three remain well above the best baseline in localization (G-Acc 19.9 and 21.2).
We further run the head probe on 500 unlabeled questions from the MM-GCoT training set and compute the saliency from the head it selects. This variant gives the best pruned results on all three metrics under answer-then-grounding (64.5 A-Acc, 42.3 G-Acc, 39.7 Cons.) and the best G-Acc and consistency with a single query. It improves G-Acc over the unified configuration by 4.2 and 4.6 points and also exceeds the VG configuration in both settings.
Table R5. MM-GCoT (Qwen2.5-VL-7B, 9.9% retention). A-Acc: answer accuracy; G-Acc: box IoU > 0.5; Cons.: answer-box consistency. VQA/VG config.: the task-specific configurations. Bold: best pruned.
| Method | Answer, then grounding | Single query | ||||
|---|---|---|---|---|---|---|
| A-Acc | G-Acc | Cons. | A-Acc | G-Acc | Cons. | |
| Unpruned | 70.9 | 63.4 | 56.1 | 66.1 | 70.3 | 56.9 |
| VisionZip | 58.8 | 17.8 | 18.9 | 54.5 | 19.1 | 18.7 |
| DivPrune | 63.6 | 19.9 | 20.0 | 58.8 | 21.2 | 20.9 |
| AnchorPrune | 61.0 | 17.6 | 19.0 | 56.1 | 20.3 | 18.3 |
| EADP | 58.5 | 16.4 | 17.6 | 52.0 | 18.5 | 16.9 |
| Nüwa | 61.2 | 18.0 | 17.9 | 55.3 | 17.6 | 16.6 |
| CAR, VQA config. | 63.8 | 25.6 | 24.0 | 61.1 | 30.9 | 27.3 |
| CAR, VG config. | 64.3 | 41.0 | 38.1 | 55.4 | 47.8 | 38.1 |
| CAR, unified | 62.1 | 38.1 | 37.3 | 58.7 | 43.8 | 37.4 |
| CAR, unified, MM-GCoT probe | 64.5 | 42.3 | 39.7 | 59.0 | 48.4 | 39.6 |
Guideline. (i) When the task family is known, use its task-specific configuration. (ii) When it is unknown or the query is mixed, use the unified configuration. (iii) For a new task family with unlabeled inputs, a one-time probe on these inputs can further improve accuracy. We have added this guideline, the unified configuration, and Tables R4 and R5 to Appendix C.15 of the revised manuscript.
Rebuttal for Reviewer Yw16 (1/4)
Thank you for your positive recommendation and insightful comments. Our responses to each point follow.
W1/Q3. Total output error and output KL
For every configuration of Fig. 3 (Qwen2.5-VL-7B, grounding, 9.9% retention), we report the total error ∑∥O′−O∥2/∑∥O∥2 (%) over layers 13–19, where ∥O′−O∥2=∥Esum∥2+∥Ealloc∥2+2⟨Esum,Ealloc⟩ is the sum of the image summary error, the attention allocation error and their cross term. We also report the KL divergence between the output distributions of the unpruned and pruned models over the vocabulary (Tables R1 and R2).
Total error. The bias lowers the total error of every method. CAR has the lowest total error both without the bias (16.7, against 18.5–24.2 for the existing methods) and with it (14.0, against 17.4–22.4). In the component ablation, the total error decreases from 16.8 for coverage selection alone to 14.0 for CAR, mainly through the bias (−1.5) and the selected head (−1.1).
Output KL. The bias also lowers the KL of every method. With the bias, CAR has the lowest KL (0.150, against 0.194–0.302 for the existing methods). In the component ablation, each component lowers the KL from 0.405 for coverage selection alone: the bias to 0.218, the saliency weights to 0.175, the selected head to 0.158, and the spatial kernel to 0.150 (CAR). Thus, CAR lowers not only the two terms of Fig. 3 but also their total and the shift in the final output.
Table R1. Total error (% of ∥O∥2, layers 13–19) and output KL for the existing methods of Fig. 3 (Qwen2.5-VL-7B, grounding, 9.9% retention).
Table R2. Component ablation of Fig. 3 (Table 5 of the paper); setting as in Table R1.
We have added these results to Appendix C.16 of the revised manuscript.