acceptodds
Withdrawn from ICLR 2027

Calibrate Before Reason: Robust Visual Token Reduction against Semantic Drift in VLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large Vision-Language Models (VLMs) suffer from prohibitive inference overhead due to long sequences of visual tokens. However, existing visual token reduction methods mainly improve efficiency by pruning or compressing redundant tokens without examining whether the resulting representation remains semantically consistent with the original representation. Mapping the original -token visual sequence to tokens may discard, dilute, or misassign critical visual cues, triggering severe **semantic drift** that deviates the VLM’s understanding. In this paper, we first introduce the principle of **"Calibrate Before Reason"** to visual token reduction and propose **CaRe**, a training-free robust framework that calibrates compact visual representations before reasoning to preserve semantic fidelity in VLMs. CaRe consists of two mutually complementary modules: 1) Perturbation-Robust Calibration Anchoring, which identifies calibration anchors with stable model-side influence under multi-directional perturbations; 2) Confidence-Gated Token Calibration, which extracts reliable calibration signals from unselected tokens and injects them into anchors. Extensive evaluations across diverse VLM architectures and benchmarks verify that CaRe outperforms state-of-the-art token reduction baselines. While pruning **94.4%** of visual tokens, our method retains **96.4%** of the original full-token performance, delivering up to faster end-to-end inference speed relative to unpruned vanilla models.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Models, Visual Token Reduction, Semantic Drift

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Y2aiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that visual token reduction in VLMs causes "semantic drift": selecting tokens discards visual cues, while merging tokens dilutes them. It proposes CaRe, a training-free method that refines the retained tokens before reasoning, in two stages. Perturbation-Robust Calibration Anchoring identifies tokens with strong, stable responses that are also mutually diverse are chosen as anchors. Confidence-Gated Token Calibration keeps only those that can be confidently matched to an anchor by visual similarity and spatial proximity among the unselected tokens, and blends their information into the matched anchors. Experiments on LLaVA-1.5-7B, LLaVA-NeXT-7B and Qwen2.5-VL-7B show that CaRe outperforms existing token reduction methods. It preserves 96.5% of full-token performance on LLaVA-1.5-7B with 64 tokens (88.9% pruned), 96.4% on LLaVA-NeXT-7B with 160 tokens (94.4% pruned), and 89.3% on Qwen2.5-VL-7B with 10% of tokens retained, and achieves up to 2.30× end-to-end speedup.

Critical strengths

Consistent gains across three backbones and several benchmarks. PRCA and CGTC improves the normalized average by about 1–2 points at every budget on LLaVA-1.5 and LLaVA-NeXT. On Qwen2.5-VL, the TextVQA gain over ZOO-Prune at 10% (65.33 vs 61.99, Table 8) suggests the residual calibration recovers some fine-grained evidence.

The budget-preserving "recycle discarded tokens into kept anchors" design is practical. The mechanism is plug-in on top of any selector, it could be adopted elsewhere.

Critical weaknesses

PRCA overhead on LLaVA-1.5. For LLaVA-1.5, the released code (llava_arch.py) passes all 2 × 64 × 576 ≈ 74K perturbed tokens through the full two-layer projector (1024→4096→GELU→4096; the line calling the low-rank projector is commented out), which costs about 3.09 TFLOPs. However, the FLOPs script (scripts/estimate_care_flops.py) implements this term as 4·m·N·d_v·H, i.e. only the first 1024→4096 linear layer (about 0.62 TFLOPs), omitting the 4096→4096 layer (about 2.47 TFLOPs). Table 1's CaRe numbers (3.47 / 3.02 / 2.18) can be reproduced by this script with roughly 26–56 text tokens. Adding the missing layer at the same settings gives about 5.9 / 5.5 / 4.7 TFLOPs at 192 / 128 / 64 tokens. At 64 tokens this is more than double the reported 2.18 and clearly above FastV (1.98) and VisionZip (2.25), so the "lower TFLOPs" comparison in Table 1 does not hold.

The default hyperparameters in the released code differ from those reported in the paper. Section 4 and Appendix C.2 report h = 5×10⁻⁴, λ = 0.5, η = 0.45, θc = 0.9 and k_f = min(K, max(2, ⌈0.1K⌉)). The LLaVA run script (run_care.sh, called by run_care_batch.sh for both LLaVA-1.5 and LLaVA-NeXT) instead defaults to h = 0.01, λ = 0.2, η = 0.15, θc = 0.8 and a fixed k_f = 4; run_care_qwen25.sh uses θc = 0.9.

Questions for authors

FLOPs. Were the TFLOPs in Table 1 and Table 5 computed with this script? Recompute them and state the number of text tokens assumed, and re-check the comparison with FastV and VisionZip in Table 1.

Hyperparameters. Which configuration produced Tables 1–3 ?

Reviewer_FuVpOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that visual token reduction causes "semantic drift": selection drops useful cues, and merging dilutes them. It proposes CaRe, a training-free method with two stages before the LLM. PRCA (Perturbation-Robust Calibration Anchoring) perturbs each visual token along m=64m=64 random directions before the projector and measures the finite-difference response of the projector. Responses are normalized per direction with median/MAD, and the score μ−λσ\mu-\lambda\sigma penalizes directional variance. Combined with a cosine-diversity term, this score greedily selects KK anchors. CGTC (Confidence-Gated Token Calibration) admits unselected tokens whose semantic–spatial affinity to the anchors is high and concentrated. Admitted tokens are softly assigned to anchors and added as a weighted residual (α=0.15\alpha=0.15), and the anchor norm is preserved, so the output still has KK tokens.

On LLaVA-1.5-7B over nine benchmarks, CaRe retains 99.9/98.3/96.5% at 192/128/64 tokens versus 98.0/97.2/94.8% for ZOO-Prune. It retains 96.4% at 160 tokens on LLaVA-NeXT-7B and reports up to 2.30× end-to-end speedup. Ablations add PRCA and CGTC to a ZOO-Prune baseline, and the appendix covers hyperparameter sensitivity and failure cases.

Critical strengths

S1. Consistent improvements across three backbones and many budgets, with a clean ablation path from a strong baseline. CaRe has the best average at every budget on LLaVA-1.5-7B (nine benchmarks), LLaVA-NeXT-7B, and Qwen2.5-VL-7B. The margin over ZOO-Prune is 1.9/1.1/1.7 points on LLaVA-1.5 and 1.5/1.4/1.9 on LLaVA-NeXT. The ablation starts from ZOO-Prune and adds each component in turn: PRCA alone, ungated merging, gated merging, and both. This makes the incremental contribution of each part visible.

S2. Training-free, attention-free, and thoroughly documented. CaRe operates entirely before the LLM, so it is compatible with FlashAttention and needs no access to attention maps or retraining. The paper reports all hyperparameters, the FLOPs derivation including the perturbation cost, the evaluation framework and seed, failure cases at 94.4% pruning, and an anonymized code archive. This makes the results straightforward to reproduce.

Critical weaknesses

W1. Limited novelty relative to ZOO-Prune and token-merging methods, and a framing that overstates the mechanism. The paper itself builds on sensitivity-based selection and information recycling, and its ablation baseline is ZOO-Prune. PRCA changes ZOO-Prune's score by MAD normalization and a directional-variance penalty (μ−λσ\mu-\lambda\sigma); CGTC is gated residual merging of discarded tokens into kept ones, close to VisionZip's contextual tokens, FiCoCo, and PruMerge. The paper describes this as reformulating token reduction as "calibration" that preserves "task-relevant semantics". But neither module uses the query, and the only query-conditioned evidence is a metric the authors define themselves (CSR), used for the motivation figure.

W2. Small module gains, test-set tuning, and mixed baseline provenance. PRCA adds 0.4–1.0 points over ZOO-Prune, and the confidence gate adds 0.3–0.4 points over ungated merging, all from a single seed (42). The gate's θs×θc\theta_s\times\theta_c sweep changes POPE F1 only within 85.41–85.66 while the signal acceptance rate varies from 33% to 100%, suggesting the gate has almost no effect. More than ten hyperparameters were chosen on the same benchmarks used for evaluation. In addition, rows for FastV, SparseVLM, and VisionZip reproduce the original papers' numbers exactly (e.g., VisionZip at 192 tokens), while CaRe was run with LMMs-Eval on an RTX 5090 D. Margins of 1–2 points are therefore hard to attribute to the method.

Questions for authors
  1. Variance and held-out tuning. What are mean and standard deviation over several seeds (perturbation directions are random) for CaRe and ZOO-Prune? What happens if the hyperparameters are tuned on a split disjoint from the reported benchmarks?
  2. Baseline provenance. Which baseline numbers were re-run under your LMMs-Eval setup and which were copied? Can you re-run FastV, SparseVLM, and VisionZip under the identical pipeline?
  3. Is the gate needed? Given the flat θs×θc\theta_s\times\theta_c heatmap, how does CGTC compare with VisionZip-style merging of all discarded tokens, or with merging into ZOO-Prune anchors without gating, across all nine benchmarks?
  4. Overhead. What is the wall-clock time of PRCA (128 projector passes) and CGTC per image on LLaVA-NeXT, compared with ZOO-Prune's scoring cost?
Reviewer_4hjHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CaRe, a training-free method that reduces visual tokens while aiming to preserve information needed for reasoning. It selects representative tokens using stable projector responses to multi-directional perturbations, then updates them with information from unselected tokens filtered by a confidence gate. Experiments on three VLM backbones show better average performance retention than pruning and compression baselines at matched token budgets, with reported end-to-end speedups of up to 2.30×.

Critical strengths
  • Strong performance under aggressive token reduction. On LLaVA-NeXT-7B, CaRe retains 96.4% of full-token performance while removing 94.4% of visual tokens, outperforming the reported pruning and compression baselines at the same budget.

  • Effective combination of selection and information recovery. CaRe selects stable, diverse anchors and supplements them with filtered residual information. Module ablations show that both components contribute, supporting the benefit of combining them.

  • Practical applicability across architectures. The method requires no parameter updates and is evaluated on three VLM backbones, including high- and dynamic-resolution models. Reported end-to-end speedups of up to 2.30× demonstrate practical gains beyond token-count reduction.

Critical weaknesses
  • The link between projector sensitivity and semantic importance remains insufficiently validated. PRCA measures local projector responses, which need not reflect relevance to the question or importance to the final answer. The question-independent selection may miss small but critical evidence. CSR provides a representation-similarity diagnostic, but does not establish that the required evidence or answer is preserved.

  • The added value of confidence gating is modest. Table 9 shows that PRCA with simple, ungated calibration already achieves 99.6%, 97.9%, and 96.1% performance retention; adding the gate improves these by only 0.3–0.4 percentage points. Without uncertainty estimates or repeated runs, it is difficult to judge whether these gains justify the additional thresholds and assignment machinery.

  • The practical overhead needs clearer measurement. PRCA uses 64 symmetric perturbation directions, requiring substantial additional projector computation. Although the paper includes FLOP accounting and latency comparisons, a measured breakdown of anchor selection, calibration, and LLM processing is missing. Experiments on a single GPU with batch size one and POPE-based timing also leave uncertainty about the speedup under other deployment workloads.

  • The comparison omits recent training-free methods, particularly AnchorPrune and CRISP, which address closely related questions of query relevance and contextual coverage. AnchorPrune reports competitive results on LLaVA-NeXT-7B at the same retained-token counts. A comparison under a common evaluation protocol would help establish CaRe’s advantage over these alternatives. Methods with layer-wise pruning schedules, such as TOPS and PIO-FVLM, should additionally be compared at matched computation or end-to-end latency.

Questions for authors
  1. What evidence links PRCA’s projector sensitivity to answer-relevant information? Since selection is question-independent, does it retain the evidence needed for different questions about the same image, particularly small objects or text?

  2. How reliable is the gain from confidence gating over simple calibration? Table 9 reports improvements of only 0.3–0.4 percentage points. Are these consistent across random seeds and backbones, and does the gate specifically prevent harmful information mixing?

  3. What is PRCA’s measured per-image overhead? Please separate anchor selection, calibration, and LLM processing, and report how increasing mm changes both accuracy and end-to-end latency. This would clarify whether 64 perturbation directions offer a favorable practical tradeoff.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.