- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper argues that visual token reduction in VLMs causes "semantic drift": selecting tokens discards visual cues, while merging tokens dilutes them. It proposes CaRe, a training-free method that refines the retained tokens before reasoning, in two stages. Perturbation-Robust Calibration Anchoring identifies tokens with strong, stable responses that are also mutually diverse are chosen as anchors. Confidence-Gated Token Calibration keeps only those that can be confidently matched to an anchor by visual similarity and spatial proximity among the unselected tokens, and blends their information into the matched anchors. Experiments on LLaVA-1.5-7B, LLaVA-NeXT-7B and Qwen2.5-VL-7B show that CaRe outperforms existing token reduction methods. It preserves 96.5% of full-token performance on LLaVA-1.5-7B with 64 tokens (88.9% pruned), 96.4% on LLaVA-NeXT-7B with 160 tokens (94.4% pruned), and 89.3% on Qwen2.5-VL-7B with 10% of tokens retained, and achieves up to 2.30× end-to-end speedup.
Critical strengths
Consistent gains across three backbones and several benchmarks. PRCA and CGTC improves the normalized average by about 1–2 points at every budget on LLaVA-1.5 and LLaVA-NeXT. On Qwen2.5-VL, the TextVQA gain over ZOO-Prune at 10% (65.33 vs 61.99, Table 8) suggests the residual calibration recovers some fine-grained evidence.
The budget-preserving "recycle discarded tokens into kept anchors" design is practical. The mechanism is plug-in on top of any selector, it could be adopted elsewhere.
Critical weaknesses
PRCA overhead on LLaVA-1.5. For LLaVA-1.5, the released code (llava_arch.py) passes all 2 × 64 × 576 ≈ 74K perturbed tokens through the full two-layer projector (1024→4096→GELU→4096; the line calling the low-rank projector is commented out), which costs about 3.09 TFLOPs. However, the FLOPs script (scripts/estimate_care_flops.py) implements this term as 4·m·N·d_v·H, i.e. only the first 1024→4096 linear layer (about 0.62 TFLOPs), omitting the 4096→4096 layer (about 2.47 TFLOPs). Table 1's CaRe numbers (3.47 / 3.02 / 2.18) can be reproduced by this script with roughly 26–56 text tokens. Adding the missing layer at the same settings gives about 5.9 / 5.5 / 4.7 TFLOPs at 192 / 128 / 64 tokens. At 64 tokens this is more than double the reported 2.18 and clearly above FastV (1.98) and VisionZip (2.25), so the "lower TFLOPs" comparison in Table 1 does not hold.
The default hyperparameters in the released code differ from those reported in the paper. Section 4 and Appendix C.2 report h = 5×10⁻⁴, λ = 0.5, η = 0.45, θc = 0.9 and k_f = min(K, max(2, ⌈0.1K⌉)). The LLaVA run script (run_care.sh, called by run_care_batch.sh for both LLaVA-1.5 and LLaVA-NeXT) instead defaults to h = 0.01, λ = 0.2, η = 0.15, θc = 0.8 and a fixed k_f = 4; run_care_qwen25.sh uses θc = 0.9.
Questions for authors
FLOPs. Were the TFLOPs in Table 1 and Table 5 computed with this script? Recompute them and state the number of text tokens assumed, and re-check the comparison with FastV and VisionZip in Table 1.
Hyperparameters. Which configuration produced Tables 1–3 ?