acceptodds
Under review as a conference paper at ICLR 2027

Beyond Token Content: Visual Token Retirement by Tracking Evolving Influence

Abstract

Token compression inside the vision encoder can reduce computation in both subsequent vision blocks and the downstream language model, but a key challenge is determining when a token can be safely retired while its representation is still evolving. Existing criteria largely rank tokens by redundancy estimated from their current representations or local relations, using signals such as salience and representativeness, without directly assessing whether subsequent computation still depends on each token's continued participation. Through controlled interventions, we show that retaining an intermediate token state as frozen key–value memory does not substitute for allowing the token to continue updating and interacting across later blocks. We therefore recast encoder-internal compression as token retirement and propose Influence-Confirmed Token Retirement (ICTR), a training-free and instruction-agnostic procedure that progressively shortens the sequence to a fixed terminal budget. We estimate external influence without additional forward passes by analytically computing how removing a token from the current key–value set would alter other tokens' outputs and averaging the largest relative changes. As representations and the active set evolve, we recompute external influence and confirm candidates whose ranks remain low across consecutive blocks. Across 10 benchmarks with LLaVA-1.5-7B, the method retains over 96% of dense-model performance on average at the 64-token budget, after retiring 88.9% of visual tokens; at the 128-token budget, it achieves a prefill speedup and reduces the KV-cache footprint by 71.2%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models, Visual Token Compression, Token Retirement, Computational Influence

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_vtggOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper investigates when visual tokens should be removed during visual encoding. Its motivation distinguishes between retaining a token in a frozen intermediate state and allowing it to continue participating in subsequent computation. ICTR estimates the influence of each active token on other tokens by measuring the exact change in the current attention-head output caused by removing that token's key/value contributions, while keeping the current queries, keys, and values fixed. The method averages the relative output changes over a selected subset of the most affected recipient tokens, ranks source tokens by the resulting influence scores, and progressively removes low-influence tokens after confirmation across two consecutive blocks, until a fixed final token budget is reached. Experiments cover LLaVA-1.5, LLaVA-NeXT, and Qwen2.5-VL, with ablations on the influence formula, active-set reevaluation, and confirmation mechanism. With 128 retained tokens on LLaVA-1.5, the paper reports 98.9% normalized performance retention and a 2.048× prefill speedup.

Critical strengths
  1. The scoring criterion has a concrete intervention-based interpretation. Equation 3 measures the immediate effect of removing a source token rather than simply treating attention weights as importance scores. The coefficient-only and frozen-score controls in Table 4 help isolate the contribution of value-output differences and the role of active-set reevaluation. Figure 3 further examines downstream representation perturbations under a shared prefix. Combining an exact local measure with controlled downstream diagnostics makes the contribution more informative than another insufficiently justified importance heuristic.
  2. The empirical evaluation demonstrates both improvements and architectural limitations. Table 2 shows that the proposed method substantially outperforms the compared methods on the high-resolution LLaVA-NeXT architecture.
Critical weaknesses
  1. Several relevant methods are missing from the comparisons. In particular, two related training-free baselines should be considered. AutoPrune (Wang et al., 2025) evaluates five overlapping tasks on LLaVA-1.5-7B under nominal token budgets of 192, 128, and 64 in Table 1. CDPruner (Zhang et al., 2025) reports results with 128 and 64 tokens, as well as 32 tokens, on the same backbone across ten overlapping benchmarks in Table 1.
  2. The evidence supporting the specific two-observation timing criterion is relatively limited and backbone-dependent. In Table 4, removing the confirmation mechanism changes the normalized mean by only 0.40–0.73 points. Although downstream perturbation analyses provide additional evidence, the advantage over current-layer scoring remains limited.
Questions for authors

Please address the issues identified as weaknesses.

Reviewer_yNyCOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Influence-Confirmed Token Retirement (ICTR), a training-free method for progressive visual token pruning within vision encoders. ICTR estimates each token's influence on other active tokens through analytical attention-output perturbations, confirms low-influence tokens across consecutive layers, and dynamically updates retirement decisions. Experiments on LLaVA-1.5, LLaVA-NeXT, and Qwen2.5-VL demonstrate competitive performance under aggressive compression.

Critical strengths
  1. The method is well-argued and analytically grounded theoretical framework. This paper distinguishes between token content and its ongoing computational impact. The proposed source removal metric provides a principle-based criterion for token retirement without requiring additional forward propagation.

  2. The evaluation covers a variety of architectures, compression budgets, and benchmarks. Controlled intervention and component-stripping experiments provide strong evidence supporting impact-based criteria and dynamic re-evaluation.

Critical weaknesses
  1. This method offers limited methodological innovation. Cross-layer importance tracking and progressive pruning techniques have previously been studied in FlowCut and EvoCut. ICTR primarily combines these existing approaches with a relatively simple layer-by-layer confirmation mechanism. Ablation experiments show that the confirmation mechanism itself yields only limited performance improvements; therefore, its unique contribution is, to some extent, merely incremental.

  2. Although ICTR performs well on LLaVA-based models, its performance consistently lags behind SCTS on the Qwen2.5-VL dataset, and the gap between the two widens further under strong compression conditions. Furthermore, the 2.05x speedup ratio mentioned in the paper is only slightly higher than VisionZip’s 1.91x, raising questions about the practical advantages of the newly added impact tracking mechanism in real-world applications.

Questions for authors

Could the author explain whether the significant performance gap between ICTR and SCTS on the Qwen2.5-VL task is primarily due to the constraints imposed by the atomic patch set or to limitations inherent in the influence criteria themselves? What evidence suggests that this architecture achieves an optimal trade-off between accuracy and latency?

Reviewer_wCtMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a strategy for token compression. The key presented idea is to identify candidates to be retired by measuring their impact on successive computations via interventions. This training-free method supports progressive retirement until a fixed token budget is reached. The strategy is evaluated on 10 benchmarks keeping dense accuracy comparable to other recent methods, while substantially improving memory and timing requirements compared to the dense baseline.

Critical strengths

The work proposes an intervention-based strategy, which is interesting, motivated effectively, and shown to be effective with generally convincing results and ablations.

Critical weaknesses
  1. While intuitively reasonable, the results show limited gain over previous methods. Therefore, it is hard to understand how effective the proposed strategy can be. More extreme settings (eg with even fewer retained tokens) could better depict weaknesses of the proposed and the previous methods. Maybe, the authors also potentially find qualitative examples that show where the intervention-based method is conceptually more effective than competing methods. This could substantially strengthen the argumentation of the paper.
  2. The intervention-driven strategy seems to be comparably expensive. Tab. 5 presents speedups. However, it remains a bit hard to understand what the bottlenecks in this pipeline are. Explaining this (and potentially adding more baseline + different budgets) could help evaluating this more effectively, while strengthening the paper.
Questions for authors

The pipeline contains various hyperparameters (such as in Eq. (5)). Could the authors explain / provide analysis of the sensitivity of the method with respect to the relevant hyperparameters?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.