- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper investigates when visual tokens should be removed during visual encoding. Its motivation distinguishes between retaining a token in a frozen intermediate state and allowing it to continue participating in subsequent computation. ICTR estimates the influence of each active token on other tokens by measuring the exact change in the current attention-head output caused by removing that token's key/value contributions, while keeping the current queries, keys, and values fixed. The method averages the relative output changes over a selected subset of the most affected recipient tokens, ranks source tokens by the resulting influence scores, and progressively removes low-influence tokens after confirmation across two consecutive blocks, until a fixed final token budget is reached. Experiments cover LLaVA-1.5, LLaVA-NeXT, and Qwen2.5-VL, with ablations on the influence formula, active-set reevaluation, and confirmation mechanism. With 128 retained tokens on LLaVA-1.5, the paper reports 98.9% normalized performance retention and a 2.048× prefill speedup.
Critical strengths
- The scoring criterion has a concrete intervention-based interpretation. Equation 3 measures the immediate effect of removing a source token rather than simply treating attention weights as importance scores. The coefficient-only and frozen-score controls in Table 4 help isolate the contribution of value-output differences and the role of active-set reevaluation. Figure 3 further examines downstream representation perturbations under a shared prefix. Combining an exact local measure with controlled downstream diagnostics makes the contribution more informative than another insufficiently justified importance heuristic.
- The empirical evaluation demonstrates both improvements and architectural limitations. Table 2 shows that the proposed method substantially outperforms the compared methods on the high-resolution LLaVA-NeXT architecture.
Critical weaknesses
- Several relevant methods are missing from the comparisons. In particular, two related training-free baselines should be considered. AutoPrune (Wang et al., 2025) evaluates five overlapping tasks on LLaVA-1.5-7B under nominal token budgets of 192, 128, and 64 in Table 1. CDPruner (Zhang et al., 2025) reports results with 128 and 64 tokens, as well as 32 tokens, on the same backbone across ten overlapping benchmarks in Table 1.
- The evidence supporting the specific two-observation timing criterion is relatively limited and backbone-dependent. In Table 4, removing the confirmation mechanism changes the normalized mean by only 0.40–0.73 points. Although downstream perturbation analyses provide additional evidence, the advantage over current-layer scoring remains limited.
Questions for authors
Please address the issues identified as weaknesses.