- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes MUTE, a training-free token reduction method for ViTs. In essence, it applies K-means++ to token clustering. The squared attention mass ω_i = Σ_j P_ji² serves two roles: it weights the aggregation of each cluster's tokens into a representative, and it weights the K-means++ initialization. The procedure is framed as minimizing a "marginal utility" objective derived from a first-order Taylor expansion, though I did not find the extra packaging brings any additional insights.
Critical strengths
The method is reasonable overall: clustering-based reduction with attention-aware weighting is a natural and sensible design.
Critical weaknesses
-
Over-packaging, with limited novelty over Token Pooling.
- Token Pooling (Marin et al., WACV 2023) already uses attention-weighted K-means/K-medoids with weighted-mean aggregation. It is not compared in the experiments.
- MUTE differs from it mainly in two ways, and neither is supported:
- (i) Squared attention as the aggregation weight. The paper's own ablation shows it is unimportant: it differs from ToMe's size-weighted mean by only 0.02% (App. C).
- (ii) K-means++ initialization instead of top-K. The paper calls this load-bearing (§4.5.2) but provides no explicit ablation.
- Even with the missing experiments, the improvement is likely limited.
-
The experiments lag behind the current field.
- The evaluation is limited to ImageNet classification with small-to-mid ViTs.
- Token reduction today matters most for MLLMs, video, and larger backbones, and none of these are evaluated.
-
FlashAttention compatibility.
- ω requires the explicit attention matrix, which FlashAttention never materializes. The log-size bias also needs kernel support.
- The paper cites Representation Shift (ICCV 2025), which targets exactly this issue, but does not discuss it. A discussion of the cost and of possible alternatives would help.
Questions for authors
See Weaknesses
3 comments
Official Author Response to Reviewer f3rm (2/2)
W3. Relationship with flash attention
This is a practically important observation, since FlashAttention is now the default in most deployments. The submission did not discuss this compatibility, and we will add the discussion in the revision. The submitted implementation forms the explicit attention matrix at the three reduction blocks, so FlashAttention cannot be applied there directly. We implemented a compatible variant, MUTE-F, and confirmed that it works without accuracy loss.
- The log-size bias needs no kernel support. Appending a constant channel to the queries and a channel to the keys makes the inner product equal to the original logit plus , so the unmodified kernel computes it exactly.
- The sensitivity weight is estimated from 64 query rows, so the full attention matrix is never formed, and the cluster mean uses the size-weighted variant of Appendix C.
| Attention | top-1 change | |
|---|---|---|
| MUTE | Explicit at the three reduction blocks | reference |
| MUTE-F | Fused in every block | 0.00 |
(DeiT-S/16, full ImageNet-1k validation set.)
The core of MUTE is its formulation and not a particular implementation, and this experiment indicates that the formulation does not conflict with FlashAttention. We will also discuss Representation Shift in this context in the revision.
Thanks to the authors for the detailed responses. The rebuttal has answered all my questions. However, I still feel the novelty of this work is limited. So I keep the score of 2 and hold a borderline attitude to this work.
Official Author Response to Reviewer f3rm (1/2)
We are grateful to the reviewer for the thorough review. The requests for an initialization ablation, an MLLM evaluation, and a discussion of FlashAttention each led to a new experiment, and we address each point below.
W1. Novelty
The K-Medoids row of Table 1 is Token Pooling, so the requested comparison is part of our main table. The paper of the testbed we build on (Haurum et al., 2023) attributes this baseline to Marin et al. Under the same backbone and protocol, after fine-tuning, MUTE attains 79.24% and Token Pooling 76.15%. In the revision we will rename the row to "Token Pooling (K-Medoids)" and cite Marin et al. directly.
On difference (i), the Appendix C result that the reviewer cites is what the paper reports. Our objective defines a family of reducers and not a single weight, and the weights inside the family fall in the same performance band.
On difference (ii), the comment that an explicit ablation is missing is valid, and we measured it as requested. Only the rule that selects the initial representatives is changed, with everything else fixed.
The differences are measured on the full ImageNet-1k validation set and are statistically significant. We would like to highlight that the top-K rule, which Token Pooling uses, loses 0.46%p against our initialization under otherwise identical conditions. A rule that uses distance alone also loses accuracy, so the combination of the two criteria is what matters. We will include this table in the appendix as the explicit ablation supporting the statement of Section 4.5.2.
W2. Limited experiments
Thank you for pointing out the scope of our work. MLLMs and video fall outside the scope we target, and we will clarify this boundary in the revised manuscript. However, this does not mean that MUTE fails outside that scope. We further provide an experiment on an MLLM below.
Within the targeted scope, the paper covers both larger backbones and tasks beyond ImageNet classification. Table 2 includes ViT-L and ViT-H backbones (MAE-L, MAE-H, BEiT-L) under both supervised and self-supervised pretraining. Table 4 evaluates iNat 2019 classification, ADE-20K semantic segmentation, NYU-Depth-v2 depth estimation, four fine-grained datasets, and zero-shot domain shift at two keep ratios, and MUTE is the best reducer in eight cells out of ten.
For the MLLM experiment, we inserted the reduction rule unchanged into the CLIP ViT-L/14-336 vision tower of LLaVA-1.5-7B, at layers 6, 12, and 18 following the location convention of the paper, with nothing tuned for this task. The results on POPE, averaged over the three splits (9,000 questions), are as follows.
At 197 tokens, MUTE is statistically indistinguishable from the unreduced model, and MUTE and ToMe are indistinguishable from each other. At 72 tokens, an 8x reduction, MUTE loses 1.12%p against the unreduced model but remains 4.69%p above ToMe, a significant difference under a paired test (p=1.4×10−37). This reproduces the trend reported in the paper, parity at light reduction and a gap that grows as the budget tightens, on a different architecture, task, and modality. The scope of this evaluation is reduction inside the vision tower, with the reduction rule as the only variable. We will include these results in the appendix.