acceptodds
Under review as a conference paper at ICLR 2027

Marginal Utility Token Elimination for Vision Transformers

Abstract

Existing token reduction methods leave their framework choice to ad-hoc heuristics over the backbone's inductive bias or to task-specific learning of an auxiliary reducer module. We instead organize token reduction into two axes, a partition that groups tokens into clusters and an aggregation that produces a representative for each cluster, and state a single explicit objective over both axes: a sensitivity-weighted clustering distortion in the key-projected space, which a first-order Taylor expansion of the next block motivates and three stated approximations make computable. The proposed MUTE minimizes this objective by alternating its two analytic coordinate updates, a weighted-mean aggregation and a nearest-representative partition, which descend the objective monotonically and recover the size bias of bipartite merging as a derived correction rather than as a heuristic. Prior token pruning, merging, and clustering reducers occupy restricted cells of the same design grid, each replacing at least one of the two coordinate updates with a mechanism chosen outside the objective. Under the unified protocol MUTE attains the first-rank training-free classification accuracy, and the same reducer transfers without modification to backbones spanning supervised and self-supervised pretraining and to tasks spanning dense prediction, fine-grained classification, and zero-shot domain shift. A systematic grid search shows that design choices inside the framework make up an equivalent family within multi-seed noise while design choices outside it incur substantial losses, locating the sensitivity of token reduction on the partition axis as the structure of the objective anticipates.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
vision transformer, token reduction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_f3rmOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes MUTE, a training-free token reduction method for ViTs. In essence, it applies K-means++ to token clustering. The squared attention mass ω_i = Σ_j P_ji² serves two roles: it weights the aggregation of each cluster's tokens into a representative, and it weights the K-means++ initialization. The procedure is framed as minimizing a "marginal utility" objective derived from a first-order Taylor expansion, though I did not find the extra packaging brings any additional insights.

Critical strengths

The method is reasonable overall: clustering-based reduction with attention-aware weighting is a natural and sensible design.

Critical weaknesses
  1. Over-packaging, with limited novelty over Token Pooling.

    • Token Pooling (Marin et al., WACV 2023) already uses attention-weighted K-means/K-medoids with weighted-mean aggregation. It is not compared in the experiments.
    • MUTE differs from it mainly in two ways, and neither is supported:
      • (i) Squared attention as the aggregation weight. The paper's own ablation shows it is unimportant: it differs from ToMe's size-weighted mean by only 0.02% (App. C).
      • (ii) K-means++ initialization instead of top-K. The paper calls this load-bearing (§4.5.2) but provides no explicit ablation.
    • Even with the missing experiments, the improvement is likely limited.
  2. The experiments lag behind the current field.

    • The evaluation is limited to ImageNet classification with small-to-mid ViTs.
    • Token reduction today matters most for MLLMs, video, and larger backbones, and none of these are evaluated.
  3. FlashAttention compatibility.

    • ω requires the explicit attention matrix, which FlashAttention never materializes. The log-size bias also needs kernel support.
    • The paper cites Representation Shift (ICCV 2025), which targets exactly this issue, but does not discuss it. A discussion of the cost and of possible alternatives would help.
Questions for authors

See Weaknesses

3 comments
AuthorsOpenReview ↗

Official Author Response to Reviewer f3rm (1/2)

We are grateful to the reviewer for the thorough review. The requests for an initialization ablation, an MLLM evaluation, and a discussion of FlashAttention each led to a new experiment, and we address each point below.

W1. Novelty

The K-Medoids row of Table 1 is Token Pooling, so the requested comparison is part of our main table. The paper of the testbed we build on (Haurum et al., 2023) attributes this baseline to Marin et al. Under the same backbone and protocol, after fine-tuning, MUTE attains 79.24% and Token Pooling 76.15%. In the revision we will rename the row to "Token Pooling (K-Medoids)" and cite Marin et al. directly.

On difference (i), the Appendix C result that the reviewer cites is what the paper reports. Our objective defines a family of reducers and not a single weight, and the weights inside the family fall in the same performance band.

On difference (ii), the comment that an explicit ablation is missing is valid, and we measured it as requested. Only the rule that selects the initial representatives is changed, with everything else fixed.

Initializationtop-1 change (%p)
Importance weighted K-means++ (ours)reference
Farthest-first, importance ignored−0.36
Top-K by importance−0.46
Random−2.01

The differences are measured on the full ImageNet-1k validation set and are statistically significant. We would like to highlight that the top-K rule, which Token Pooling uses, loses 0.46%p against our initialization under otherwise identical conditions. A rule that uses distance alone also loses accuracy, so the combination of the two criteria is what matters. We will include this table in the appendix as the explicit ablation supporting the statement of Section 4.5.2.

W2. Limited experiments

Thank you for pointing out the scope of our work. MLLMs and video fall outside the scope we target, and we will clarify this boundary in the revised manuscript. However, this does not mean that MUTE fails outside that scope. We further provide an experiment on an MLLM below.

Within the targeted scope, the paper covers both larger backbones and tasks beyond ImageNet classification. Table 2 includes ViT-L and ViT-H backbones (MAE-L, MAE-H, BEiT-L) under both supervised and self-supervised pretraining. Table 4 evaluates iNat 2019 classification, ADE-20K semantic segmentation, NYU-Depth-v2 depth estimation, four fine-grained datasets, and zero-shot domain shift at two keep ratios, and MUTE is the best reducer in eight cells out of ten.

For the MLLM experiment, we inserted the reduction rule unchanged into the CLIP ViT-L/14-336 vision tower of LLaVA-1.5-7B, at layers 6, 12, and 18 following the location convention of the paper, with nothing tuned for this task. The results on POPE, averaged over the three splits (9,000 questions), are as follows.

Visual tokensAccuracyF1
No reduction57685.8984.60
MUTE19786.1485.08
ToMe19785.8384.71
MUTE7284.7783.38
ToMe7280.0876.57

At 197 tokens, MUTE is statistically indistinguishable from the unreduced model, and MUTE and ToMe are indistinguishable from each other. At 72 tokens, an 8x reduction, MUTE loses 1.12%p against the unreduced model but remains 4.69%p above ToMe, a significant difference under a paired test (p=1.4×10−37p = 1.4 \times 10^{-37}). This reproduces the trend reported in the paper, parity at light reduction and a gap that grows as the budget tightens, on a different architecture, task, and modality. The scope of this evaluation is reduction inside the vision tower, with the reduction rule as the only variable. We will include these results in the appendix.

AuthorsOpenReview ↗

Official Author Response to Reviewer f3rm (2/2)

W3. Relationship with flash attention

This is a practically important observation, since FlashAttention is now the default in most deployments. The submission did not discuss this compatibility, and we will add the discussion in the revision. The submitted implementation forms the explicit attention matrix at the three reduction blocks, so FlashAttention cannot be applied there directly. We implemented a compatible variant, MUTE-F, and confirmed that it works without accuracy loss.

  • The log-size bias needs no kernel support. Appending a constant channel to the queries and a log⁡sk\log s_k channel to the keys makes the inner product equal to the original logit plus log⁡sk\log s_k, so the unmodified kernel computes it exactly.
  • The sensitivity weight ω\omega is estimated from 64 query rows, so the full attention matrix is never formed, and the cluster mean uses the size-weighted variant of Appendix C.
Attentiontop-1 change
MUTEExplicit at the three reduction blocksreference
MUTE-FFused in every block0.00

(DeiT-S/16, full ImageNet-1k validation set.)

The core of MUTE is its formulation and not a particular implementation, and this experiment indicates that the formulation does not conflict with FlashAttention. We will also discuss Representation Shift in this context in the revision.

Thanks to the authors for the detailed responses. The rebuttal has answered all my questions. However, I still feel the novelty of this work is limited. So I keep the score of 2 and hold a borderline attitude to this work.

Reviewer_eY57OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents MUTE, a training-free token pruning method that groups similar tokens and merges them to reduce the inference cost of vision transformers while preserving accuracy. The authors argue that existing methods mostly use heuristic rules to decide which tokens to group and how to merge them, without a common objective to guide both choices. Motivated by a first-order Taylor expansion and several approximations, they introduce a sensitivity-weighted clustering objective in the key-projected space. MUTE alternates between assigning tokens to the nearest cluster representative and updating each representative using an attention-weighted average. The paper also places prior pruning, merging, and clustering methods within this grouping-and-aggregation framework and studies the impact of each component. The evaluation compares accuracy, FLOPs, and inference throughput against existing methods and examines transfer across different backbones and downstream tasks, including segmentation and depth estimation. The reported results show competitive accuracy–efficiency trade-offs at moderate-to-aggressive pruning ratios and suggest that token grouping has a larger effect on accuracy than aggregation within the evaluated design

Critical strengths
  • The authors jointly consider grouping and aggregation, providing a framework to examine their interaction and assess the impact of different design choices.

  • MUTE does not require training an additional pruning module, making it easier to apply to existing pretrained models and adjust the pruning ratio at deployment.

  • The evaluation covers multiple backbones and downstream tasks, including classification, segmentation, and depth estimation.

Critical weaknesses
  • The novelty beyond prior clustering-based methods is unclear.

  • The presentation is difficult to follow because different experiments use different versions of MUTE and different training settings, making it hard to interpret the results.

  • Several technical and experimental issues weaken the evidence supporting the proposed method.

Questions for authors
  • Q1: What does MUTE add beyond prior clustering-based methods such as Token Pooling? A direct comparison would help establish the contribution, particularly since Appendix C shows that size weighting performs comparably to sensitivity weighting, while Appendix D finds cosine distance competitive with the proposed metric.

  • Q2: Could you clearly state which version of MUTE and which training settings are used in each experiment? For example, Table 1 uses Mahalanobis distance while Table 4 uses cosine distance, and some experiments freeze the backbone while others fine-tune it. Please explain these choices and clarify which results support the main claims.

  • Q3: Section 3.2 motivates the objective through a first-order Taylor expansion, but Table 14 reports that the discarded remainder is 75–2330 times larger than the retained term. How does this affect the justification for the proposed objective? Since Section 4.5.1 also shows that lower clustering distortion does not always improve accuracy, it would help to clarify what the theoretical formulation explains about MUTE’s empirical benefits.

  • Q4: Could you clarify the settings behind the main results in Tables 1 and 2? Appendix B describes linear-probe heads for the self-supervised backbones, but the MAE baseline accuracies closely match official fine-tuned checkpoints. Also, Section 4.2 acknowledges that the shared reduction schedule penalizes some baselines. How does MUTE compare when these methods use appropriate schedules at matched compute or inference throughput?

5 comments
AuthorsOpenReview ↗

Official Author Response to Reviewer eY57 (1/3)

We appreciate the reviewer's detailed and precise reading of our paper. The questions on the experimental settings identified an error in our appendix and a presentation problem across our tables, and we address each point below.

W1, Q1. Novelty

Table 1 contains a direct comparison with Token Pooling: its K-Medoids row is Token Pooling. The paper of the testbed we build on (Haurum et al., 2023) attributes this baseline to Marin et al. Under the same backbone and protocol, after fine-tuning, MUTE attains 79.24% and Token Pooling 76.15%, a gap of 3.09%p. The row label and the citation obscured this connection, so in the revision we will rename the row to "Token Pooling (K-Medoids)" and cite Marin et al. directly.

In terms of method, MUTE differs from prior clustering reducers at three points. It measures distance in the key space of the block that consumes the tokens. Once the representatives are chosen, it assigns every token independently in a single step, so the result does not depend on a merge order. It selects the initial representatives by both importance and mutual distance. The initialization criterion can be isolated by a controlled experiment. Changing only the rule that selects the initial representatives, with everything else fixed, loses 0.46%p with the top-K-by-importance rule that Token Pooling uses, 0.36%p with a rule that uses distance and ignores importance, and 2.01%p with random selection. These differences are measured on the full ImageNet-1k validation set and are statistically significant. We will include this ablation in the appendix to make the role of the initialization explicit.

Regarding Appendices C and D, both results point in the same direction as our claim. Our objective defines a family of reducers, not a single configuration. That size weighting is within 0.02%p of sensitivity weighting, and that cosine is competitive with Mahalanobis, means that the choices inside the family fall in the same band. What the paper reports is that choices leaving this band, namely those that change the partition rule, incur large losses. Section 4.5.3 attributes most of the top-1 variance to the partition axis.

AuthorsOpenReview ↗

Official Author Response to Reviewer eY57 (2/3)

W2, Q2, Q4. Setting unification and readability

We want to emphasize that the clarifications and corrections below do not affect the main claims of the paper. The paper presents three results: the highest training-free accuracy under a unified protocol (Table 1), transfer across backbones and tasks without modification (Tables 2 and 4), and the concentration of design-space sensitivity on the partition axis (Section 4.5).

The reviewer's comment made clear that the settings of each experiment need to be visible at a glance. In the revision we will add the table below to the main text and to the appendix.

ExperimentE-stepBackboneClassifier head
Table 1MahalanobisFrozen (TF), fine-tuned (FT)ImageNet-1k head
Table 2, supervisedMahalanobisFrozenCheckpoint head
Table 2, self-supervisedMahalanobisFrozenSee correction below
Table 4CosineFrozenTask head, trained
Appendix, multi-task FTMahalanobisFine-tunedTask head, trained

First, on classification the cosine variant used in Table 4 and the Mahalanobis variant are within multi-seed noise of each other. On dense prediction, Mahalanobis is stronger than the cosine variant. This means that the MUTE row of Table 4 reports the conservative variant, not the better of the two. We will state this in the table caption.

Second, regarding the multi-task results of the reference variant, we acknowledge that they are difficult to compare at a glance. The paper reports them, but in three separate places:

  • Table 4 is in the main text,
  • the same five tasks measured with the reference (Mahalanobis) variant are in Appendix M,
  • and the side-by-side comparison of the two variants is in Appendix D.

The results of the variant are as follows, and they are the best among the listed reduction baselines.

MUTE (reference)iNatADE mIoUNYUd RMSEFGDS
alpha = 0.573.9233.190.592268.6453.83
alpha = 0.774.5033.920.573170.4854.87

(Appendix M, backbone fine-tuned protocol.)

In the revision we will gather these values in one place as a minor update. We will keep the settings table above in the main text, and include the variant and the protocol directly in the caption and the row labels of the multi-task tables, so that the configuration behind each number can be read from the table itself. This changes the placement and labeling of existing results and does not add new measurements.

Third, the reviewer's observation on the classifier heads of the self-supervised backbones is correct. Unlike the description in Appendix B, MAE-B/L/H and BEiT-B/L use the official fine-tuned checkpoints, and only DINO ViT-B/8 and MoCo-v3 use linear probes. The measurements and the conclusions of the comparison are unaffected, and the comparison remains fair because every reducer is evaluated on the same checkpoints. We will correct the sentence in Appendix B by stating the head of each backbone separately. We thank the reviewer for pointing this out.

Lastly, regarding a matched-compute comparison under an appropriate schedule, we directly measured it. Appendix P evaluates a ToMe variant with six reduction locations instead of three, a schedule that favors ToMe. As stated in the submission, MUTE's accuracy at that variant's budget was obtained by interpolation. At the same 2.09 GFLOPs, MUTE attains 77.17% against 76.67% for the six-location ToMe. We will replace the interpolated value with this measurement in the revision.

Our main comparison matches the token budget. Every reducer is evaluated at the same keep ratio and the same reduction locations, and the FLOPs and throughput that each setting actually consumes are reported in the last three columns of Table 1, so the same table also supports a cost-versus-accuracy reading. We do not claim an advantage on the cost metrics. The main text and Appendix O state that the two cost metrics rank the reducers differently and that MUTE is the cheapest under neither, and they scope its operating regime to moderate-to-aggressive reduction.

AuthorsOpenReview ↗

Official Author Response to Reviewer eY57 (3/3)

W3, Q3. Effect of remainder terms in Taylor expansion

The ratio reported in Table 14 is a mean, and the distribution of the term-wise norms is heavy-tailed. A small number of tokens dominates the mean, so it hides how the expansion behaves for a typical token. In terms of medians, the expansion decreases monotonically at the deepest reduction stage: the first-order term is 16.89 and the following terms are 4.16, 2.58, 1.82, and 1.38, so the first-order term dominates at about four times the second. At the two earlier stages the expansion does not decrease, and we will report this difference together with the result above in the revision.

StageOrder 1Order 2Order 3Order 4Order 5Monotone decrease
1st11.659.8310.1611.2013.45No
2nd15.328.759.4212.2317.17No
3rd16.894.162.581.821.38Yes

(DeiT-S/16, median norm of each Taylor term.)

Our objective is not used to predict the magnitude of the loss. Its role is to rank candidate partitions, and the reduction rule makes the right choice as long as that order is preserved. Our claims therefore do not depend on the absolute size of the remainder, but on whether the first-order term orders the partitions reliably. The result of Section 4.5.1, that lower distortion does not always give higher top-1, should be read within the same scope. We did not present the objective as a quantitative predictor. We will state this role explicitly in Section 3.2 in the revision.

We tested this directly on the full ImageNet-1k validation set. The rate of errors induced by reduction rises with the objective value, from 1.82% in the lowest decile to 5.26% in the highest, and a Cochran-Armitage trend test confirms that the increase is significant with z=10.99z = 10.99. This is not an effect of image difficulty. Over deciles 1 to 8 the error rate of the unreduced model stays between 18.3% and 20.0% and shows no trend, with z=0.76z = 0.76. Over the same deciles the induced error rate still rises from 1.82% to 2.90%, and this trend is significant with z=5.26z = 5.26. The objective therefore orders inputs by the damage that reduction actually causes.

Reviewer_MdocOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Marginal Utility Token Elimination (MUTE), a training-free token reduction method for Vision Transformers. The authors formulate token reduction as a joint partition-and-aggregation problem and introduce a sensitivity-weighted clustering objective in the key-projected feature space. Based on a first-order Taylor approximation, MUTE alternates between assigning tokens to their nearest representatives and updating the representatives using sensitivity-weighted mean aggregation. Experimental results show that MUTE achieves better classification accuracy than existing token reduction methods, including ToMe and ATC. The authors also evaluate its generalization across different backbones and downstream tasks.

Critical strengths

The paper presents a clear optimization framework that jointly considers token partitioning and aggregation. This formulation provides a useful perspective for understanding and comparing existing pruning, merging, and clustering methods.

Critical weaknesses
  1. novelty. Despite the theoretically motivated formulation, the proposed algorithm is essentially sensitivity-weighted K-means, consisting of standard nearest-centroid assignment and weighted-mean updates. Since clustering-based token reduction has been investigated in previous work, the main contribution appears to be the formulation and combination of existing techniques rather than a substantially new reduction algorithm.

  2. The proposed distortion objective relies on several approximations, including replacing the Jacobian-induced metric with the key-projection metric and ignoring inter-token interactions. Furthermore, the authors acknowledge that lower distortion does not always translate into higher classification accuracy. This raises concerns about how well the theoretical objective explains the empirical improvements, rather than simply serving as a surrogate optimization objective.

  3. Although MUTE achieves better accuracy, the additional clustering operations introduce computational overhead, resulting in lower throughput than simpler methods such as ToMe (6,027 vs. 7,367 images/s). Moreover, the performance improvements are sensitive to the token reduction schedule and compression ratio. As discussed in the paper, alternative schedules can reduce or even reverse the performance gap, suggesting that the practical advantages of MUTE may be less consistent than the main comparisons indicate.

Questions for authors
  1. For novelty, can you compare your method with recent clustering based method?

[1] Not All Tokens Are Equal: Human-centric Visual Analysis via Token Clustering Transformer

[2] Agglomerative Token Clustering

3 comments
AuthorsOpenReview ↗

Official Author Response to Reviewer Mdoc (1/2)

We sincerely thank the reviewer for the careful and constructive review. The comments on the approximation and on the clustering overhead prompted new measurements that sharpened the paper, and we address each point below.

W1, Q1. Novelty

MUTE differs from Agglomerative Token Clustering (ATC) and TCFormer (DPC-kNN) in its distance metric, merging procedure, and initialization. Both requested methods are already included in Table 1 of the submission, where ATC appears as the ATC row and TCFormer's clustering as the DPC-kNN row. Under the same backbone and protocol, MUTE attains 78.57%, ATC 77.76%, and DPC-kNN 68.95%. We will add the full method names to the row labels in the revision.

The three differences are as follows:

  • Distance metric: ATC normalizes the KK-vectors, so token magnitude is ignored. TCFormer measures distance in a space that has not passed through the projection the block actually uses. MUTE measures distance in the key space of the consuming block, M=WK⊤WKM = W_K^\top W_K, which is the space the next block actually operates in.
  • Merging procedure: ATC's agglomerative merging is sequential and path-dependent; one merge alters the subsequent merges within the same layer. In MUTE, once the representatives are chosen, every token is assigned independently in a single step, so the result does not depend on a merge order.
  • Initialization: ATC selects no initial representatives and starts from every token as its own cluster. TCFormer initializes by density, which concentrates representatives in crowded regions. MUTE selects tokens that are both important and far from one another.

The center of our contribution is the derivation. The metric, the assignment rule, and the aggregation weight are not separately chosen design decisions; they follow from a single objective, and the initialization is weighted by the same sensitivity so that it serves that objective.

AuthorsOpenReview ↗

Official Author Response to Reviewer Mdoc (2/2)

W2. Approximation

Regarding the replacement of the Jacobian-induced metric by the key-projection metric, the submission measures its effect directly. Table 10 in Appendix D clusters with the Jacobian-induced metric itself and obtains 78.51%, against 78.57% for the key-projection metric, at about 960 times the cost. The approximation therefore does not cost accuracy. Regarding token interactions, the layer-ll tokens already carry the interactions up to layer ll, and WKl+1W_K^{l+1} is a fixed linear map that does not mix tokens. The interactions that will occur in the next layer are indeed not reflected in the metric.

Table 14 reports the mean ratio of the remainder to the first-order term (75 to 2,330), and the distribution of the term-wise norms is heavy-tailed. A small number of tokens dominates the mean, so it hides how the expansion behaves for a typical token. The medians of the same measurement show that the behavior depends on the stage.

StageOrder 1Order 2Order 3Order 4Order 5Monotone decrease
1st11.659.8310.1611.2013.45No
2nd15.328.759.4212.2317.17No
3rd16.894.162.581.821.38Yes

(DeiT-S/16, median norm of each Taylor term.)

At the deepest reduction stage the expansion decreases monotonically, and the first-order term dominates at about four times the second-order term. At the two earlier stages it does not decrease. Our objective is therefore used as a ranking function over candidate partitions rather than as a predictor of the loss value, and its regime of validity concentrates at the deep stage. In the revision we will add this median decomposition to Appendix F, which currently reports only the mean ratio, and state the role of the objective as a ranking function explicitly.

Regarding prediction drift at low distortion, we examined it directly. For every one of the 50,000 ImageNet-1k validation images we recorded the value of our objective together with the predictions of the model before and after reduction. We define a prediction drift as an image that the unreduced model classifies correctly and the reduced model classifies incorrectly.

Prediction drift at low distortion is rare, and it occurs on images where the unreduced model already had a very small margin between its top two classes. Among the 10,000 images in the lowest 20% of the objective value, 194 drift, which is 1.94%. The median logit margin of the unreduced model on these 194 images is 0.35, about ten times smaller than the 3.69 of the non-drifting images in the same range and the 3.47 of all images. The margin is below 1.0 for 84.0% of these images, against 17.7% of all images. In 99.0% of the 194 cases the correct class remains in the top-5 of the reduced model. The residual errors at low distortion are therefore swaps between the top-1 and top-2 classes on images with a very small margin, not catastrophic failures.

The same measurement also shows that the drift rate rises with the objective value, from 1.82% in the lowest decile to 5.26% in the highest, and a Cochran-Armitage trend test confirms that this increase is significant with z=10.99z = 10.99. Both the objective and the first-order approximation behind it are therefore useful in practice, since the objective orders images by the damage that reduction actually causes, which is the role we assign to it as a ranking function.

W3. Computational overhead on clustering

Most of the overhead comes from an implementation pattern. We measured where it arises: the sequential loop that selects the initial representatives accounts for 80% of it.

ConfigurationThroughput vs. unreduced
Current implementation1.50x
Without the sequential initialization loop1.76x
No clustering at all1.83x

The gap to ToMe depends on the compression ratio because MUTE and ToMe do not reach the same budgets. Under the aggressive setting the targeted token counts are 58, 17, and 5 per stage. MUTE realizes them and computes 1.84 GFLOPs, while bipartite matching leaves ToMe at 98, 57, and 45 tokens and 2.43 GFLOPs. The advantage therefore grows as the budget tightens, and we scope our claim to moderate-to-aggressive reduction.

At equal compute the gap does not reverse. Appendix P evaluates a ToMe variant with six reduction locations instead of three, a schedule that favors ToMe. As stated in the submission, MUTE's accuracy at that variant's budget was obtained by interpolation. We have now measured it directly: at the same 2.09 GFLOPs, MUTE attains 77.17% against 76.67% for the six-location ToMe. We will replace the interpolated value with this measurement in the revision.

Thanks for the authors for their rebuttal, which addresses my concerns. However, the limited novelty of the proposed K-means-style algorithm leads me to maintain my original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Notification of Revised Manuscript

Dear Reviewers,

We sincerely thank you for the time and thoughtful feedback during the discussion phase. We have uploaded a revised version of the manuscript reflecting your suggestions. All substantial updates are highlighted in blue. The numbering of the sections and tables cited in our responses is unchanged; the new settings table is labeled Table S and the new experiments are collected in Appendix V, to be integrated into the regular numbering in the camera-ready version.

Summary of Key Updates

Main Manuscript

  • Role of the Objective: Stated that the objective serves to rank candidate reductions (§3.2).
  • Baseline Names: Added the full names of ATC, TCFormer (DPC-kNN), and Token Pooling (K-Medoids) (§4.1, Table 1).
  • Experimental Settings: Added a table of the MUTE variant, backbone state, and classifier head behind each experiment (§4.1, Table S).
  • Schedule Sensitivity: Added a direct measurement against the six-location ToMe at the same computational cost (§4.3, Appendix P).
  • Scope: Clearly stated the scope as vision transformer encoders on image tasks (§5).

Appendix

  • Corrections and Labels: Corrected the classifier-head description and labeled the variant and protocol of the multi-task tables (Appendix B, D, M, Table 4).
  • Taylor Diagnostics: Added an order-by-order decomposition of the Taylor expansion (Appendix F, Table 15).
  • Additional Experiments: Added an initialization ablation, an MLLM evaluation on POPE, a FlashAttention-compatible variant with a discussion of Representation Shift, and a prediction-drift analysis (Appendix V, §4.2).

We hope these revisions address your main concerns and would greatly appreciate your re-evaluation of the updated manuscript. We welcome any further questions.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.