acceptodds
Under review as a conference paper at ICLR 2027

What Survives Token Merging? Merge-Invariant Low-Rank Attention for Vision Transformers

Abstract

Token reduction and low-rank factorization offer complementary ways to accelerate vision transformers, yet their composition introduces dependencies that neither addresses independently. Factorizing the joint query–key product changes the representations used for token merging, while merging destroys the stable positional correspondence required by conventional logit-residual correction. We resolve both incompatibilities without retraining. First, we characterize fixed logit corrections invariant to patch relabelling: their space is five-dimensional, but two directions are invisible to the softmax, leaving exactly three effective degrees of freedom. These three coefficients per head recover and of the accuracy recovered by a -entry position-indexed correction on DeiT-Small and DeiT-Tiny, respectively, while remaining applicable after merging. Second, we decouple token matching from the current query–key factorization by computing similarity from normalized block inputs, improving matching stability under rank reduction. We further identify an accuracy–compute crossover: mild token reduction favors merging alone, whereas more aggressive reduction benefits from combining both compression axes. Beyond this crossover, our method improves accuracy by up to percentage points at matched compute and achieves up to throughput at matched accuracy, with no retraining or gradient computation. Experiments across DeiT scales, ViT-L, and an alternative token-reduction rule establish that the composition holds beyond a single setting, and results on ADE20K segmentation and five CLIP image encoders examine the broader applicability of low-rank correction. Code and compressed checkpoints will be released to support reproduction and further work.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision transformers, efficient inference, training-free, token merging, low-rank approximation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pibCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1(−1)
Read review
Paper summary

The paper studies the training-free composition of token merging and query-key rank reduction in pretrained vision transformers. It factorizes the bias-augmented query-key product with an activation-weighted truncated SVD. For the resulting logit error, it proves that a fixed correction invariant to arbitrary patch permutations has five parameters, two of which are softmax-invisible row shifts, leaving three effective coefficients per attention head. To reduce the dependence of merge decisions on the current factorization, it computes token similarity from normalized block inputs instead of compressed keys. ImageNet experiments on DeiT and ViT-L identify a compute-dependent crossover where the combination outperforms token reduction alone; throughput measurements on an RTX 3090 partly validate the FLOP-based comparison. Additional ADE20K and CLIP experiments evaluate rank reduction and correction without token merging.

Critical strengths
  1. Structured logit correction. Proposition 1 characterizes five matrix directions invariant to patch permutations and identifies three that remain observable after row-wise softmax. Table 1 supports the predicted null effect of row-wise shifts and reports the accuracy recovered by the three-coefficient correction without token reduction.
  2. Conditional evaluation of the combined method. Tables 2&3 report where combining token merging with QK rank reduction improves the accuracy-cost trade-off and where token merging alone performs better. Table 4 isolates the effect of the features used for token matching; Table 3 also reports measured throughput alongside the compute comparisons.
Critical weaknesses
  1. Incomplete comparison with recent training-free baselines. The paper’s central empirical claim concerns the accuracy-cost benefit of combining token reduction with query-key factorization. However, Table 5 omits recent training-free token-reduction methods such as (Lee & Choi, 2025), despite discussing that work in Section2 Table2 compares the combination against the authors’ block-input token-merging baseline, but does not establish whether the advantage persists against stronger recent single-axis methods
  2. The abstract highlights gains of “up to 4.34 percentage points” at matched compute and “up to 1.06× throughput” at matched accuracy. The 4.34-point result comes from the DeiT-Tiny, reduction-rate-18, rank-32 setting in Table 2, while many milder-reduction settings show small or negative gains. The 1.06× throughput result comes from separate DeiT-Small development-set experiments in Table 3. Although the paper acknowledges a crossover between regimes, presenting only these maxima in the abstract risks giving an inflated impression of their typical magnitude.
Questions for authors

Please refer to the weaknesses part

3 comments

Thank you for recognizing the structural correction result. We have added the recent baseline you identified and clarified the operating regime and magnitude of the gains.

1. Recent training-free baselines.

Table 6 now includes ATM (Lee & Choi, 2025), evaluated using its official implementation on all 50,000 ImageNet validation images. ATM achieves 79.76% at 3.28 GMAC, and our composition achieves 79.46% at 3.25 GMAC. ATM’s compute is calculated from its observed token schedule using the same counting convention as our method. This provides the requested recent comparison at approximately matched compute and identifies ATM’s accuracy advantage at this mild-compression setting.

Our composition addresses the regime in which dividing compression across both axes becomes beneficial. On the 45,000-image test split, the rate-16/rank-32 configuration achieves 77.76% at 2.064 GFLOPs, compared with 76.66% at 2.059 GFLOPs for token reduction alone using the same block-input matching (Table 14): 1.10 additional accuracy points at approximately matched model compute. This controlled comparison isolates the benefit of composition; it does not establish superiority over ATM at that budget. The revised evaluation places the method alongside the recent baseline while identifying where combining token and rank reduction improves the accuracy–cost trade-off.

2. Magnitude of the highlighted gains.

We revised the abstract and introduction to report 0.6–4.3 percentage points at matched model compute across the evaluated DeiT backbones at reduction rates 16–18, rather than presenting only the maximum. The abstract explicitly states that mild reduction favors merging alone and identifies the stronger-reduction regime in which composition helps.

Table 3 retains the small and negative gains at milder settings. Table 4 separately reports measured throughput on DeiT-Small and labels the development split. Its 1.06× speedup is an additional gain over token merging alone at matched accuracy, rather than a comparison against the dense model. The revised presentation connects each result to the model, compression budget, evaluation split, and baseline that support it.

I thank the reviewer for their response, but my concerns remain. Compared with ATM, a work from 2025, the proposed method achieves lower accuracy (79.76 → 79.46) with a similar computational cost (3.28 GMACs vs. 3.25 GMACs). What, then, is the significance of this improvement?

Thank you for the follow-up. As discussed in Section 4.4 and demonstrated in Table 3, the improvement we claim occurs beyond the crossover, where further token merging becomes costly in accuracy. The ATM comparison at approximately 3.25 GMAC evaluates the mild-compression regime, where adding rank reduction offers little benefit. At approximately 2.06 GFLOPs, composition achieves 77.76% versus 76.66% for token reduction alone using the same matching rule, a 1.10-point accuracy gain at approximately matched compute (Table 14). Our contribution is identifying this regime and enabling effective composition without retraining. These results establish the benefit over the controlled token-reduction baseline.

Reviewer_jF4EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper combines token merging with query–key low-rank compression to speed up ViTs without retraining. Based on a theoretical analysis, it uses three coefficients per attention head to correct compression errors. It also computes token similarity from normalized block inputs to reduce interference between the two methods.

Critical strengths
  1. Clear problem diagnosis. The paper identifies why token merging and low-rank factorization can interfere with each other, affecting both residual correction and token matching.
  2. Simple and well-grounded solution. The authors prove that fixed additive logit corrections that are invariant to patch permutations have three effective degrees of freedom after removing softmax-invariant row shifts. This result supports a compact correction with three scalars per head. The method requires calibration but no retraining, which makes it easy to apply to existing models.
Critical weaknesses

W1. The correction contributes little to the largest reported gains and reduces accuracy at the highest tested rates on DeiT-Tiny. Tables 11–12 allow the share of the correction in the Table 2 gains to be separated. In the largest cell (DeiT-Tiny, r=32r=32, rate 18), the composition gains +4.34+4.34 over token reduction alone, but the correction adds only +0.18+0.18; for DeiT-Base (r=48r=48, rate 18), it adds +0.08+0.08. At rates 20 and 22, it lowers DeiT-Tiny accuracy in all four cells by 0.26–0.60 points. The correction does help at rate 16 (Table 10), but the paper presents it as one of the two components that enable the composition (p. 2, l. 79–80), and this is not shown where the composition gains most. Please report results with and without the correction side by side in Table 2, include rates 20 and 22, and provide paired significance tests for the correction at each measured configuration.

W2. The efficiency comparisons are not specified or validated in enough detail. Tables 2 and 16 compare at matched compute, and Table 3 at matched throughput or matched accuracy, but the matching procedures, baseline points, and interpolation are not fully specified. Table 12 also omits the GFLOPs of the DeiT-Tiny and DeiT-Base composition settings. Throughput is measured only for DeiT-Small on one device (1.02×1.02\times–1.06×1.06\times), with development-set accuracy. Since all configurations materialize the attention logits for the size bias (p. 14), it is also unclear whether the advantage holds with fused attention implementations.

W3. A transported full table is missing as a baseline. The paper states that under reduction “only the three remain defined.” However, retaining the merge groups allows transport of the full table as TΔT⊤T\Delta T^\top. The reported failure does not clarify whether transport is used. This comparison matters: on DeiT-Base at r=12r=12, the three coefficients recover only 60.4% of the full-table gain without merging. Please compare accuracy and cost against a transported table, and against the full table without merging on ADE20K and CLIP.

W4. The evidence for generalization is limited. The ViT-L and EViT results use only the development split (Tables 13 and 16). The highlighted ViT-L gain is 0.32 percentage points, with no paired significance test. My interpolation within the baseline’s compute range gives gains from −0.32 to +0.43 percentage points. Although the conclusion is limited to budgets beyond the crossover, test-set results and paired tests at measured configurations are needed.

Questions for authors

Q1. Which configurations show significant gains or losses from the correction under paired tests, including rates 20 and 22? Please separate its contribution from the overall composition gains in Table 2.

Q2. How does the transported full table, TΔT⊤T\Delta T^\top, compare with the three coefficients in accuracy and cost? For the table calibrated at rate 0 and applied at rate 20, please explain the indexing and whether transport was used.

Q3. What matching procedures, baseline points, and interpolation methods produce Tables 2, 3, and 16? Do the gains hold at directly measured configurations? Can you also evaluate whether they persist with fused attention implementations?

Q4. Can you report the ViT-L results on the test split, with paired significance tests at directly measured configurations?

1 comment

Thank you for recognizing the theoretical basis of the compact correction. We have strengthened the experiments to separate its contribution and directly evaluate the comparisons you requested. References below use the revised manuscript’s numbering.

W1/Q1. Correction contribution and paired tests.

Table 12 now reports 60 paired comparisons on 45,000 test images, spanning three backbones, four ranks, and rates 12–22. At rates 12–18, all twelve comparisons at ranks 24 and 16 on DeiT-Tiny/Small show significant improvements under the reported unadjusted McNemar tests. At rate 16, correction contributes up to 2.50 points on Tiny and 1.42 on Small. At rank 16 under stronger merging, it adds +1.44 on Tiny at rate 20 and +1.04/+0.97 on Small at rates 20/22 (p=5.06×10−17p=5.06\times10^{-17}, 2.27×10−132.27\times10^{-13}, and 8.46×10−108.46\times10^{-10}).

We now distinguish these contributions from the overall composition gain: in the +4.34-point configuration, correction contributes +0.18 points (p=0.22p=0.22). Table 2 isolates correction in the main paper; Tables 14–15 provide accuracies with and without it. Table 12 includes all high-rate results, including significant losses on Tiny at rank 48/rates 20–22 and rank 32/rate 22, and on Base at rank 16/rates 18–22. This separates the benefit of allocating compression across both axes from the additional accuracy recovered by correction.

W2/Q3. Matching procedures and efficiency.

Appendix B.1 specifies how directly measured composition points are compared with the token-reduction baseline through interpolation in compute, throughput, or accuracy. Tables 14–15 provide the absolute accuracy and compute grids, including Tiny and Base. The benefit also appears without interpolation: Tiny at rate 16/rank 32 reaches 67.74% at 0.556 GFLOPs, versus 65.92% at 0.560 GFLOPs for merging alone at rate 18, a 1.82-point gain with slightly less model compute.

Runtime evaluation now includes an H100 and DeiT-Base. All eight H100 configurations improve accuracy at matched throughput, with 1.04–1.08× throughput over token reduction alone at matched accuracy (Table 28). These are additional gains over an already accelerated baseline. Table 27 identifies throughput gains at batches 128 and 256. Section 4.4 also reports an additional fused-attention comparison yielding 1.04× throughput at matched accuracy, separately from the original materialized-attention measurements. Runtime tables retain their development-split labels; the main DeiT accuracy–compute results use the test split.

W3/Q2. Transported full table and cross-task comparisons.

We implemented the proposed baseline, TΔT⊤T\Delta T^\top, with TT averaging original tokens within each current merge group. At merge rate 16 and rank 32, it shows no significant accuracy advantage over three coefficients on Small or Base (p=0.170p=0.170 and 0.1580.158). It stores 38,809 values per head instead of three, with 16.5–22.7% lower throughput across the tested configurations. At rank 12, it adds 1.32 and 0.64 accuracy points (Table 13). The revised text distinguishes fixed position-free correction from a positional table transported using merge membership; the new Table 13 explicitly evaluates transport.

We also completed the requested comparison without merging on ADE20K and all five CLIP encoders (Table 29). Both corrections improve over uncorrected truncation in every configuration shown. At rank 16, the full table adds 0.55 mIoU on ADE20K and 0.64–1.26 accuracy points on CLIP over the compact correction. Differences are smaller at rank 32, where three coefficients give higher measured accuracy on two CLIP encoders. These experiments quantify both the value of positional information and the efficiency of the compact correction.

W4/Q4. ViT-L test-set evaluation.

Table 17 now evaluates ViT-L/16-384 on 45,000 test images. Within the measured baseline compute range, the two composition configurations differ from merging alone by −0.05 and +0.01 points after interpolation. The revised main text reports this as closely tracking the baseline, replacing the highlighted development-set improvement with the test-set finding.

The evaluation now separates the roles of the experiments: DeiT test results establish the accuracy–compute gains, ViT-L examines a larger backbone and longer sequences, and EViT tests a different reduction criterion on the explicitly labeled development split. Table 17 provides the requested test-set evaluation. The paired significance tests requested in Q4 remain outstanding.

Reviewer_uuXKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problems that arise when token merging, which combines similar image tokens, is used together with reducing the attention dimension. ToMe, the token merging method used in this paper, merges tokens with similar keys. However, reducing the attention dimension changes the key representations, so tokens that were originally similar may be judged differently. In addition, once tokens are merged, an error correction built for the original token positions can no longer be used as is. The authors build three correction coefficients that remain valid when token positions change, and select tokens to merge using the token features before the query and key projections, instead of the compressed keys. Under strong compression, accuracy improves by up to 4.34pp at the same compute, but the actual speedup at equal accuracy is at most 1.06x.

Critical strengths
  1. The paper clearly explains why problems arise when the two compression methods are combined.
  2. It selects tokens to merge more stably by using token features computed before attention, instead of keys that can change with compression.
  3. It can be applied with only a small amount of data, without retraining the model or computing gradients.
  4. The authors also report that under mild compression, using token merging alone is better than combining the two methods.
  5. It reports measured GPU throughput in addition to theoretical compute.
Critical weaknesses
  1. The experiments that fully combine the two compression methods are mostly limited to ImageNet image classification. It is not sufficiently shown whether the same effect holds on other types of tasks.
  2. The three correction coefficients have the advantage of remaining valid when token positions change, but they cannot represent all errors that depend on image position. The paper itself observes spatial structure in the error remaining after correction.
  3. The main compute comparisons do not include the cost of computing token similarity. Actual throughput measurements include this cost, but the speedup at equal accuracy is at most 1.06x, which is not large.
  4. Experiments are run on a single GPU with a specific batch size. It is not shown whether the same speedup holds on other GPUs or with small, near-real-time batch sizes.
  5. Although no retraining is needed, separate calibration images are required to set the compression and the correction coefficients. It is unknown whether the method works well when calibrated on images that differ from the evaluation data.
Questions for authors

See Weaknesses

1 comment

Thank you for the careful reading and for recognizing the problem diagnosis, stable matching strategy, and training-free formulation. We have expanded the evaluation around your five concerns. References below use the revised manuscript’s numbering.

1. Applicability beyond the original experiments.

We added controlled composition experiments on CLIP ViT-L/14, including merging-only, rank-reduction-only, and corrected/uncorrected composition on 10,000 test images (Appendix D.10, Table 30). Correction improves the composed configurations by 0.72 and 1.87 accuracy points. These controls distinguish the benefit of correction from the choice of compression axes: rank reduction is more costly in accuracy for this encoder, and the evaluated budgets favor merging alone.

The new comparison also evaluates compact and full-table correction on ADE20K and five CLIP encoders (Table 29). Both corrections improve over uncorrected truncation in every configuration shown. On ADE20K, rank reduction preserves the spatial grid required by our DPT implementation. Together with EViT, these experiments extend the evaluation across encoder families, reduction rules, and dense prediction, while identifying where composition improves the accuracy–compute trade-off.

2. Position-dependent residual error.

We directly measured the value of retaining positional information. Table 13 compares three coefficients against a full table transported through the merge groups as TΔT⊤T\Delta T^\top. At rank 32 and merge rate 16, the full table shows no significant accuracy advantage on DeiT-Small or DeiT-Base (p=0.170p=0.170 and 0.1580.158). It stores 38,809 values per head rather than three, and its throughput is 16.5–22.7% lower across the tested configurations.

At rank 12, the additional positional information provides 1.32 and 0.64 accuracy points. The comparison establishes the compact correction’s accuracy–storage–runtime trade-off: three coefficients capture a useful position-free component at very low cost, while the full table offers additional accuracy under more severe truncation.

3. Matching cost and measured speedup.

The end-to-end measurements include matching and merging. Table 4 reports up to 1.06× throughput at matched accuracy and 2.67 additional accuracy points at matched throughput on DeiT-Small. The 1.06× speedup is an additional throughput gain over token merging alone at matched accuracy, rather than a comparison against the uncompressed model. These capture two practical benefits: processing more images at a chosen accuracy, or improving accuracy within a fixed throughput budget.

Section 3.5 quantifies the matching cost omitted from model FLOPs. At 197 tokens on DeiT-Small, block-input similarity costs 24.7% of attention-score computation, compared with 4.1% for head-averaged keys. We therefore report model-compute savings separately from measured runtime improvements. The new H100 measurements further show 1.04–1.08× throughput at matched accuracy (Table 28).

4. Other GPUs and batch sizes.

We extended the runtime evaluation to an H100 with DeiT-Small and DeiT-Base. All eight configurations improve accuracy at matched throughput (Appendix D.8, Table 28), extending the evidence beyond the original GPU and backbone.

Table 27 evaluates batches 1–256. Composition achieves higher throughput at batches 128 and 256, while merging alone is faster at batches 1–32. These measurements identify the larger-batch settings in which the compute savings translate into end-to-end throughput gains.

5. Calibration distribution.

We refit both the activation statistics and correction coefficients on ADE20K training images, then evaluated on the ImageNet test split (Appendix D.3, Table 24). At rank 32, the change relative to ImageNet calibration is −0.33 points without merging and −0.44 points at merge rate 16. With this different calibration distribution, composition still improves over token reduction alone by 0.62 points at matched model compute.

At rank 16, the calibration shift costs approximately 1.0–1.2 points. This experiment directly measures calibration sensitivity and shows that the composition benefit at rank 32 persists when calibration and evaluation images come from different datasets.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.