acceptodds
Under review as a conference paper at ICLR 2027

The Price of a Merge Is Paid in Depth: A Tight, Label-Free Bound on Token Merging in Vision Transformers

Abstract

Token merging accelerates Vision Transformers by replacing pairs of similar tokens with their average, and no existing method states what a merge costs the layer that receives it. We give that cost. Because the next layer aggregates tokens in proportion to the attention they receive, the perturbation a merge induces downstream is bounded by an influence-weighted error that one forward pass computes with no labels; the salience-proportional average minimizes the bound, and uniform averaging inflates it by (si+sj)2/4sisj. We measure the bound on a pretrained ViT-B/16. It holds on every one of 52,190 merged pairs and is tight to 1.6%; uniform averaging realizes 2.27×the downstream perturbation of the proportional average; and the price is paid in depth, with a ratio of 1.00 in the shallow blocks, 3.1 in the deep ones, and a per-pair cost that rises by more than an order of magnitude from block 2 to block 11. Two further measurements say what the price buys. Halving the perturbation moves ImageNet accuracy by 0.06 pp, so the network absorbs it; forcing merges into the deep blocks costs five to eleven times the marginal rate of a schedule that avoids them. Accuracy is set by where and how much a method merges, not by how it averages. We turn the bound into a schedule, ADAMERGE, which spends a merge budget where the bound says merging is cheap and is calibrated on 256 unlabeled images: its margin over TOME grows with compression on four backbones, from +0.01 to +1.67 pp on ViT-B/16 and +0.53 to +4.64 pp on ViT-L/16, and it survives 30 epochs of fine-tuning. The bound applies to any merging method and costs one column sum of a matrix the method already forms.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
token merging, vision transformers, efficient inference, influence-weighted bound, training-free compression, adaptive computation, measurement study, interpretability, attention sinks

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_yC8JOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper looks at the downstream perturbation that token merging causes in Vision Transformers, proposes an influence-weighted bound on it, and designs ADAMERGE, which sets the number of merges per layer according to the current features. Direct perturbation measurements, leave-one-out ablations, and experiments on several backbones are used to analyze how the aggregation rule and the depth of merging shape final classification accuracy.

Critical strengths

The theoretical bound and the empirical measurements fit the method design. Section 3 derives an influence-weighted upper bound on the downstream perturbation caused by token merging. Section 5 measures this bound directly and separates the perturbation due to aggregation weights from that due to depth; the result reveals that accuracy is set by where a method merges and by how much it merges. Section 4 then uses these findings to build ADAMERGE, which sets the number of token merges in each layer from influence-weighted similarity.

Critical weaknesses
  1. The accuracy gains over fairly strong baselines are small, and their stability needs more evidence. At the six FLOPs tiers on ViT-B/16 in Table 1, ADAMERGE improves Top-1 ACC over PiToMe by 0.13, 0.18, 0.20, 0.06, 0.60, and 0.39 pp. The authors point out that the margin over TOME grows with compression, but the gains over PiToMe are overall smaller and do not follow the same trend. Some tiers also match FLOPs only approximately. Because ADAMERGE uses slightly more FLOPs at certain tiers, small accuracy advantages may be confounded by both the method design and differences in the compute budget. Table 1 additionally reports the mean and standard deviation of ADAMERGE over repeated runs but gives no variability for PiToMe, nor the variation of the accuracy difference between the two across runs. The present results therefore do not show whether the smaller gains reproduce reliably, and the advantage over fairly strong baselines still lacks evidence of stability.
  2. The ablations do not fully separate the effect of total merge count from that of the adaptive per-layer budget. In Table 13, full ADAMERGE beats TOME by 0.33 pp while averaging 12.8 fewer merges (47.2 vs. 60.0). The authors add a 70.7-merge configuration to compare against the 68.8-merge budget-only variant, but matching, aggregation, and budgeting are still confounded.
Questions for authors
  1. Are the accuracy gains over strong baselines large enough to matter? Across the six tiers in Table 1, ADAMERGE improves over PiToMe by just 0.06–0.60 pp. The margin over TOME does widen with compression, but the gains over PiToMe stay limited and show no such trend. How do the authors argue that the proposed aggregation strategy and adaptive budget constitute a clear and substantive accuracy advance?
  2. How much of the accuracy gain comes from the adaptive per-layer budget alone? Table 13 shows full ADAMERGE ahead of TOME by 0.33 pp, with 12.8 fewer merges on average (47.2 vs. 60.0). The extra comparison (70.7 vs. 68.8 merges) addresses this in part, yet matching and aggregation still differ. Could the authors hold those rules fixed, match the total merge count, and compare fixed against adaptive per-layer budgets directly? That would isolate what adaptive budgeting contributes on its own.
  3. What design guidance does the perturbation bound actually provide? When matching and budget are fixed and only the merge weights change, the perturbation is halved, yet Top-1 ACC moves by at most 0.06 pp. A smaller perturbation therefore does not translate directly into accuracy. Which specific choices in ADAMERGE does the bound then support, and what exactly links the bound to the adaptive per-layer budget?
1 comment
AuthorsOpenReview ↗

Response to Reviewer yC8J

Line numbers refer to the submitted PDF.

Both weaknesses are answered with the controls you specified. At matched compute there is no margin over PiToMe, and the adaptive budget contributes a depth profile that matters only at strong compression.

Q1. Are the gains over strong baselines large enough to matter?

Over PiToMe, no. With FLOPs matched exactly by interpolation and a paired bootstrap over 50,000 images, AdaMerge and PiToMe are within 0.03 pp at two tiers (CIs include 0) and PiToMe is 0.20 pp higher at the third (CI [−0.36, −0.04]); the submitted tiers were not compute-matched. Isolating the aggregation inside PiToMe gives the same reading: with official PiToMe matching and four merges per layer for both arms (15.205 G), PiToMe 84.930% vs with our aggregation 84.944% (p=0.80). Official ToMe is likewise 0.3–0.7 pp above AdaMerge at matched block FLOPs (GR-4).

The advance is analytical: a label-free per-merge cost, validated against the real block, and a controlled result on what accuracy responds to, now on three operators (GR-1, GR-2). The revised abstract no longer presents a margin over ToMe or PiToMe as a result.

Q2. How much of the gain comes from the adaptive per-layer budget alone?

At mild compression, nothing; at strong compression, +0.2 pp, and it comes from the depth profile, not from adaptivity. Matching and collision-safe aggregation fixed, 50k images:

ControlAdaptiveDepth-fixedFlatAdaptive − Flat
Matched compute, rmax⁡=23r_{\max}=23 (14.24 G)83.92883.94283.733+0.195 pp, 95% CI [+0.055, +0.333]
Matched compute, rmax⁡=14r_{\max}=14 (15.37 G)84.43084.40284.416+0.014, CI [−0.10, +0.12]
Matched total merges, rmax⁡=14r_{\max}=1484.42884.38884.376+0.052, n.s.

The fixed profile read off the calibration matches the adaptive gate in both regimes. Table 13's confound (47.2 vs 60.0 merges) is removed and the table is replaced by this one.

To test depth allocation beyond our operator, we kept official ToMe unmodified and changed only its schedule:

Block FLOPsConstantDecreasingAdaMerge profileKL-calibrated
~14.09 G84.85284.72084.79484.942
~10.79 G83.99083.70283.62084.062
~8.66 G82.35081.49881.17682.222

The schedule alone moves accuracy by up to 1.17 pp. For ToMe the constant schedule is near-optimal: a label-free KL calibration matches it (n.s.), and our front-loaded profile is worse (−0.37, −1.17 pp). Depth choice is consequential in both operators; the profile that helps is operator-specific.

Q3. What design guidance does the bound provide?

It tells which average minimizes the local perturbation, and the measurements tell that this is not the lever. The proportional average lowers real [CLS] / logit perturbation by 6.9% / 3.5% (2,288 interventions; within-layer Spearman 0.71 / 0.58 between surrogate and real perturbation), yet accuracy moves by ≤0.02 pp in our operator and +0.014 pp in PiToMe. The guidance that survives is where to look: the cost is concentrated by depth, and depth allocation is what accuracy responds to. The link from Eq. 9 to that allocation is a calibrated rule, not a theorem, and §4.4 says so.

Presentation. §3–4 label each statement proved, measured or chosen; the real-block table and the two tables above are added.

If a row is missing from these tables that you would want in order to judge the budget's contribution, please name it and we will add it before the discussion closes.

Reviewer_5TAAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks at token merging in ViTs and tries to measure how much a single merge changes the next layer. The authors bound this change by an error term where each token is weighted by its attention column sum, which they call its influence. From this, the best way to merge two tokens is to average them in proportion to their influence, and a plain average is worse by a factor that depends on how different the two influences are. On ViT-B they show the bound is close to tight and that the plain average causes a bigger change, mostly in the deep layers. However, the choice of average barely affects accuracy, while merging in the deep layers hurts a lot more. Based on this they propose AdaMerge, which uses influence-weighted matching and averaging and decides how many tokens to merge at each layer using a score calibrated on a small set of unlabeled images. It beats ToMe on four backbones at matched FLOPs, and the gap gets bigger at higher compression.

Critical strengths

The analysis is simple and easy to follow, and the authors actually test it instead of just stating it. The finding that the averaging rule hardly matters for accuracy, but where in the network you merge matters a lot, is useful for people working on token reduction.

The method needs no training and only a small unlabeled calibration set, and it does better than ToMe at higher compression on several backbones.

Critical weaknesses

My main concern is that the theory does not really explain the accuracy results. The bound is computed on softmax(XX^T) of the block input, which is not the attention the block actually uses, and it keeps this matrix fixed and ignores the residual, LayerNorm and MLP. So it is not surprising that it holds and is tight, and I am not sure it tells us much about the real network. Also, the main thing the theory recommends, the influence-weighted average, does not change accuracy. In the ablation, neither the weighted matching nor the adaptive budget beats ToMe on its own, and the gain only shows up when both are used. The budget rule is mostly a heuristic, and since each layer is only compared to its own calibration statistics, it does not really compare merge costs between layers. At small budgets it even merges the most in the last block, which the paper says is the most expensive one. Because the paper presents the bound as what drives the method, this gap is the main reason for my score.

I am also not convinced the method is practical yet. All ViT-B results use mild compression, with speedups below about 1.25x, which is much less than how ToMe is usually used. When comparing at the same measured speed instead of FLOPs, ToMe is still better at the low end, and AdaMerge is slower than ToMe on the smaller models. PiToMe is also more accurate on ViT-L. The number of merges is computed from an average over the evaluation batch, so the output for one image depends on the other images in the batch, which is a problem in practice. Finally, some numbers don't match between tables (for example, the same default setting reported with different accuracy, and the same run with different FLOPs). Some of these differences are as big as the effects the paper's main claims are based on.

I also have doubts about the fine-tuning results. The paper says the gain over ToMe holds after 30 epochs of fine-tuning, but after fine-tuning both methods end up below their own training-free accuracy (for AdaMerge, around 83.9% compared to about 84.5% without training). Fine-tuning should normally help, so this makes me think something is off in the setup, and I would not count this experiment as support for the method until it is explained.

Questions for authors
  1. What is the accuracy at batch size 1, and do predictions change when an image is put in a different batch?
  2. How are merges actually spread over the layers? At layer 0 nothing has been merged yet, so I expected about rmax/2 merges there, but the reported number is lower. How stable is this across calibration seeds?
  3. When several tokens match the same target, the code keeps only the last one, so the others are dropped instead of merged. How often does this happen, and does fixing it change the results? Also, was ToMe run with its size-weighted averaging and proportional attention? The paper describes ToMe as a plain average.
  4. Do the reported FLOPs include the extra N x N affinity that AdaMerge computes?

Not required, but it would change how I see the significance: results at stronger compression, closer to the usual ToMe settings, and a comparison with Zero-TPrune.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 5TAA

Line numbers refer to the submitted PDF.

We agree with your reading, and the new controls confirm it: the averaging rule is not where accuracy comes from, the budget rule is a calibrated heuristic, and the per-batch gate is a deployment problem, for which we now give a measured remedy.

W1. Theory vs. accuracy; neither half alone; the budget rule

Agreed. The bound holds on a frozen aggregate, so we tested it on the real block: it ranks real [CLS] and logit perturbations in every active layer (Spearman 0.71 / 0.58), and the proportional average lowers them by 6.9% / 3.5%. That does not reach accuracy: ≤0.02 pp in our framework and +0.014 pp (p=0.80) with official PiToMe matching. Table 13 mixed merge counts (47.2 vs 60.0) and is replaced by a matched-compute control with matching and aggregation fixed: at rmax⁡=14r_{\max}=14 adaptive, depth-fixed and flat schedules coincide (84.430 / 84.402 / 84.416), last-block allocation included; at rmax⁡=23r_{\max}=23 the calibrated profile is +0.195 pp over flat (95% CI [+0.055, +0.333]). Eq. 9 is now labelled a scheduling rule; it compares each layer with its own statistics, as you say.

W2. Practicality and table numbers

Throughput against the in-framework baseline on ViT-S/B and PiToMe's lead on ViT-L stand as reported. Recomputed FLOPs put AdaMerge up to 0.9 G above its nominal tier, so Table 1's aggressive tiers are not compute-matched (GR-5). Single-run values at rmax⁡=17r_{\max}=17 vary with the calibration draw (84.14–84.32% over nine recomputed runs); Table 17 is a separate run. Table 3 merge counts, σ5\sigma_5 and the Table 20 caption are corrected.

W3. Fine-tuning

The decline is not specific to our merge. From the same initialization, one epoch (LR 5×10⁻⁷, no Mixup/CutMix), official PiToMe matching at 15.205 G: PiToMe 84.930 → 84.398% (−0.53 pp); with our aggregation 84.944 → 84.514% (−0.43 pp; difference +0.116 pp, p=0.060). A stronger recipe lowers PiToMe to 83.06%. No unmerged control was run; the fine-tuning claim is removed from the abstract.

Q1. Batch size 1; do predictions change across batches?

Yes to both for the per-batch gate: 83.628% at true batch size 1, and replacing the 63 companions changes 21/512 predictions (unmerged ViT: 0/512). The remedy is measured: a fixed per-layer profile read off the calibration is batch-independent by construction and matches the adaptive gate at matched compute (83.942 vs 83.928 at rmax⁡=23r_{\max}=23; 84.402 vs 84.430 at 14). It is now the recommended mode.

Q2. Layer-wise merges and calibration seeds

Three independent calibrations (12,811 training images each, same 2,048 validation images): Top-1 84.49 ± 0.16%, total merges 51.3 ± 0.4, layer-0 merges 5.79 ± 0.15. Layer 0 is below rmax⁡/2r_{\max}/2 because its standardized statistic is negative on validation batches (zˉ0≈−0.18\bar z_0\approx-0.18).

Q3. Shared targets; ToMe settings

19.35% of source writes shared a target; they are now accumulated (+0.022 pp at identical budgets). ToMe: no. The submission's ToMe rows are a uniform-schedule baseline inside our framework, without size weighting or proportional attention. Official ToMe (commit af95e4b) is the stronger operator: 85.04–84.69% at 15.76–12.99 G, 0.3–0.7 pp above AdaMerge at matched block FLOPs. L785–786 is updated; the revised Table 1 labels the baseline and a new table reports official ToMe.

Q4. FLOPs

No: they are analytical block counts excluding the N×NN\times N affinity (about 2% of a block). On an H200 (batch 64, FP16) AdaMerge runs at 4,013 img/s against 4,906 unmerged; we claim no wall-clock gain on ViT-B there.

Q5. Stronger compression, Zero-TPrune

Not run for AdaMerge or Zero-TPrune. On official ToMe at 10.79 and 8.66 G, the schedule alone moves accuracy by up to 1.17 pp and our depth profile is worse than constant (GR-2), so we make no claim at that regime.

You named the gap between what the bound is presented as driving and what it explains as the main reason for your score. The paper now presents the bound as a validated diagnostic and the depth finding as operator-specific, not as the source of a margin over ToMe. Does that close the gap?

Reviewer_5kunOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper derives an influence-weighted perturbation bound for token replacement under a fixed affinity matrix and proposes AdaMerge, which uses influence scores to determine token matching, aggregation, and layerwise merge budgets. Experiments report improved accuracy over ToMe across four vision backbones at similar FLOPs, with additional ablations and multimodal evaluations.

Critical strengths
  • The classification results demonstrate meaningful gains over ToMe across multiple backbones and compression levels, including a 1.67 percentage-point improvement on ViT-B at approximately 13.4 GFLOPs. Reporting measured throughput and comparisons with PiToMe makes the practical tradeoffs assessable.
  • The controlled aggregation experiment provides an informative distinction: substantially reducing the surrogate perturbation barely changes classification accuracy. This helps identify the limits of aggregation-error minimization as an explanation for performance.
Critical weaknesses
  • The bound does not establish the claimed cost of actual token merging. It holds the affinity matrix fixed and replaces two tokens without reducing sequence length. AdaMerge instead removes tokens and recomputes learned attention. The tightness measurements evaluate the same frozen surrogate, so they do not validate its approximation to actual block-output changes. This leaves the paper’s central explanatory claim insufficiently supported.
  • The normalization breaks the stated theoretical connection. Proposition 1 uses raw column-sum influences, whereas Eqs. 6–7 use min–max-normalized influences. This transformation generally changes the optimal averaging weights. Moreover, Eq. 6’s proposed upper bound becomes zero for a minimum-influence source even when Eq. 4’s cost is positive. The empirical similarity between normalized and raw variants does not establish the claimed identities.
  • The documented implementation discards contributions when targets collide. Appendix K and Algorithm 4 explicitly specify that the last write wins when multiple sources share a target, while all selected sources are removed. Consequently, some selected tokens are discarded rather than aggregated, and the pairwise analysis does not describe the resulting operation.
Questions for authors
  • How accurately does the surrogate predict perturbations in actual Transformer-block outputs after token removal and attention recomputation?
  • Can you provide corrected guarantees connecting the normalized scores and aggregation rule to the original bound?
  • How frequently do shared-target collisions occur, and how do accuracy and throughput change when all contributing sources are accumulated correctly?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 5kun

Line numbers refer to the submitted PDF.

Your three weaknesses are correct. We measured the quantity Q1 asks for, restricted the guarantee to raw influence, and replaced the overwrite by a group merge the lemma covers.

Q1. How accurately does the surrogate predict perturbations in actual block outputs?

It ranks them in every active layer and predicts their size only partially. 2,288 image–layer interventions on 256 images; matching, groups and budgets fixed, only the aggregation weights changed; real block with learned attention recomputed on the shortened sequence, residual, LayerNorm, MLP, then the remaining blocks to the logits.

Mean relative perturbationUniformRaw influenceChange
Frozen-affinity surrogate0.09160.0836−8.8%
Real block, [CLS] output0.007760.00722−6.9%
Final logits0.07180.0694−3.5%
Real block, surviving tokens0.07730.0829+7.3%

Within-layer Spearman correlation between surrogate and real perturbation is positive in all nine active layers: median 0.71 against [CLS], 0.58 against the logits. Two limits are stated in the revised §5: surviving-token perturbation moves the other way, and the influence ss is the mass under softmax(XX⊤)\mathrm{softmax}(XX^\top), whose token ranking agrees only weakly with the learned attention's (0.07). The bound is exact on the aggregate and ranks, without bounding, the real block's response; "tight to 1.6%" now reads "on the aggregate".

Q2. Corrected guarantees for the normalized scores?

None is needed for aggregation: the rule is stated for raw ss, where it holds as written. ∑isi=N\sum_i s_i=N, so raw influence already averages one per layer; subtracting the minimum changes the ratios and can assign zero weight. Aggregating with ss or s/max⁡ss/\max s instead of s^\hat s changes Top-1 by ≤0.02 pp at identical budgets. For matching, s^\hat s is labelled a design choice: raw matching + raw aggregation at exactly replayed budgets scores 84.290% vs 84.430% (p=0.019). We no longer cite the closeness of the two variants as support for the identity.

Q3. Collision frequency, and accuracy/throughput when sources are accumulated?

19.35% of selected source writes (8,866/45,824; rmax⁡=23r_{\max}=23, 1,024 images) landed on a target already written, in 40.6% of image–layer cases. Sources are now accumulated into the group centroid,

\tilde{x}^{\ast}=\frac{\sum_{m\in G\cup\\{t\\}}s_mx_m}{\sum_{m\in G\cup\\{t\\}}s_m}$$ for which the bound holds (Lemma 1′), so the analysis describes the operation that is run. At identical per-layer budgets on 50k images: 84.160 → 84.182% (+0.022 pp). Throughput under the native gate is 9.7% lower with the accumulation kernel (4,213 → 3,806 img/s, H200). ### What this means for the central claim The explanatory claim is restated to what the measurements support: the averaging rule the bound identifies lowers real perturbation but does not move accuracy, on our operator and with PiToMe's matching (+0.014 pp, p=0.80); what moves accuracy is depth allocation, by +0.195 pp in our operator at strong compression and by up to 1.17 pp in official ToMe, with an operator-specific direction (GR-1, GR-2). The measurement you asked for now exists and both implementation points are closed; we would be grateful if you could revisit the soundness assessment in that light.

Thank you for the substantive experiments and corrections. These substantially address my soundness concerns, and I will raise that score. However, the narrower theoretical guarantee and the throughput reduction after correcting aggregation leave the overall contribution borderline. I therefore maintain my overall rating.

AuthorsOpenReview ↗

Response to Reviewer 5kun

Thank you for re-reading the paper and for raising the soundness score. On the two remaining points:

Theoretical guarantee. The proved content did not shrink. Lemma 1 and Proposition 1 were stated for raw ss in the submission and are unchanged. The revision removes one unproved extension (Eq. 6 for the rescaled s^\hat s) and adds a proved one, from pairs to groups of any size (Lemma 1′). The guarantee now covers exactly the operation that is run.

Throughput. The 9.7% is a configuration-level figure: under the native gate the two arms also select different budgets, and the accumulation is a first, unoptimized kernel. Replaying the same per-layer schedule without the gate, safe accumulation runs at 4,138 img/s against 3,782 with the gate. We make no speed claim for the corrected method; the paper’s claims do not rest on throughput.

Contribution. What we ask to be judged is the result your review singled out as informative: reducing the surrogate perturbation barely changes accuracy. It now holds on three operators, with the surrogate validated on the real block, together with its complement, that depth allocation alone moves accuracy by up to 1.17 pp.

Is there a specific addition that would move the contribution above the borderline for you?

Reviewer_1LYXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper derives an influence-weighted bound on token-merge error for a fixed affinity matrix and uses it to motivate AdaMerge's aggregation and layer-wise schedule. Experiments examine how merge location and averaging affect accuracy, with comparisons across four vision transformer backbones.

Critical strengths
  1. Separating averaging error from budget allocation is useful. The aggregation experiment shows that substantially reducing the fixed-affinity perturbation has little effect on accuracy, while the deep-layer intervention suggests that merge location matters more in the tested setting.
  2. The evaluation includes calibration, fine-tuning, and measured throughput across several backbones. I appreciate that the paper reports where AdaMerge is slower than ToMe or less accurate than PiToMe, rather than relying only on favorable FLOPs comparisons.
Critical weaknesses
  1. The implemented weights do not preserve the proved objective. Lemma 1 uses raw affinity column sums, whereas Eqs. (6)–(7) subtract the minimum before rescaling. Preserving weight order does not preserve the optimal weighted average. A minimum-weight token has ŝ_i=0, so Eq. (6) gives U_ij=0 even when the original J*_ij is positive. The claimed bound therefore does not transfer to this score.
  2. The stated merge operation is not a consistent aggregation of shared-target groups. Algorithm 4 allows multiple sources to choose one destination and specifies last-write-wins. As described, earlier source contributions can be overwritten while those sources are removed. This is not covered by the single-pair proof, and it can make the result depend on write order rather than the combined group weights.
  3. The connection between the local bound and the layer schedule remains heuristic. The bound holds the affinity fixed, while Eq. (9) applies a sigmoid to a layer-standardized mean matching score. This neither directly controls the sum of raw merge costs nor accounts for changes in attention and subsequent blocks. The paper acknowledges the first-order approximation, but should distinguish the proved local result from the chosen scheduling rule.
Questions for authors
  1. How can Eqs. (6)–(7) retain the raw-weight guarantees after subtracting the minimum influence?
  2. Are shared targets updated by vectorized overwrite or weighted accumulation, and does the experimental code match Algorithm 4?
  3. Which property of Eq. (9) follows from the bound, and what connects its standardized statistic to actual downstream error?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 1LYX

Line numbers refer to the submitted PDF.

All three points are correct. Both implementation gaps are closed with a proof and no change in results, and the paper now states which parts are proved and which are chosen.

Q1. How can Eqs. 6–7 retain the raw-weight guarantees after subtracting the minimum?

They cannot; the guarantee is stated for raw ss. With row-stochastic A^\hat A, si>0s_i>0 and ∑isi=N\sum_i s_i=N, so raw influence already has mean one at every layer. For unit-normalized features,

J_{ij}^{\ast} = \frac{s_i s_j}{s_i+s_j}\lVert x_i-x_j\rVert^2 \leq 2s_i(1-\cos(x_i,x_j))$$ A positive rescaling preserves this; subtracting the minimum does not, and your counterexample ($\hat s_i=0\Rightarrow U_{ij}=0$) is exact. In aggregation the choice is immaterial: $s$ or $s/\max s$ in place of $\hat s$ changes Top-1 by ≤0.02 pp at identical budgets. In matching, $\hat s$ is a tempered weight and is now labelled a design choice: with every layer's merge count replayed exactly, raw matching + raw aggregation scores 84.290% vs 84.430% (paired p=0.019). ### Q2. Overwrite or weighted accumulation? Does the code match Algorithm 4? **Overwrite, as Algorithm 4 printed: 8,866 of 45,824 selected source writes (19.35%) landed on a target already written ($r_{\max}=23$, 1,024 images).** It is now weighted accumulation, covered by the group version of the lemma. For sources $G$ sharing target $t$, with $\hat A$ fixed, $$\lVert \Delta Y\rVert_F^2 \leq J_G(\tilde{x}) = \sum_{m\in G\cup\\{t\\}} s_m \lVert \tilde{x}-x_m\rVert^2, \qquad \tilde{x}^{\ast} = \frac{\sum_{m\in G\cup\\{t\\}} s_m x_m}{\sum_{m\in G\cup\\{t\\}} s_m}$$ by the weighted-mean inequality applied row-wise with $\sum_m\hat A_{km}\le1$ (Lemma 1′, Appendix A). The result is independent of write order. At identical per-layer budgets on 50k images: 84.160 → 84.182% (+0.022 pp), so the submitted results stand. ### Q3. Which property of Eq. 9 follows from the bound, and what connects it to downstream error? **Only Eq. 8 follows; Eq. 9 is a calibrated scheduling rule, and §4.4 now says so.** It standardizes each layer against its own statistics and we do not claim it controls the sum of merge costs. What connects it to downstream error is measured: - **Real block.** With attention recomputed and all subsequent blocks included, the surrogate ranks real perturbations in every active layer (median Spearman 0.71 [CLS], 0.58 logits). - **Matched compute, matching and aggregation fixed.** At $r_{\max}=23$ the calibrated depth profile is +0.195 pp over a flat schedule (95% CI [+0.055, +0.333]); at $r_{\max}=14$ the schedules coincide (+0.028 / +0.014 pp, n.s.). The adaptive gate adds nothing beyond the fixed profile it produces. - **Another operator.** In official ToMe the schedule alone moves accuracy by up to 1.17 pp, but the constant schedule is near-optimal and our profile does not transfer (GR-2). The rule is specific to our operator. Every statement in §3–4 is now either proved for the quantity the code uses or labelled as chosen. Do the revised Appendix A and §4 resolve the soundness points as stated?

Thank you for the careful response. Using raw weights for aggregation, fixing shared-target accumulation, and clarifying that the depth schedule is empirical address my soundness concerns. The matched-compute experiments also strengthen the case that where tokens are merged matters. I’ve raised my overall rating to 3

AuthorsOpenReview ↗

Response to Reviewer 1LYX

Thank you so much for your thoughtful review and for taking the time to revisit our work! We are very happy to hear that our response addressed your concerns. Your feedback genuinely helped us improve the paper. Thank you again!

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to all reviewers

Line numbers refer to the submitted PDF. New results: ViT-B/16, 50,000 ImageNet validation images.

We thank the reviewers. The new experiments test the paper's central finding (L350–351): accuracy is set by where and how much a method merges, not by how it averages. It now holds on three operators, and the experiments delimit what the bound and AdaMerge's schedule contribute.

1. How to average does not move accuracy — on three operators. In our framework, replacing the rescaled weight by raw ss or s/max⁡ss/\max s in aggregation changes Top-1 by ≤0.02 pp at identical budgets. With official PiToMe matching, our aggregation vs PiToMe's: 84.944 vs 84.930% at 15.205 G (p=0.80). The bound still does what it states: in 2,288 image–layer interventions with the real block, the proportional average lowers surrogate / [CLS] / logit perturbation by 8.8 / 6.9 / 3.5%, and the surrogate ranks real perturbations within every active layer (median Spearman 0.71 [CLS], 0.58 logits). It ranks the real block; it does not bound it.

2. Where to merge does — and the best profile is operator-specific. In our framework, with matching and aggregation fixed and compute matched, the calibrated depth profile is +0.195 pp over a flat schedule at rmax⁡=23r_{\max}=23 (95% CI [+0.055, +0.333]) and equal to it at rmax⁡=14r_{\max}=14; a fixed profile read off the calibration matches the adaptive gate (83.942 vs 83.928). In official ToMe (unmodified; size-weighted merging, proportional attention), changing only the schedule at matched block FLOPs:

Block FLOPsConstantDecreasingAdaMerge profileKL-calibrated (label-free)
~14.09 G84.85284.72084.79484.942
~10.79 G83.99083.70283.62084.062
~8.66 G82.35081.49881.17682.222

Schedule choice alone moves accuracy by up to 1.17 pp. For ToMe the constant schedule is near-optimal: label-free calibration matches it (differences n.s.) and AdaMerge's front-loaded profile is worse (−0.37, −1.17 pp, p<10⁻⁵). Depth allocation is consequential and its direction depends on the operator; Eq. 9 is a calibrated rule for our operator, not a consequence of the bound (1LYX Q3).

3. Theory = implementation (1LYX, 5kun, 5TAA Q3). The bound and aggregation rule are stated for raw ss; the min–max map does not preserve the minimizer and is a design choice in matching (raw matching: −0.14 pp). Shared targets are now accumulated, and the bound extends to groups (Lemma 1′, centroid minimizer). 19.35% of source writes had collided; fixing it changes Top-1 by +0.022 pp at identical budgets (84.160 → 84.182).

4. Scope of the comparisons (5TAA Q1/Q3/W3, yC8J Q1). The ToMe rows of the submission are a uniform-schedule baseline inside our framework (no size weighting or proportional attention). Official ToMe is the stronger operator: 0.3–0.7 pp above AdaMerge at matched block FLOPs (e.g. 84.85% at 14.09 G vs 84.25% at 14.77 G); L785–786 is updated and both are reported. At matched compute we observe no margin over PiToMe (within 0.03 pp at two tiers, PiToMe +0.20 at one). The per-batch gate is batch-dependent (83.63% at batch size 1; 21/512 predictions change with batch companions); the fixed profile above is batch-independent at the same accuracy and is the recommended mode. Under fine-tuning, accuracy also declines for PiToMe alone (−0.53 pp vs −0.43 with our aggregation, one epoch), so the decline is not specific to our merge; the fine-tuning claim is removed from the abstract.

5. Corrections (5TAA W2/Q4). FLOPs are analytical block counts excluding the affinity. Recomputed over the full validation set, AdaMerge is at 15.97–14.33 G for rmax⁡=9r_{\max}=9–2323, up to 0.9 G above its nominal tier, so Table 1's aggressive tiers are not compute-matched; the matched comparisons are those above. Table 3 merge counts, σ5\sigma_5 in Table 18 and the caption of Table 20 are corrected. Zero-TPrune and stronger AdaMerge compression were not run.

What the paper claims after the discussion: a label-free bound on what a merge costs the next layer, validated against the real block; and the finding, now on three operators, that the averaging rule is immaterial while depth allocation is consequential and operator-specific. AdaMerge is the instrument for these measurements, not a state-of-the-art merger.

AuthorsOpenReview ↗

Summary of Revisions

Changes are confined to the points raised in review (marked in blue).

WhereChange
§4.1–4.3, Alg. 4, App. ABound and aggregation for raw ss; group accumulation; Lemma 1′
§4.4Eq. 9 labelled a scheduling rule; depth-fixed mode; batch dependence
§5, App. ESurrogate vs real block (new table)
§6, Table 1, L785–786In-framework baseline labelled; official ToMe added; recomputed FLOPs
§7, App. ESchedules at matched compute (replaces Table 13); schedules on official ToMe; aggregation inside PiToMe
Tables 3, 18, 20; §4.5Merge counts, σ5\sigma_5, FLOPs caption, FLOPs accounting
Abstract, §1, §8Claims restated as bound / measurement / depth; fine-tuning sentence removed; limits stated

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.