The Price of a Merge Is Paid in Depth: A Tight, Label-Free Bound on Token Merging in Vision Transformers

Token merging accelerates Vision Transformers by replacing pairs of similar tokens with their average, and no existing method states what a merge costs the layer that receives it. We give that cost. Because the next layer aggregates tokens in proportion to the attention they receive, the perturbation a merge induces downstream is bounded by an influence-weighted error that one forward pass computes with no labels; the salience-proportional average minimizes the bound, and uniform averaging inflates it by (si+sj)2/4sisj. We measure the bound on a pretrained ViT-B/16. It holds on every one of 52,190 merged pairs and is tight to 1.6%; uniform averaging realizes 2.27×the downstream perturbation of the proportional average; and the price is paid in depth, with a ratio of 1.00 in the shallow blocks, 3.1 in the deep ones, and a per-pair cost that rises by more than an order of magnitude from block 2 to block 11. Two further measurements say what the price buys. Halving the perturbation moves ImageNet accuracy by 0.06 pp, so the network absorbs it; forcing merges into the deep blocks costs five to eleven times the marginal rate of a schedule that avoids them. Accuracy is set by where and how much a method merges, not by how it averages. We turn the bound into a schedule, ADAMERGE, which spends a merge budget where the bound says merging is cheap and is calibrated on 256 unlabeled images: its margin over TOME grows with compression on four backbones, from +0.01 to +1.67 pp on ViT-B/16 and +0.53 to +4.64 pp on ViT-L/16, and it survives 30 epochs of fine-tuning. The bound applies to any merging method and costs one column sum of a matrix the method already forms.