What Survives Token Merging? Merge-Invariant Low-Rank Attention for Vision Transformers

Token reduction and low-rank factorization offer complementary ways to accelerate vision transformers, yet their composition introduces dependencies that neither addresses independently. Factorizing the joint query–key product changes the representations used for token merging, while merging destroys the stable positional correspondence required by conventional logit-residual correction. We resolve both incompatibilities without retraining. First, we characterize fixed logit corrections invariant to patch relabelling: their space is five-dimensional, but two directions are invisible to the softmax, leaving exactly three effective degrees of freedom. These three coefficients per head recover $93.9\%$ and $103.7\%$ of the accuracy recovered by a $38,809$-entry position-indexed correction on DeiT-Small and DeiT-Tiny, respectively, while remaining applicable after merging. Second, we decouple token matching from the current query–key factorization by computing similarity from normalized block inputs, improving matching stability under rank reduction. We further identify an accuracy–compute crossover: mild token reduction favors merging alone, whereas more aggressive reduction benefits from combining both compression axes. Beyond this crossover, our method improves accuracy by up to $4.34$ percentage points at matched compute and achieves up to $1.06\times$ throughput at matched accuracy, with no retraining or gradient computation. Experiments across DeiT scales, ViT-L, and an alternative token-reduction rule establish that the composition holds beyond a single setting, and results on ADE20K segmentation and five CLIP image encoders examine the broader applicability of low-rank correction. Code and compressed checkpoints will be released to support reproduction and further work.