DISTILLATION REPAIRS WHAT POST-HOC K=V SURGERY BREAKS IN ATTENTION

Sharing key and value projections (K=V ) works when chosen before training: Kayyam et al. (2026) show it matches full QKV attention when trained jointly from scratch. We ask what happens when the same equality is instead imposed after training, by merging an already-trained model’s independent K/V projections with no retraining. The two settings are qualitatively different: a single attention sub-block’s output is provably confined to the convex hull of the shared K/V vectors regardless of attention weights, and this post-hoc surgery collapses every routing-dependent task toward chance—across five synthetic tasks, four vision datasets, a 300M-parameter language model, and an ImageNet-1k ViT-S/16 teacher—while tasks solvable without cross-token content routing are unaffected. The damage is cheap to repair: a short distillation pass (5–10 epochs) against the original model recovers nearly every configuration to within a point of the fromscratch ceiling, with one consistent exception: reusing the key projection alone (KEEP_K) recovers markedly slower and less completely than reusing the value projection or averaging the two. Mechanism tests (a content probe, a randomprojection baseline, an addressing/payload decomposition, and a learned K ! V basis correction) trace this to a representation-compatibility problem, not lost content: K and V are equally decodable, but a cheap, closed-form change of basis closes most of KEEP_K’s gap with no retraining. Finally, we confirm on the same ImageNet-1k family that choosing K=V before training lands within a point of this recovered ceiling directly, no repair needed — one picture: the constraint is easy to accommodate when the network is optimized around it from the start, and only becomes a repair problem when imposed after independent specialization.