acceptodds
Under review as a conference paper at ICLR 2027

DISTILLATION REPAIRS WHAT POST-HOC K=V SURGERY BREAKS IN ATTENTION

Abstract

Sharing key and value projections (K=V ) works when chosen before training: Kayyam et al. (2026) show it matches full QKV attention when trained jointly from scratch. We ask what happens when the same equality is instead imposed after training, by merging an already-trained model’s independent K/V projections with no retraining. The two settings are qualitatively different: a single attention sub-block’s output is provably confined to the convex hull of the shared K/V vectors regardless of attention weights, and this post-hoc surgery collapses every routing-dependent task toward chance—across five synthetic tasks, four vision datasets, a 300M-parameter language model, and an ImageNet-1k ViT-S/16 teacher—while tasks solvable without cross-token content routing are unaffected. The damage is cheap to repair: a short distillation pass (5–10 epochs) against the original model recovers nearly every configuration to within a point of the fromscratch ceiling, with one consistent exception: reusing the key projection alone (KEEP_K) recovers markedly slower and less completely than reusing the value projection or averaging the two. Mechanism tests (a content probe, a randomprojection baseline, an addressing/payload decomposition, and a learned K ! V basis correction) trace this to a representation-compatibility problem, not lost content: K and V are equally decodable, but a cheap, closed-form change of basis closes most of KEEP_K’s gap with no retraining. Finally, we confirm on the same ImageNet-1k family that choosing K=V before training lands within a point of this recovered ceiling directly, no repair needed — one picture: the constraint is easy to accommodate when the network is optimized around it from the start, and only becomes a repair problem when imposed after independent specialization.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
key-value sharing, K=V, attention, projection tying, post-hoc compression, distillation, transformer compression, representation compatibility, model surgery, attention routing, parameter sharing, ViT, language models, efficient transformers.

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_phE8OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper asks the question of what happens when independently trained key and value projections are replaced by a shared projection after training. It reports substantial performance degradation followed by recovery through supervised training combined with distillation, with recovery depending on how the shared projection is initialized. Probing and alignment experiments suggest that incompatibility between representations and downstream weights contributes to the damage, while experiments with models trained from scratch show that the sharing constraint can be accommodated during training.

Critical strengths
  • The research direction is important and interesting.
Critical weaknesses
  • The paper is heavily AI-written and bombarded with a lot of numbers in the text, making it really hard to follow a story line through the paper.
  • The central benefit and efficiency of distillation are not established against ordinary fine-tuning. In more detail, main repair procedure uses the original training data, hard labels, and updates throughout the student, in addition to the teacher’s soft targets. However, there is no matched hard-label-only fine-tuning baseline using the same initialization, data, optimizer, and training budget. Consequently, the experiments cannot determine whether distillation is important or whether ordinary adaptation after a disruptive architectural edit would recover similarly.
  • Proposition 1 is a correct convex-combination observation, but standard attention is likewise confined to the convex hull of its value vectors. As a result, the proposition alone does not establish a performance obstruction or explain the difference between post-hoc and joint training.
Questions for authors
  • How does hard-label-only fine-tuning compare with CE-plus-KL distillation under matched initialization, data, optimizer, and compute?
  • What are the measured repair FLOPs or GPU-hours, including teacher inference, and the resulting inference-time memory and throughput benefits? What exactly does the “Cache” column in Table 6 measure?
2 comments

Thank you for the review. We answer the two questions first. (Table and section numbers follow the submission. In the revision, the new matched-budget table is Table 7 (main text), the new zero-shot control tables are Tables 9 and 10 (Appendix B), and the ImageNet-1k sweep table is Table 8.)

Q1: hard-label-only fine-tuning vs CE+KL. You are right that the original submission could not separate them. We ran the matched control at CIFAR scale: same teacher checkpoint, initialization, data, optimizer (Adam, lr 3e-4, cosine, 10% warmup), batch size and 10 epochs, all parameters trained, 3 fine-tuning seeds; only the loss differs. Top-1 %, CE only / KL only / CE+KL: CIFAR-10 KEEP_K 64.8/66.5/66.5, KEEP_V 69.5/68.8/69.4, AVG 70.1/69.1/69.9; CIFAR-100 KEEP_K 40.0/41.2/41.2, KEEP_V 42.9/42.7/43.2, AVG 43.1/43.2/43.5 (sd ≤0.3). So for KEEP_V and AVG, hard-label fine-tuning alone is within ~0.4 pt of CE+KL: that repair is largely ordinary adaptation after the edit, not distillation. For KEEP_K the teacher signal adds 1.1–1.7 pt. Continuing the unmodified QKV teacher with CE for the same budget gives 70.7 on CIFAR-10 (+1.1 over its own 69.6), so repaired students "exceeding the teacher" reflect the training budget. The Abstract and Conclusion now state this. We did not run the control at ImageNet-1k or LM scale (one LM repair run is ≈4 h on two A30s per merge mode, and the ImageNet data is no longer accessible to us); the paper says the teacher signal's contribution there is untested.

Q2: repair cost, inference benefit, and the "Cache" column. Cost (from logs): LM (305.5M-parameter teacher; 284.5M after K=V): 500M tokens ≈ 4 h 19 min per merge mode on 2×A30, vs 58.8 h for the 10B-token pretraining (same two A30s, per our launch scripts; ≈7% of wall-clock, ≈6% of FLOPs counting the teacher's forward pass as 2ND). ImageNet-1k: 5 epochs ≈ 4.5–5 h per mode on one RTX 2080 Ti (from file timestamps) vs ≈92 GPU-hours for a from-scratch run. Inference benefit: not measured; our LM code has no cached-decoding path, and Limitations now says so. Cache column: it was not measured. It was an arithmetic count and internally inconsistent (33% for Q=K=V but 50% for K=V, though both store one tensor at decode time since queries are not cached). It is now labelled "KV cache" and reads 50% for every shared variant, marked as arithmetic.

Proposition 1. Agreed. Standard attention is also confined to the convex hull of its value vectors, so the bound alone shows no obstruction and cannot explain post-hoc vs joint training. We added this to the paper, retitled §3 as a scoping observation about which tasks can be affected, and removed the causal wording from the Abstract and Conclusion.

Presentation. We respectfully disagree that the paper lacks a storyline, and would welcome specifics. It is built around one contrast (Table 1: imposing K=V after training vs choosing it before), with one sub-question per section in order: which tasks can be affected (§3), how bad the collapse is and how much training repairs it (§4), why (§4.1), and whether choosing the constraint up front avoids the problem (§5). Numbers sit in tables, and the prose around each states its takeaway. Both other reviewers rated presentation 3 ("clear and easy to follow"). If particular passages were hard to follow, please point us to them and we will rewrite them. On AI assistance: the paper's AI-use statement discloses it (drafting and editing, merging two source documents, writing analysis scripts, formatting) and states what we did not delegate (experimental design, hypotheses, Proposition 1 and its proof). The authors checked numbers against both source documents and sanity-checked the scripts against known baselines, and take responsibility for the content. We do not think the use of an editing tool bears on whether the experiments and claims are sound, and we are glad to address any specific technical point.

Thank you for the, again, very very detailed and numbered response. While these address some of my concerns and help clarify some of the findings, they do not change my overall assessment of the paper’s contribution and its understandability. I will therefore maintain my original score.

Reviewer_hRcDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper compares sharing key and value projections from the beginning of training with imposing the same constraint on a model trained with separate projections. Post-hoc merging causes large immediate degradation on routing-dependent tasks, while distillation often recovers much of the lost performance. Synthetic, vision, and language experiments, together with probes and a key-to-value alignment intervention, investigate why recovery differs among merge choices.

Critical strengths
  1. The paper isolates a practically relevant distinction between choosing a projection-sharing constraint before training and imposing it on an existing checkpoint. The immediate collapse after surgery makes that distinction concrete.
  2. The experiments span routing-dependent synthetic tasks, image classification, and language modeling. Comparing different merge choices reveals a consistent disadvantage for retaining the key projection, which gives the mechanism analysis a specific phenomenon to explain.
  3. The probe, random-projection control, addressing-versus-payload test, and key-to-value alignment intervention provide a useful investigation of representation compatibility. The paper also reports exceptions and limits rather than claiming uniform recovery.
Critical weaknesses
  1. The convex-hull observation applies to every K=V attention sub-block, including one trained jointly from scratch. It does not by itself explain why post-hoc surgery collapses while joint training works, and residual connections and later layers limit its direct model-level interpretation.
  2. Recovery is less complete than the broad framing suggests. In language modeling, after 500 million distillation tokens the best reported merge still has perplexity 22.87 versus 21.28 for the jointly trained reference, while retaining the key projection reaches 29.00. The claim of cheap, near-complete repair therefore depends strongly on task and merge choice.
  3. The practical repair comparison is incomplete. Distillation uses real training inputs, labels, and teacher logits, but the paper does not compare distillation with matched-budget ordinary fine-tuning or alignment followed by fine-tuning. It also does not measure the resulting inference latency or memory benefit in a deployed language model.
  4. The ImageNet confirmation uses one seed per condition and a shortened training schedule. The shared-projection model remains 2.71 percentage points below the independent-projection baseline, so the experiment supports a qualitative timing distinction more clearly than a stable accuracy-efficiency tradeoff.
  5. The basis-mismatch account is not equally established across scales. The alignment correction leaves the ImageNet zero-shot model far below usable accuracy, and the key/value content probes differ between synthetic tasks and the language model. These exceptions weaken a single general mechanism explanation.
Questions for authors
  1. What evidence distinguishes the explanation of post-hoc collapse from the convex-hull constraint that also holds when K=V is chosen before training?
  2. Under the same data and compute budget, how does distillation compare with standard fine-tuning or a lightweight alignment-plus-fine-tuning repair?
  3. How should the near-complete-recovery claim be qualified for language modeling, especially for the KEEP_K merge?
  4. Can the authors report seed variation or a longer-schedule check for the ImageNet joint-training comparison?
  5. What KV-cache, memory, and latency savings are measured after repair on the language model, and how do these compare with the repair cost?
1 comment

Thank you for the review. Several concerns overlap with those of other reviewers; we answer your five questions directly. (Table and section numbers follow the submission. In the revision, the new matched-budget table is Table 7 (main text), the new zero-shot control tables are Tables 9 and 10 (Appendix B), and the ImageNet-1k sweep table is Table 8.)

Q1 (hull vs. post-hoc collapse). Nothing in Proposition 1 distinguishes them, and we agree it cannot: the bound holds equally for K=V trained from scratch, and the K→V correction improves accuracy with the constraint unchanged. §3 is now titled "Which Tasks Can Be Affected", cites Tsai et al. (2019), and a new remark states it does not explain the collapse (residual connections and later layers also limit its model-level reading). Causal wording is removed from the Abstract, Table 1 and Conclusion. What distinguishes the regimes in our data is empirical: whether the downstream weights could co-adapt.

Q2 (distillation vs fine-tuning vs alignment+fine-tuning). We ran both at CIFAR scale: same teacher checkpoint, optimizer, data, 10 epochs, all parameters trained, 3 fine-tuning seeds. Top-1 %, CE only / KL only / CE+KL: CIFAR-10 KEEP_V 69.5/68.8/69.4, AVG 70.1/69.1/69.9, KEEP_K 64.8/66.5/66.5; CIFAR-100 KEEP_V 42.9/42.7/43.2, AVG 43.1/43.2/43.5, KEEP_K 40.0/41.2/41.2. So for KEEP_V/AVG, hard-label fine-tuning alone is within ~0.4 pt of CE+KL: the repair is largely ordinary continued training. For KEEP_K the teacher signal adds 1.1–1.7 pt. Continuing the unmodified teacher with CE for the same budget raises it by 1.1 pt on CIFAR-10 (69.6→70.7), which explains repaired students that "exceed the teacher". Alignment + fine-tuning: initializing KEEP_K from the closed-form K→V correction (linear / Procrustes) and fine-tuning with CE+KL gives 69.3 / 69.2 on CIFAR-10 (vs 66.5 for the raw KEEP_K copy; KEEP_V is 69.4) and 43.0 / 42.4 on CIFAR-100 (vs 41.2; KEEP_V 43.2), and with CE only 69.3 vs 64.8 (CIFAR-10). So the correction is a useful initialization that closes KEEP_K's deficit at this scale. These controls are CIFAR-scale with one teacher checkpoint; we did not run them at ImageNet-1k or LM scale, and the paper says so.

Q3 (qualifying LM recovery). Agreed. After 500M tokens (~5% of pretraining): AVG 22.87, KEEP_V 27.00, KEEP_K 29.00 vs 21.28 (+7.5%, +26.9%, +36.3%). The Abstract and Conclusion now give per-scale statements (KEEP_V/AVG within ~0.5 pt on MNIST, FMNIST and ImageNet-1k; KEEP_K 3–3.5 pt short on CIFAR; no merge at the ceiling at LM scale); "within a point", "at every scale" and "one consistent exception" are removed.

Q4 (seeds / longer schedule). We could not add either for ImageNet-1k in this period: a second from-scratch run is ~90 GPU-hours per condition on our hardware, a 300-epoch schedule about three times that, and the ImageNet data is not currently accessible to us. The Abstract describes the §5 sweep as a single-seed coherence check, not a controlled result, the gap to QKV (74.38 vs 71.67, 2.71 pt) is stated, and the positional-correction ablation is labelled single-run and suggestive. CIFAR fine-tuning seeds (3) are in the table above (sd ≤0.3), and at least the CIFAR ordering KEEP_K < KEEP_V is stable across them.

Q5 (KV-cache, memory, latency; repair cost). We have not measured any: our LM code has no cached-decoding path, so the saving in the paper is arithmetic (one stored tensor per layer vs two), and Limitations says so. Table 6's "Cache" column was likewise arithmetic and inconsistent (33% for Q=K=V vs 50% for K=V); it is now "KV cache", 50% for every shared variant, marked arithmetic. Repair cost, from logs: LM 500M tokens ≈ 4 h 19 min per merge mode on 2×A30 vs 58.8 h pretraining (same two A30s, per our launch scripts; ≈7% wall-clock, ≈6% of FLOPs including the teacher forward pass); ImageNet-1k 5 epochs ≈ 4.5–5 h per mode on one RTX 2080 Ti (from file timestamps) vs ≈92 GPU-hours from scratch.

Mechanism across scales. Agreed. The Abstract/Conclusion now say the account is consistent with the data at synthetic, CIFAR and LM scale but not established at ImageNet-1k zero-shot, where the linear correction recovers about three quarters of the top-1 gap to KEEP_V (0.0145 vs 0.0191, from 0.0008) and Procrustes 41%, but the model stays far below usable accuracy, and that the probes differ between synthetic tasks and the LM (V decodes better than K at all 20 LM layers). We also added zero-shot controls: rotation-free mean/covariance matching of c_k to c_v and the reverse V→K map do not recover (CIFAR-10 .106/.072 vs .152–.170 for the fitted K→V maps; SORT .534/.586 vs .740/.710), and part of the recovery is statistical (a random rotation inside the Procrustes pipeline reaches .122 on CIFAR-10).

Comments on the paper

2 comments

Thank you for a careful, fair review; we agree with much of it and have changed the paper accordingly. (Table and section numbers follow the submission. In the revision, the new matched-budget table is Table 7, the new control tables are Tables 9 and 10 (Appendix B), and the ImageNet-1k sweep is Table 8.)

W1. Agreed: the bound is identical for post-hoc and from-scratch K=V, and the K→V correction raises accuracy with the constraint unchanged (CIFAR-10 0.076→0.152), so it cannot explain collapse vs. repair. §3 is retitled "Which Tasks Can Be Affected", credits Tsai et al. (2019), and a new remark says it does not explain the collapse (and that standard attention is also confined to the hull of its values). Causal wording is removed from the Abstract, Table 1 and Conclusion.

W2. Agreed; your numbers are right. The Abstract and Conclusion now state it per scale: KEEP_V/AVG within ~0.5 pt of the ceiling on MNIST, FMNIST and ImageNet-1k; KEEP_K ~3–3.5 pt short on CIFAR; at LM scale no merge reaches the ceiling (+7.5% AVG, +26.9% KEEP_V, +36.3% KEEP_K). "Within a point", "at every scale" and "one consistent exception" are gone.

W3. You were right. We ran matched controls on CIFAR (Table 7): same teacher checkpoint, initialization, data, optimizer, 10 epochs, all parameters trained, 3 fine-tuning seeds; only the loss changes. Top-1 %, CE only / KL only / CE+KL (sd ≤0.3):

  • CIFAR-10: KEEP_K 64.8/66.5/66.5; KEEP_V 69.5/68.8/69.4; AVG 70.1/69.1/69.9
  • CIFAR-100: KEEP_K 40.0/41.2/41.2; KEEP_V 42.9/42.7/43.2; AVG 43.1/43.2/43.5

For KEEP_V/AVG, hard-label fine-tuning alone is within ~0.4 pt of CE+KL, so the repair there is largely continued training, not distillation. For KEEP_K the teacher signal adds 1.1–1.7 pt. Continuing the unmodified teacher with CE gives 70.7 on CIFAR-10 (+1.1 over its own 69.6), so repaired students "exceeding the teacher or ceiling" reflect the budget. The Abstract and Conclusion say this. Limits: CIFAR scale, one teacher checkpoint, not run at ImageNet-1k or LM scale (the ImageNet data is no longer mounted on our machine); the paper says the teacher signal's role there is untested. Trainable parameters: there are three separate protocols. Tables 2–3 train all student parameters (0.5·CE+0.5·KL, T=2); §4.1 trains only the tied c_k (21.0M of 305.5M) with KL alone; Table 5 trains nothing. §4 now has a table of the three, and the §6 "searching for a single linear map" sentence (speculation) is removed.

W6. The Abstract and Conclusion are rescaled as in W2, and the positional-correction ablation (66.76→69.26) is described as single-run and suggestive. CIFAR-10 CE+KL KEEP_K vs KEEP_V is 66.49±0.07 vs 69.44±0.12 over 3 fine-tuning seeds (the teacher is fixed, so this does not address teacher-seed variance). We could not add ImageNet seeds: a second from-scratch run is ~90 GPU-hours and the data is not accessible; sub-0.5 pt ImageNet comparisons should be read as qualitative.

W8. All confirmed and fixed. Size: unique parameters are 305.5M (teacher) and 284.5M (student); "389M" double-counted tied weights, and the paper says 305M throughout. 13345 vs 16589.7: two different pipelines, not noise. Table 3 uses a 2048-token context with padding masked; the c_k-only scripts (§4.1) use 512 tokens with padded positions in the loss. Re-running the original code path reproduces Table 3 (teacher 20.82; KEEP_K 13270; KEEP_V 7263; AVG 6365). Under §4.1's pipeline the teacher is 29.0, not the 20.82 drawn in Fig. 1; the figure is regenerated and "within 2.3× of the teacher" becomes 1.7×. The "no data-free variant" sentence is removed, Vaswani is cited, MASA is credited to Zhussip et al., and "four vision datasets" is now "four small-scale vision datasets".

QVV(3). You are right. In Gao & Xu, QVV(2) is shared-KV and QVV(3) adds a routing matrix W^R. Ours composes two full-width d×d query maps with K=V: the same functional form, containing a per-head W^R as a special case, but not verified against the authors' code. The paper now says "our implementation of the depth-3 member" and limits the non-replication claim to our implementation and 100-epoch schedule.

W4. We agree "compatibility" was a hypothesis; the Abstract now says "consistent with". We ran the controls (zero-shot, no training; Table 9), top-1:

KEEP_KKEEP_Vfitted K→V (lin / Procr.)rescale onlycov.-match, no rot.V→K linrand. rot. of c_v (5 seeds)rand. rot., Procr. pipeline
SORT.583.717.740 / .710.598.534.586.574±.013.623±.004
CIFAR-10.076.152.152 / .170.080.106.072.095±.038.122±.019
CIFAR-100.007.021.023 / .025.010.010.007.010±.002.012±.003

(cov.-match: the symmetric, rotation-free optimal-transport map between the Gaussians of c_k and c_v. Last column: the Procrustes pipeline with a random rotation.) (i) Rotation-free scale/covariance matching and the reverse V→K map do not recover on SORT or CIFAR-10, and a random rotation of c_v collapses to chance, so a pure scale/covariance account is not supported there. (ii) Part of the recovery is statistical: a random rotation in the same pipeline reaches .623 (SORT) and .122 (CIFAR-10), so we no longer attribute all of it to rotation. (iii) MNIST, FMNIST, REVERSE and SWAP are near chance in every column (KEEP_V included), so they do not discriminate; ImageNet-1k was not run. AVG's advantage shrinks with difficulty and reverses at ImageNet-1k (0.0066 vs 0.0191); this is noted in §4 and unexplained. You are right that a small mean singular value does not show low dimensionality (ridge shrinks it); the paper now reports the effective rank of the fitted map: 48.7 of 256 (vision), 68.1 of 256 (synthetic).

W5. Agreed. The Table 5 caption now states the bolding rule (the better of the two corrections, bold only where it matches or exceeds KEEP_V) and names the exceptions (REVERSE, SWAP, ImageNet-1k); "closes the entire gap" is qualified; the LM row is added (ppl: KEEP_K 13345, linear 6667, Procrustes 9281, KEEP_V 7347). On a continuous scale (Table 10), CIFAR-100 top-5 / KL to teacher: KEEP_K .041 / 8.33; random rotations .049–.052 / 8.29–8.86; KEEP_V .086 / 7.41; linear .092 / 7.08; Procrustes .101 / 6.13. So KEEP_K is indistinguishable from random bases while KEEP_V and both corrections are better. On CIFAR-10 the KL does not separate KEEP_K, KEEP_V and random bases. Not redone at ImageNet-1k.

W7. Fair. Our AVG is the K/V analogue of GQA's mean-pool conversion, the ordering matches Ainslie et al.'s ablation, and "orthogonal" was too quick. Related Work now says the protocol and ordering are prefigured by GQA, cites SkipV1Former (Wu et al., 2025), and does not claim them as new; what differs is what is merged (K and V of one head) and the task-type collapse/immunity split. We did not run a GQA uptraining baseline or measure cache/throughput (our LM code has no cached decoding); the saving is arithmetic, and Limitations says so. Table 6's "Cache" column was arithmetic and inconsistent (33% vs 50%) and is relabelled "KV cache" (50% for all shared variants). Repair cost, from logs: LM 500M tokens ≈ 4 h 19 min per mode on 2×A30 vs 58.8 h pretraining (same two A30s per our launch scripts; ≈7% wall-clock, ≈6% of FLOPs including the teacher forward pass); ImageNet-1k 5 epochs ≈ 4.5–5 h per mode on one RTX 2080 Ti vs ≈92 GPU-hours from scratch.

W9. Agreed. The Abstract calls the §5 sweep a coherence check (single seed, no controlled comparison), and Limitations states the two ImageNet deviations together as one open question. The LM experiment differs from the ViT in architecture, data and pipeline, but we did not add a second large-scale vision point (e.g., a public DeiT-S).

Not done: ImageNet seeds, an ImageNet/LM CE-only control, a GQA baseline, cache/latency measurements, and an external-teacher check.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.