- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper asks the question of what happens when independently trained key and value projections are replaced by a shared projection after training. It reports substantial performance degradation followed by recovery through supervised training combined with distillation, with recovery depending on how the shared projection is initialized. Probing and alignment experiments suggest that incompatibility between representations and downstream weights contributes to the damage, while experiments with models trained from scratch show that the sharing constraint can be accommodated during training.
Critical strengths
- The research direction is important and interesting.
Critical weaknesses
- The paper is heavily AI-written and bombarded with a lot of numbers in the text, making it really hard to follow a story line through the paper.
- The central benefit and efficiency of distillation are not established against ordinary fine-tuning. In more detail, main repair procedure uses the original training data, hard labels, and updates throughout the student, in addition to the teacher’s soft targets. However, there is no matched hard-label-only fine-tuning baseline using the same initialization, data, optimizer, and training budget. Consequently, the experiments cannot determine whether distillation is important or whether ordinary adaptation after a disruptive architectural edit would recover similarly.
- Proposition 1 is a correct convex-combination observation, but standard attention is likewise confined to the convex hull of its value vectors. As a result, the proposition alone does not establish a performance obstruction or explain the difference between post-hoc and joint training.
Questions for authors
- How does hard-label-only fine-tuning compare with CE-plus-KL distillation under matched initialization, data, optimizer, and compute?
- What are the measured repair FLOPs or GPU-hours, including teacher inference, and the resulting inference-time memory and throughput benefits? What exactly does the “Cache” column in Table 6 measure?
2 comments
Thank you for the, again, very very detailed and numbered response. While these address some of my concerns and help clarify some of the findings, they do not change my overall assessment of the paper’s contribution and its understandability. I will therefore maintain my original score.
Response
Thank you for the review. We answer the two questions first. (Table and section numbers follow the submission. In the revision, the new matched-budget table is Table 7 (main text), the new zero-shot control tables are Tables 9 and 10 (Appendix B), and the ImageNet-1k sweep table is Table 8.)
Q1: hard-label-only fine-tuning vs CE+KL. You are right that the original submission could not separate them. We ran the matched control at CIFAR scale: same teacher checkpoint, initialization, data, optimizer (Adam, lr 3e-4, cosine, 10% warmup), batch size and 10 epochs, all parameters trained, 3 fine-tuning seeds; only the loss differs. Top-1 %, CE only / KL only / CE+KL: CIFAR-10 KEEP_K 64.8/66.5/66.5, KEEP_V 69.5/68.8/69.4, AVG 70.1/69.1/69.9; CIFAR-100 KEEP_K 40.0/41.2/41.2, KEEP_V 42.9/42.7/43.2, AVG 43.1/43.2/43.5 (sd ≤0.3). So for KEEP_V and AVG, hard-label fine-tuning alone is within ~0.4 pt of CE+KL: that repair is largely ordinary adaptation after the edit, not distillation. For KEEP_K the teacher signal adds 1.1–1.7 pt. Continuing the unmodified QKV teacher with CE for the same budget gives 70.7 on CIFAR-10 (+1.1 over its own 69.6), so repaired students "exceeding the teacher" reflect the training budget. The Abstract and Conclusion now state this. We did not run the control at ImageNet-1k or LM scale (one LM repair run is ≈4 h on two A30s per merge mode, and the ImageNet data is no longer accessible to us); the paper says the teacher signal's contribution there is untested.
Q2: repair cost, inference benefit, and the "Cache" column. Cost (from logs): LM (305.5M-parameter teacher; 284.5M after K=V): 500M tokens ≈ 4 h 19 min per merge mode on 2×A30, vs 58.8 h for the 10B-token pretraining (same two A30s, per our launch scripts; ≈7% of wall-clock, ≈6% of FLOPs counting the teacher's forward pass as 2ND). ImageNet-1k: 5 epochs ≈ 4.5–5 h per mode on one RTX 2080 Ti (from file timestamps) vs ≈92 GPU-hours for a from-scratch run. Inference benefit: not measured; our LM code has no cached-decoding path, and Limitations now says so. Cache column: it was not measured. It was an arithmetic count and internally inconsistent (33% for Q=K=V but 50% for K=V, though both store one tensor at decode time since queries are not cached). It is now labelled "KV cache" and reads 50% for every shared variant, marked as arithmetic.
Proposition 1. Agreed. Standard attention is also confined to the convex hull of its value vectors, so the bound alone shows no obstruction and cannot explain post-hoc vs joint training. We added this to the paper, retitled §3 as a scoping observation about which tasks can be affected, and removed the causal wording from the Abstract and Conclusion.
Presentation. We respectfully disagree that the paper lacks a storyline, and would welcome specifics. It is built around one contrast (Table 1: imposing K=V after training vs choosing it before), with one sub-question per section in order: which tasks can be affected (§3), how bad the collapse is and how much training repairs it (§4), why (§4.1), and whether choosing the constraint up front avoids the problem (§5). Numbers sit in tables, and the prose around each states its takeaway. Both other reviewers rated presentation 3 ("clear and easy to follow"). If particular passages were hard to follow, please point us to them and we will rewrite them. On AI assistance: the paper's AI-use statement discloses it (drafting and editing, merging two source documents, writing analysis scripts, formatting) and states what we did not delegate (experimental design, hypotheses, Proposition 1 and its proof). The authors checked numbers against both source documents and sanity-checked the scripts against known baselines, and take responsibility for the content. We do not think the use of an editing tool bears on whether the experiments and claims are sound, and we are glad to address any specific technical point.