- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper analyzes the geometry of the multi-head latent attention in language models regarding the weight matrices of the keys and of the values. The authors introduce two theorems and conduct an empirical analysis considering three different sizes of DeepSeek V2/V3. They demonstrate that the keys and values are geometrically proximate (in the shared low-rank space) but functionally orthogonal, meaning each adds an individual contribution. While the motivation is set out clearly at the beginning, the paper reveals page by page the actual analysis, which one strongly suggests improving.
Critical strengths
- Formal analysis with theorems
- Supported by empirical analysis
Critical weaknesses
- Empirical study on three sizes of one model
- Conclusion drawn from analysis is too strong (Line 412)
Questions for authors
-
Line 080 aims to make the novelty and contribution beyond prior work precise; this sentence should go to the abstract, intro, and conclusion. In particular, the Discussion in Line 423 is weak in convincing the reader of the novelty of the paper.
-
Line 227 should give more context to SVD
-
Line 235 ff and 254 ff: Remarks are better embedded in prose motivating the remark itself; this allows to keep the remark short and precise instead of writing a paragraph and then turning it into a remark environment. Similarly, Corollary 1 stands isolated in its own subsection.
-
Line 376 is a very good one-sentence summary of the paper; one wishes to have read this clearly in the abstract/introduction and to be used to build up the paper structure. Will make the reading of the paper much better.
-
Line 412: too strong statement of "a phenomenon of very large models"; this is neither motivated by the theory nor investigated beyond the single model considered in the paper
-
Section 3.4 states generally that the subspace is shared whenever two projections target the same low-rank approximation: Thus, one suggests consequently rewriting the "Surrogate Rank-Constrained Problem" in general fashion, turning to a specific KV analysis in the empirical analysis. Same for the introduction.
-
Metrics Section 4.1: mention the two metrics in the intro/contribution, with the conclusion drawn from Section 4.2 in Line 376. Will make the paper much clearer.
Overall, reading the paper was OK, but not particularly enjoyable, as one has to rethink the story on the read.
2 comments
Response to authors' response
Thank you for your responses clarifying my questions. A keep my score.
Response
Thank you for the constructive suggestions. We agree with most of them. One caveat: the empirical results you evaluated rest on the slicing error described in the general comment, so they are not established as stated (Section 4.5), but your structural suggestions stand independently. The updated PDF has room for only one added page, so the changes below are for the next revision unless marked otherwise.