acceptodds
Under review as a conference paper at ICLR 2027

One Subspace, Many Functions: The Geometry of Multi-Head Latent Attention

Abstract

Multi-head Latent Attention (MLA) routes key and value projections through a shared low-rank bottleneck. A natural question is whether this creates functional redundancy between the two pathways and whether that redundancy can be exploited for compression. We show that it does not. The bottleneck forces keys and values into the same representational coordinate system, but the model uses that system to encode maximally diverse, non-interchangeable representations. We demonstrate this using the Eckart-Young-Mirsky theorem applied to gauge-invariant effective operators, since the raw factor matrices are non-identifiable under latent reparameterization and therefore cannot be compared directly. Auditing DeepSeek-V2-Lite, DeepSeek-V2, and DeepSeek-V3 reveals a universal middle-layer subspace collapse, with a scale-dependent U-shaped profile that becomes clearly visible only in the 671B-parameter model. Bidirectional causal interventions confirm that despite near-zero Gram distance in collapsed layers, forcing the key and value representations to be identical causes a perplexity increase of approximately 10 points in both directions. A 2.9× asymmetry further shows that corruption of the key pathway is substantially more damaging than corruption of the value pathway.

Primary area:
learning theory
Keywords:
Multi-head Latent Attention, DeepSeek, Representation Learning, Subspace Collapse, Functional Redundancy, Key-Value Representations, Low-Rank Bottlenecks, Gauge-Invariant Analysis, Scaling Laws, Representation Geometry, Causal Interventions, Mechanistic Interpretability, Model Compression, Efficient Transformers

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pc7mOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzes the geometry of the multi-head latent attention in language models regarding the weight matrices WKW_K of the keys and WVW_V of the values. The authors introduce two theorems and conduct an empirical analysis considering three different sizes of DeepSeek V2/V3. They demonstrate that the keys and values are geometrically proximate (in the shared low-rank space) but functionally orthogonal, meaning each adds an individual contribution. While the motivation is set out clearly at the beginning, the paper reveals page by page the actual analysis, which one strongly suggests improving.

Critical strengths
  • Formal analysis with theorems
  • Supported by empirical analysis
Critical weaknesses
  • Empirical study on three sizes of one model
  • Conclusion drawn from analysis is too strong (Line 412)
Questions for authors
  • Line 080 aims to make the novelty and contribution beyond prior work precise; this sentence should go to the abstract, intro, and conclusion. In particular, the Discussion in Line 423 is weak in convincing the reader of the novelty of the paper.

  • Line 227 should give more context to SVD

  • Line 235 ff and 254 ff: Remarks are better embedded in prose motivating the remark itself; this allows to keep the remark short and precise instead of writing a paragraph and then turning it into a remark environment. Similarly, Corollary 1 stands isolated in its own subsection.

  • Line 376 is a very good one-sentence summary of the paper; one wishes to have read this clearly in the abstract/introduction and to be used to build up the paper structure. Will make the reading of the paper much better.

  • Line 412: too strong statement of "a phenomenon of very large models"; this is neither motivated by the theory nor investigated beyond the single model considered in the paper

  • Section 3.4 states generally that the subspace is shared whenever two projections target the same low-rank approximation: Thus, one suggests consequently rewriting the "Surrogate Rank-Constrained Problem" in general fashion, turning to a specific KV analysis in the empirical analysis. Same for the introduction.

  • Metrics Section 4.1: mention the two metrics in the intro/contribution, with the conclusion drawn from Section 4.2 in Line 376. Will make the paper much clearer.

Overall, reading the paper was OK, but not particularly enjoyable, as one has to rethink the story on the read.

2 comments

Thank you for the constructive suggestions. We agree with most of them. One caveat: the empirical results you evaluated rest on the slicing error described in the general comment, so they are not established as stated (Section 4.5), but your structural suggestions stand independently. The updated PDF has room for only one added page, so the changes below are for the next revision unless marked otherwise.

  • Single model family, three sizes. Agreed. The next revision states the empirical scope as a limitation and makes no claim about scale.
  • "A phenomenon of very large models" (Line 412). Retracted. Neither the theory nor the data supports it: one model family, checkpoints that differ in more than size, no functional test on V2/V3, and a measurement error that affects every model. V2/V3 also have 128 heads versus 16 in V2-Lite, which alone could change a head-group statistic (Section 4.5).
  • Novelty statement, one-sentence summary, metrics (Lines 080, 376, Section 4.1). We will move the one-sentence summary into the abstract and introduction, state the contribution relative to prior work there and in the conclusion, and name the metrics in the contributions list so the structure follows the claim.
  • Section 3.4 generality. We will state the surrogate problem for any two projections sharing a low-rank factor, and specialise to K and V only where the paper discusses MLA.
  • Remarks, Corollary 1, SVD context (Lines 227, 235 ff, 254 ff). Remarks will be folded into prose, and Corollary 1 will be merged into the surrounding discussion. We will add the SVD context. The theory section will be shortened (see our reply to yLkb).
  • Discussion (Line 423). The Discussion will be rewritten around what the paper supports after the corrections, with the contribution and its limits stated plainly.
Reviewer_pc7mOpenReview ↗

Response to authors' response

Thank you for your responses clarifying my questions. A keep my score.

Reviewer_zGsGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This submission examines whether MLA’s shared KV bottleneck produces functional redundancy, but its theoretical and empirical results do not establish its central functional claims. The rank bound is largely a consequence of matrix dimensions, while the Eckart–Young–Mirsky argument solves an idealized approximation problem without establishing the reported training behavior or intervention sensitivity. Equality of the effective column spaces, and hence zero principal angles under the stated full-rank conditions, follows from the architecture and is explicitly acknowledged by the authors. The functional evidence consists of direct parameter replacement in one model on one dataset, without adequate controls for perturbation magnitude, attention-logit scale, or coordinate compatibility. The absence of recovery experiments further limits conclusions about exploitable redundancy. The scaling interpretation rests on comparisons between different checkpoints that do not isolate model size, with no corresponding functional validation on the larger models. More seriously, the reported checkpoint configurations and KV slicing conflict with the public implementations: if implemented as written, the clamping procedure would not perform the claimed selective K←V or V←K intervention. The reported collapse-zone boundaries also contradict the manuscript’s own threshold definition and tabulated measurements. These problems affect what was measured, what was intervened upon, and whether the conclusions follow. They substantially undermine the reliability of the analysis and leave the paper’s central claims unestablished.

Critical strengths

The paper raises a relevant question about whether MLA’s shared KV bottleneck implies functional redundancy. This provides useful motivation for studying compression, but it carries limited weight in my assessment given the concerns about the validity of the reported measurements and interventions.

Critical weaknesses
  1. The reported KV extraction and clamping are inconsistent with the public implementation, undermining the central empirical evidence.

Section 4.2, Eqs. (5)–(6), extracts KK and VV by splitting kv_b_proj.weight into two contiguous halves, and Appendix E.1 uses the same layout for clamping. However, the official DeepSeek-V2-Lite implementation first reshapes the projection output by attention head and then separates KK and VV within each head. The two halves therefore contain the complete key and value projections of different groups of heads. If implemented as written, copying one half into the other would replace both KK and VV for a group of heads, rather than perform the claimed selective K←VK \leftarrow V or V←KV \leftarrow K intervention. The same discrepancy affects the identity of the operators used for the geometric measurements. Because both the geometric analysis and the causal interpretation depend on this extraction, I cannot rely on the central empirical results without clarification of the actual code and verification that the reported measurements and interventions were performed correctly.

  1. The theoretical results and replacement experiments do not establish the central claim about functional redundancy.

Theorem 2 characterizes an optimal low-rank approximation under a Frobenius surrogate. It does not establish actual training dynamics, predict small normalized Gram distances, or imply symmetric sensitivity to replacement between KK and VV. Nevertheless, Section 4.4.4 and Appendix E.5 present the empirical observations as confirmation of the theorem; Appendix E.5 explicitly attributes the nearly symmetric perplexity changes to its prediction. Even assuming a correct clamping implementation, the intervention does not resolve this gap. The unmodified baseline and bidirectional replacements test the consequences of direct substitution, but do not distinguish the proposed mechanism from disruption of learned coordinates or attention-logit scales. Without appropriate alignment and perturbation controls, degradation in one model on one dataset cannot establish the absence of exploitable redundancy. Recovery through adaptation also remains untested. These limitations leave the paper’s principal functional interpretation unsupported, independently of the implementation concerns above.

Questions for authors
  1. Did the reported experiments use the slicing shown in the paper? Please provide the code used to extract and replace KK and VV.
  2. How does Theorem 2 imply the nearly equal perplexity changes reported in Appendix E.5? I do not see this connection in the theorem.
  3. Did you check whether the performance drop comes from a mismatch in scale or coordinates? Without such a check, why does failed replacement establish a lack of functional redundancy?
2 comments

Thank you for this review. It identified the most serious problem in the submission.

Q1: Did the experiments use the slicing shown in the paper? Please provide the code. Yes. The experiments used exactly the contiguous slicing shown in the paper, and your reading is correct. The correct operation is:

W = kv_b_proj.weight.view(n_heads, d_nope + d_v, r)
W_K_up = W[:, :d_nope, :].reshape(n_heads * d_nope, r)
W_V_up = W[:, d_nope:, :].reshape(n_heads * d_v,   r)
# clamp K <- V :  W[:, :d_nope, :] = W[:, d_nope:, :]
# clamp V <- K :  W[:, d_nope:, :] = W[:, :d_nope, :]

The consequences are as you describe (see the general comment and Section 4.5). Because the geometric measurements and the interventions both depend on this extraction, the updated PDF no longer presents them as evidence about keys versus values. The corrected code and tests will be released.

Collapse-zone boundaries. You are right that they contradict our own definition (contiguous layers with D_Gram < 2·D_min), and the tables show it. For V2-Lite the threshold is 0.0406, yet layer 22 has 0.0459 and sits inside "7–26". For V2 the threshold is 0.016, yet layer 23 has 0.0181 and sits inside "22–54". For V3 the threshold is 0.0194, yet layers 9–16 are 0.018–0.034 and sit inside "9–57"; the stated 79% coverage is unsupported. The late-layer statistics are also inconsistent between text and table (V2-Lite 0.046/2.3× versus 0.0335/1.7×). The next revision removes the zone boundaries together with the measurements they describe.

Q2: How does Theorem 2 imply the nearly equal ΔPPL changes in E.5? It does not. Theorem 2 is the Eckart–Young–Mirsky solution of a one-step Frobenius surrogate. It says nothing about training dynamics, and it implies neither small D_Gram nor symmetric sensitivity to replacement. Section 4.5 already says that the symmetry is not evidence about keys versus values; as explained above, it was most likely an artifact of the head-group swap. The next revision removes Finding 2 and the sentence in E.5 that attributes the symmetry to the theorem.

Q3: Scale or coordinate mismatch? Why does failed replacement establish a lack of functional redundancy? We did not check, and as you say, failed direct replacement cannot establish that. It conflates loss of information, a mismatch in scale, and a mismatch in coordinates. Section 4.5 now lists the controls: a scale control (rescale to the original Frobenius norm), a coordinate control (orthogonal Procrustes alignment before substitution), a magnitude-matched random-perturbation baseline, and recovery through short fine-tuning of the modified up-projections, all applied per head and in both directions. It also notes that an informative per-head test is possible, because each head's block has rank at most 128, below r = 512, so per-head principal angles are not forced to zero (unlike the aggregate ones, which are). We have not run these experiments, and we make no claim about exploitable redundancy.

Theory. We agree that the rank bound follows from matrix dimensions, and that equality of the column spaces and θ̄≈0 follow from the architecture (Section 4.1 says so). The next revision moves the theory to a short appendix, presents the surrogate as motivation only, and deletes the language that treats the experiments as confirming it.

Dear Reviewer,

A gentle reminder that the discussion period ends soon. We’d be glad to hear whether our response and revised PDF address your concerns, and happy to answer anything further before the deadline.

Thank you for your time.

Reviewer_yLkbOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies Multi-Head Latent Attention as implemented in DeepSeek language models, which uses compression to produce compact representation for efficient KV-caching. The paper studied empirically to what extent the key and value embeddings are constrained by the bottleneck of the shared compressed representation c. The empirical results show that despite passing through the same low-dimensional bottleneck, keys and values are substantially independent. They are read from different subspaces of the shared representation at come layers, and cause a noticeable drop in perplexity if forced to be identical.

Critical strengths
  • the math looks solid if partly redundant for the empirical study
  • the empirical work is substantial and serious
Critical weaknesses
  • To the extent the paper asks an interesting question, it cannot be answered without model retraining. In particular, the observed perplexity results are somewhat expected and don't present a strong argument for anything.
  • The theoretical section is dressed up in flashy math but mostly essentially restates how Multi-Head Latent Attention works. Eckart-Young-Mirsky connection reads like name-dropping and doesn't seem to contribute substantially to the argument. Overall, the math offers only limited motivation to the actual empirical part of the study, and doesn't appear valuable despite the impressive appearance. The same math could be more useful if the empirical studies relied more on it, for example if the predictions about gradients were tested against training dynamics.
Questions for authors
  • U-shape claims are only partially supported. Why do you think they are?
  • what happens qualitatively at early layers for which you report that keys and values are particularly divergent? Does this correspond to distinctive attention patterns?
4 comments

Thank you for the review.

"The question cannot be answered without retraining; the perplexity results are expected." We agree that naively overwriting weights in a trained model is expected to hurt, and that such an experiment cannot settle whether the pathways are functionally redundant. As run, it did not even measure K-versus-V, because of the slicing error described above. Section 4.5 says the results are not established as statements about keys versus values and lists the controls that would make a replacement experiment informative. The next revision removes the functional conclusion drawn from them.

"The math is dressed up; the EYM connection is name-dropping; test predictions against training dynamics." We accept this. Theorem 1 (rank of gradients) and the rank bound are immediate from matrix dimensions. Theorem 2 solves an idealised surrogate and does not describe training. The next revision compresses both into a short appendix and removes the framing that the experiments confirm them. We agree that a theoretical prediction should be tested against training behaviour; we have not done this.

Q1: "U-shape claims are only partially supported. Why?" You are right, and the reason is more basic than we realised. The V2-Lite and V2 profiles are not U-shaped by our own numbers. In V3 the late rise is two isolated layers (58 and 60) with layer 59 back at baseline, and the last layer in our table is the MTP module. More importantly, the D_Gram values were computed on head groups, not on K versus V (Section 4.5), so the profile cannot currently be interpreted. Section 4.5 states that the U-shape and scale-dependence readings are not established; the next revision removes them from the abstract and text.

Q2: "What happens qualitatively in early layers? Do they have distinctive attention patterns?" We did not examine attention patterns. Because the early-layer D_Gram values were computed on head groups, we cannot say what the early-layer result means, and we do not interpret it. If a corrected per-head audit shows early-layer divergence, attention-pattern statistics for those layers would be the natural follow-up.

Dear Reviewer,

A gentle reminder that the discussion period ends soon. We’d be glad to hear whether our response and revised PDF address your concerns, and happy to answer anything further before the deadline.

Thank you for your time.

Thank you for your response, I decide to maintain my current value.

Thank you for the detailed response to my review. I appreciate that you removed the unnecessary maths from the main text.

Unfortunately, the other parts of your response simply concede the weaknesses and do not repair them substantively, only weakening the claims. This is understandable as properly addressing the research question requires retraining models, which is not feasible within the discussion period.

I keep my scores and wish the authors all the best in developing their research.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Comment to all reviewers

We thank the reviewers. Reviewer zGsG found a real implementation error, and we want to say so first.

The KV slicing is wrong. Section 4.2 and Appendix E.1 split kv_b_proj.weight into two contiguous halves, rows [0 : n_h·d_head] for "K" and the rest for "V". In DeepSeek-V2/V3 the output of kv_b_proj is reshaped to (heads, d_nope + d_v) and then split inside each head. We checked this against the transformers implementation (modeling_deepseek_v2.py: key_shape = (..., -1, qk_nope_head_dim + v_head_dim) followed by torch.split(k_nope, [qk_nope_head_dim, v_head_dim], dim=-1)). Our extraction and clamping code used the contiguous layout. As a result:

  • The "K" block was the K and V projections of the first half of the heads. The "V" block was the K and V projections of the second half.
  • D_Gram compared two groups of heads, not keys against values. Its low mid-network values may mostly reflect that two large groups of heads have similar aggregate statistics. That would also be consistent with the apparent scale effect, since V2 and V3 have 128 heads and V2-Lite has 16.
  • The "K←V" and "V←K" interventions were head-group swaps. Each overwrote one half of the heads (both K and V) with the other half. Near-symmetric ΔPPL (+10.10 / +10.72) is then expected, and it says nothing about the K-versus-V structure. The same applies to the 2.9× "key-pathway primacy" result.
  • The activation experiment (ρ ≈ 0 and its correlation with D_Gram) is affected as well: its hook on kv_b_proj split the output with the same contiguous layout.

What the updated PDF contains. The new Section 4.5 states all of this. It gives the correct per-head layout and extraction, says that the D_Gram, ρ and clamping results, the near-symmetric ΔPPL, the 2.9× asymmetry and the U-shape and scale-dependence readings are not established as statements about keys versus values until recomputed per head, and says that we have not repeated the experiments. It also lists the controls that a valid replacement test needs (below).

Other corrections, also in Section 4.5:

  • V2 and V3 have 128 heads, and hidden sizes of 5120 and 7168. Section 4 says 16 heads and d=5120 for all three models; this is wrong for all three (V2-Lite has d=2048).
  • V3 has 61 layers; layer 61 in our tables is the MTP module.
  • kv_b_proj does not contain the RoPE key. That lives in kv_a_proj_with_mqa.
  • kv_a_layernorm (RMSNorm with a learned gain) sits between W_c and the up-projections. The effective operators must include the gain, and the invariance of Section 3.2 holds only for orthogonal reparametrisations with gain compensation, not for all invertible R.

What we have not done, and what we will do. The discussion-period page budget allowed one added page, so the abstract, Section 4.4, the Discussion, the Conclusion and Appendix E still carry the original wording in this PDF, and Section 4.5 contradicts them. We will bring those sections in line with Section 4.5 in the next revision. That revision will also drop the collapse-zone boundaries (see the reply to zGsG), shorten the theory, and restructure the paper along reviewer pc7m's suggestions. We have not re-run the experiments with the corrected slicing and we make no claim about what they will show. We will release the corrected extraction and clamping code, with tests that check them against a reference attention forward pass that splits per head.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.