- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper argues that queries and keys in an LLM do not require fully independent projections because they largely operate within a shared low-dimensional addressing space. The authors posted this idea by observing that cross-layer similarity patterns for queries closely mirror those of keys. Based on this insight, they replace the separate and matrices with a shared low-rank down-projection followed by two lightweight output heads, effectively forming a joint low-rank factorization of . To recover the expressivity lost through sharing, they introduce a small low-rank correction branch on the query side. In Multi-head Latent Attention, where keys are already compressed into a cached latent vector, this correction is also applied to the key side. The proposed attention block can be trained from scratch or applied as a parameter-efficient adapter during fine-tuning. Its primary goal is to reduce QK parameter counts, compute overhead, and MLA KV-cache footprints without degrading model quality.
Critical strengths
-
The design is simple and transfers well across settings. One shared structure works for standard attention, MLA, and LoRA-style fine-tuning without major changes. This makes the method easy to adopt in practice.
-
The experiments are reasonably large and carefully run. The 3B MoE model is trained for 200B tokens, and all three runs point in the same direction. The authors also measure real training throughput and memory instead of relying only on analytical FLOP counts.
Critical weaknesses
-
Weak link between the observation and the method. Such cross-layer Q/K patterns fail to establish a shared low-rank subspace within individual layers. Existing theory does not fully slove this gap, and its key assumptions remain empirically unverified. Beyond the joint low-rank decomposition that incorporates residual branches, it is unclear what other innovations this approach offers.
-
Insufficiently controlled comparisons. The SmolLM experiments change both the content representation and RoPE width, making the source of cache savings unclear. Qwen fine-tuning employs only rank-2 QK adapters and does not involve variance estimation. The "75% savings" figure mentioned in the paper applies exclusively to the QK projection component. When viewed from an end-to-end perspective, the actual speedup achieved is significantly smaller.
-
Limited ablation evidence and excessive terminology. In standard attention mechanisms, the gains achieved by adjusting branch positions are marginal, and there is a lack of validation based on multiple repeated runs. Consequently, these findings offer only limited support for the proposed branch positions. Furthermore, an excessive use of terminology obscures a simple core mechanism, which is quiet meretricious.
Questions for authors
-
Random-projection control for : What is the value of on a randomly initialized model or between two random projections of the same residual stream?
-
Fair baseline comparisons: How does this method perform compared to MFA, Thin-Key Attention, and joint low-rank QK factorization with shared-only parameters, given comparable parameter counts and fine-tuning budgets?
-
Controlled MLA ablation: In the MLA setup, what are the KV-cache savings and perplexity results when keeping the RoPE dimension and query latent strictly identical to the baseline and introducing only the QK coupling?
11 comments
Rebuttal by Authors (part 2)
(2) Cross-model controls support the breadth and training-related nature of the observation.
We use 100 common text batches (eight 4,096-token sequences each) with SmolLM-135M, Qwen2.5-3B, Llama-3-8B, and Qwen3-30B-A3B. Controls include position centering, per-token trajectories, identical-architecture random initialization, and shuffled K-layer correspondence.
Table R3-1. Q/K coupling under different statistical treatments and random controls.
| Model | Control-protocol | Position-centered | Per-token | Random init. | Shuffled K-layer correspondence |
|---|---|---|---|---|---|
| SmolLM-135M | 0.81444 | 0.35904 | 0.58829 | 0.01610 | 0.01870 |
| Qwen2.5-3B | 0.43187 | 0.41063 | 0.25985 | 0.02455 | −0.00181 |
| Llama-3-8B | 0.80243 | 0.52540 | 0.49344 | 0.03204 | 0.01100 |
| Qwen3-30B-A3B | 0.84345 | 0.81815 | 0.78108 | 0.02738 | 0.01930 |
All four models retain positive coupling after centering and per-token calculation, while random initialization and K-layer shuffling yield near-zero scores. The relationship therefore occurs within content tokens and depends on the trained depth correspondence, rather than on batch-averaged means alone.
We also test whether the first token or a short prefix drives the signal.
Table R3-2. Q/K coupling after prefix exclusion.
| Model | Control-protocol | Exclude first token | Exclude first 16 tokens |
|---|---|---|---|
| SmolLM-135M | 0.81444 | 0.81442 | 0.81441 |
| Qwen2.5-3B | 0.43187 | 0.43184 | 0.43185 |
| Llama-3-8B | 0.80243 | 0.80242 | 0.80228 |
| Qwen3-30B-A3B | 0.84345 | 0.84336 | 0.84334 |
Excluding the first 16 tokens changes by at most about 0.00015, versus trained–shuffled gaps of about 0.43–0.82. The observed cross-layer relationship is stable under prefix exclusion.
The anchor-wide pattern is similarly broad: 26/30 SmolLM, 30/32 Llama, and 39/48 Qwen3 anchors exceed 0.7; for Qwen2.5, 17/36 exceed 0.5 (median 0.4823). Thus, the shared relational signal extends across many reference depths, consistent with applying the shared mechanism throughout the network.
(3) Matched-budget attention reconstruction tests the usefulness of the common structure.
To test whether the common structure is useful, we fit alternative low-rank Q/K maps to frozen pre-trained hidden states on calibration text and evaluate on disjoint text, at matched QK matrix-parameter budgets. All controls retain native output widths, positional encoding, and causal attention.
We measure between original and reconstructed attention probabilities, aggregated across tokens, heads, and examples. Lower KL directly indicates more faithful token-addressing distributions. Independent low-rank, shared-only, and shared–private alternatives distinguish the value of common features from that of private corrections.
Each calibration setting uses 800 sequences and 4,096 sampled tokens; evaluation uses 100 separate batches. SmolLM, Qwen2.5, and Llama cover all layers. Qwen3 uses 12 preselected full-context GQA layers spanning its 48-layer depth (1, 5, 10, 14, 18, 22, 27, 31, 35, 39, 44, 48).
Table R3-3. Attention KL on a causal submatrix of 256 common sampled positions, retaining 25% of QK matrix parameters (lower is better).
| Model | Independent low-rank | Shared-only | Q-private | K-private | Joint-private |
|---|---|---|---|---|---|
| SmolLM-135M | 0.099335 | 0.098150 | 0.094224 | 0.098634 | 0.091870 |
| Qwen2.5-3B | 0.055663 | 0.052297 | 0.041544 | 0.046260 | 0.061981 |
| Llama-3-8B | 0.043707 | 0.044527 | 0.043442 | 0.044181 | 0.049122 |
| Qwen3-30B-A3B | 0.064333 | 0.064759 | 0.051802 | 0.053944 | 0.076005 |
At 25% QK budget, Q-private correction lowers KL relative to independent low-rank by 5.15%, 25.37%, 0.61%, and 19.48% across the four models; joint correction is best on SmolLM (KL 0.091870, a 7.51% reduction). Matching the same budget to common features and appropriate private directions preserves addressing more faithfully. The preferred private allocation varies by model, motivating independent readouts and flexible corrections.
Table R3-4. Reconstruction KL for the full 4,096-token causal attention distribution (lower is better).
| Model | QK parameters retained | Independent low-rank | Shared-only | Q-private | K-private | Joint-private |
|---|---|---|---|---|---|---|
| Llama-3-8B | 12.5% | 0.123411 | 0.142341 | 0.122048 | 0.135456 | 0.130952 |
| Llama-3-8B | 25% | 0.070699 | 0.094782 | 0.071298 | 0.086500 | 0.078858 |
| Llama-3-8B | 50% | 0.034564 | 0.054900 | 0.027308 | 0.041330 | 0.038056 |
| Qwen3-30B-A3B | 25% | 1.093301 | 2.064639 | 1.043679 | 1.521450 | 1.261645 |
Rebuttal by Authors (part 3)
(3) Matched-budget attention reconstruction tests the usefulness of the common structure. (continue)
Full-4K attention reconstruction gives complementary evidence. Q-private is best for Llama at 12.5% and 50% budget and for Qwen3 at 25%: at Llama 50% and Qwen3 25%, KL falls by 20.99% and 4.54% versus independent low-rank, and by 50.26% and 49.45% versus shared-only. At Llama 25%, independent low-rank (0.070699) and Q-private (0.071298) are close, with Q-private still 24.78% below shared-only. Private directions improve full-context attention fidelity under restricted capacity, complementing the cross-layer diagnostic with functional evidence.
Standard-attention/GQA reconstruction and MLA pre-training evaluate private allocation in their respective architectures. Both retain the same design principle: reuse common Q/K inputs, preserve operator-specific flexibility, and choose its allocation under the target budget.
(4) Full training tests common-feature reuse and private supplementation separately.
Revised Tables 19, 39, and 40 compare 350M QK parameterizations over the same 40K steps and dataset. Efficient variants use approximately 50% QK matrix parameters; the independent low-rank baseline has a tuned learning rate.
Table R3-5. 350M pre-training from scratch with a common 40K-step horizon and a tuned independent low-rank baseline.
| Parameterization | QK matrix parameters retained | Final validation PPL ↓ |
|---|---|---|
| Full-rank | 100% | 18.0500 |
| Independent-LR (tuned) | 50% | 18.1900 |
| Shared-only | 49.9512% | 17.7677 |
| QK-coupling | 50% | 17.6400 |
Shared-only improves PPL from 18.0500 (full rank) and 18.1900 (independent low rank) to 17.7677; adding private corrections reaches 17.6400, a further 0.1277 improvement. The two comparisons isolate the contributions of shared reuse and private expression in end-to-end training.
The 130M control independently tunes learning rates over , with three training seeds per configuration; matched variants retain 25% of the QK matrix parameters across 12 layers.
Table R3-6. Capacity and parameterization comparisons at 130M. PPL is the three-run mean (sample SD).
| Method | QK matrix parameters | Fraction retained | Validation PPL: mean (sample SD) |
|---|---|---|---|
| Independent-LR, rank 96 | 3.538944M | 25% | 22.5760 (0.0977) |
| Independent-LR, rank 192 | 7.077888M | 50% | 22.1410 (0.0789) |
| Shared-only, rank 128 | 3.538944M | 25% | 22.3279 (0.0944) |
| Direct-narrow, head dimension 16 | 3.538944M | 25% | 22.9725 (0.1207) |
| QK-coupling | 3.538944M | 25% | 22.0955 (0.0740) |
QK-coupling has the lowest mean PPL at equal 25% QK budget and wins all three runs against independent low-rank, shared-only, and direct narrowing. Doubling the independent low-rank budget brings its mean PPL only close to QK-coupling (22.1410 versus 22.0955). The 350M and repeated 130M results therefore directly support the parameter efficiency of shared inputs with private corrections.
Placement is separately tested within each architecture: the original MLA ablation favors joint correction (PPL 10.5827 versus 10.7193 Q-private and 10.6736 K-private); three-run standard-attention ablations favor Q-private at 60M and 350M (detailed in R3-3). The architecture-specific comparisons determine where to allocate private capacity.
(5) Rank selection has a reusable starting point and direct sensitivity results.
Ranks follow an explicit resource budget. For equal-width standard attention, full-rank Q/K uses matrix parameters and the shared–private design uses . Setting , , retains 25% of the QK parameters (75% saving); , , retains 50%. These width-relative settings are directly evaluated in the pre-training and matched-budget comparisons.
At approximately 50% QK budget, default, shared-heavy, shared-only, and private-heavy 350M allocations give PPL 17.6400, 17.7048, 17.7677, and 17.8854, respectively; full rank gives 18.0500. Several allocations work, with the default best under the tested protocol.
As in CoLA [1], CR-Net [2], and MLRA [3], a structural observation motivates a trainable computation; the architecture and budget determine its realized capacity. Here, multi-anchor coupling motivates Q/K sharing, while the channel approximation result, attention reconstruction, and matched-budget training evaluate the chosen ranks and corrections.
In sum, the contribution is a Q/K-specific relational diagnostic, an implementable shared-input/separate-readout/private-correction parameterization, and direct evidence that both sharing and private capacity improve quality per QK parameter. The theory characterizes the relevant relational and channel errors; the ablations establish the component roles.
Rebuttal by Authors (part 1)
Response to Reviewer pe1j
Thank you for your detailed review. Below, we address your comments and questions using the original results and the additional experiments.
Response overview
Statistical convention. For the newly added repeated experiments, we report the mean across three independent random seeds, with the sample standard deviation (SD) in parentheses. SD is measured in PPL units for perplexity and in percentage points for percentage-valued task scores. We use s=∑i(xi−xˉ)2/(n−1) with n=3; results cited from the original submission retain their original statistical definitions.
R3-1. Structural observation, error analysis, and the shared–private design (W1)
The contribution links an observed Q/K relational structure to a specific sharing mechanism and tests its components under matched budgets. The multi-anchor diagnostic motivates reusable input features; separate readouts and low-rank private corrections retain Q/K-specific expression. We connect this observation to the paper's error analysis, attention reconstruction, and pre-training controls.
(1) Q/K-specific relational coordination identifies what to share.
Q/K attention scores determine which tokens are read. Here, routing geometry refers to the relationships among the Q/K representations forming those scores. Equations (7)–(8) compare Q-to-Q and K-to-K similarities across layers relative to the same anchor, measuring whether the two operators organize depth-wise relations similarly. This motivates reusing their common input features while preserving distinct Q/K outputs.
Theorem 1 operates in the cross-layer relational space: for normalized relational matrices CQ,CK, set M=(CQ+CK)/2 and D=(CQ−CK)/2. Reconstructing both with a shared p-dimensional factor gives Rp=τp+∥(I−Pp)D∥F2/L≤τp+(1−κˉ)/2, where τp is the discarded spectral energy of M. The common spectrum and the Q/K relational mismatch jointly govern the shared reconstruction bound.
For unit-normalized anchor trajectories cQ(a),cK(a), let ma=(cQ(a)+cK(a))/2 and da=(cQ(a)−cK(a))/2. Their energies satisfy ∥ma∥22=(1+κa)/2 and ∥da∥22=(1−κa)/2. The original training trajectory raises κˉ from 0.496 to 0.952, reducing differential relational energy from 0.252 to 0.024 (90.48%). The observed increase thus quantifies stronger common relational organization; Theorem 1 additionally accounts for the common component's spectral tail.
The trainable, within-layer design is Z=XAS, QS=ZBQ, and KS=ZBK (Eq. 10): a shared input projection with separate Q/K readouts. Proposition 1 independently characterizes the best joint channel reconstruction at shared rank rs as ∑j>rsσj2(Σ1/2[WQWK]), where Σ is the input second moment. Together, the relational analysis motivates common-feature reuse, and the input-weighted channel result characterizes the approximation capacity of its implementation.
Private corrections complement this factorization. Theorem 3 identifies their first-order score directions, δS=δQ(KS)⊤+QS(δK)⊤; in MLA, the compressed key-content support also shapes the directions accessible to query-side corrections. Sharing avoids duplicated input computation, independent readouts retain asymmetric Q/K roles, and the private branches provide additional addressing directions. The following matched-budget results test all three choices.