acceptodds
Under review as a conference paper at ICLR 2027

Multi-anchor QK-coupling in LLMs: Shared Attention Routing and Efficient Learning

Abstract

We identify multi-anchor QK-coupling, an anchor-relative geometry in which query and key trajectories remain coupled across layers, revealing shared routing structure in trained LLMs. We analyze this observation from three complementary perspectives: shared relational reconstruction error is jointly controlled by the common component's spectral tail and the mismatch measured by multi-anchor coupling; under spectral-dominance and relational-orthogonality conditions, common-core-preserving MLA-style compression can increase coupling; and private residual branches provide complementary first-order score-space correction directions around a fixed shared representation. Guided by these results, we propose a QK-coupled parameterization with a shared routing trunk and lightweight query-private/key-private residual branches. The design supports standard pre-training, MLA-style compressed-KV pre-training, and LoRA-style fine-tuning, reducing QK projection parameters, projection FLOPs, and training-state storage, with additional latent KV-cache savings on the evaluated SmolLM-style MLA backbones. Experiments show improved or competitive quality, including 75% QK-side FLOP/optimizer-memory savings with better perplexity in standard pre-training and a 50.57% QK-path projection-FLOP reduction with better final perplexity in a 3B, 200B-token MLA-style run.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Anchor QK Coupling, Shared Routing Geometry, Shared-Private Routing, Compressed-KV Attention, Query-Key Reparameterization.

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pe1jOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that queries and keys in an LLM do not require fully independent projections because they largely operate within a shared low-dimensional addressing space. The authors posted this idea by observing that cross-layer similarity patterns for queries closely mirror those of keys. Based on this insight, they replace the separate WQW_Q and WKW_K matrices with a shared low-rank down-projection followed by two lightweight output heads, effectively forming a joint low-rank factorization of [WQ,WK][W_Q, W_K]. To recover the expressivity lost through sharing, they introduce a small low-rank correction branch on the query side. In Multi-head Latent Attention, where keys are already compressed into a cached latent vector, this correction is also applied to the key side. The proposed attention block can be trained from scratch or applied as a parameter-efficient adapter during fine-tuning. Its primary goal is to reduce QK parameter counts, compute overhead, and MLA KV-cache footprints without degrading model quality.

Critical strengths
  1. The design is simple and transfers well across settings. One shared structure works for standard attention, MLA, and LoRA-style fine-tuning without major changes. This makes the method easy to adopt in practice.

  2. The experiments are reasonably large and carefully run. The 3B MoE model is trained for 200B tokens, and all three runs point in the same direction. The authors also measure real training throughput and memory instead of relying only on analytical FLOP counts.

Critical weaknesses
  1. Weak link between the observation and the method. Such cross-layer Q/K patterns fail to establish a shared low-rank subspace within individual layers. Existing theory does not fully slove this gap, and its key assumptions remain empirically unverified. Beyond the joint low-rank decomposition that incorporates residual branches, it is unclear what other innovations this approach offers.

  2. Insufficiently controlled comparisons. The SmolLM experiments change both the content representation and RoPE width, making the source of cache savings unclear. Qwen fine-tuning employs only rank-2 QK adapters and does not involve variance estimation. The "75% savings" figure mentioned in the paper applies exclusively to the QK projection component. When viewed from an end-to-end perspective, the actual speedup achieved is significantly smaller.

  3. Limited ablation evidence and excessive terminology. In standard attention mechanisms, the gains achieved by adjusting branch positions are marginal, and there is a lack of validation based on multiple repeated runs. Consequently, these findings offer only limited support for the proposed branch positions. Furthermore, an excessive use of terminology obscures a simple core mechanism, which is quiet meretricious.

Questions for authors
  1. Random-projection control for κQK\kappa_{QK}: What is the value of κQK\kappa_{QK} on a randomly initialized model or between two random projections of the same residual stream?

  2. Fair baseline comparisons: How does this method perform compared to MFA, Thin-Key Attention, and joint low-rank QK factorization with shared-only parameters, given comparable parameter counts and fine-tuning budgets?

  3. Controlled MLA ablation: In the MLA setup, what are the KV-cache savings and perplexity results when keeping the RoPE dimension and query latent strictly identical to the baseline and introducing only the QK coupling?

11 comments
AuthorsOpenReview ↗

Rebuttal by Authors (part 1)

Response to Reviewer pe1j

Thank you for your detailed review. Below, we address your comments and questions using the original results and the additional experiments.

Response overview

  • Contribution: Relational observations, error analysis, and component ablations explain the shared/private division (R3-1).
  • Random controls: Compare random initialization and fixed versus layer-wise-independent projections of a common input (R3-4).
  • Compression benefits: Separate content and positional savings; add fixed-positional-width comparisons and measured efficiency (R3-2).
  • Fine-tuning: Compare MFA, ThinKeys, and shared-only adaptation, extend to R8, and report three-seed statistics (R3-5).
  • Clarity and repeatability: Describe the components directly and validate private-branch allocation through repeated runs (R3-3).

Statistical convention. For the newly added repeated experiments, we report the mean across three independent random seeds, with the sample standard deviation (SD) in parentheses. SD is measured in PPL units for perplexity and in percentage points for percentage-valued task scores. We use s=∑i(xi−xˉ)2/(n−1)s=\sqrt{\sum_i(x_i-\bar x)^2/(n-1)} with n=3n=3; results cited from the original submission retain their original statistical definitions.

R3-1. Structural observation, error analysis, and the shared–private design (W1)

The contribution links an observed Q/K relational structure to a specific sharing mechanism and tests its components under matched budgets. The multi-anchor diagnostic motivates reusable input features; separate readouts and low-rank private corrections retain Q/K-specific expression. We connect this observation to the paper's error analysis, attention reconstruction, and pre-training controls.

(1) Q/K-specific relational coordination identifies what to share.

Q/K attention scores determine which tokens are read. Here, routing geometry refers to the relationships among the Q/K representations forming those scores. Equations (7)–(8) compare Q-to-Q and K-to-K similarities across layers relative to the same anchor, measuring whether the two operators organize depth-wise relations similarly. This motivates reusing their common input features while preserving distinct Q/K outputs.

Theorem 1 operates in the cross-layer relational space: for normalized relational matrices CQ,CKC_Q,C_K, set M=(CQ+CK)/2M=(C_Q+C_K)/2 and D=(CQ−CK)/2D=(C_Q-C_K)/2. Reconstructing both with a shared pp-dimensional factor gives Rp=τp+∥(I−Pp)D∥F2/L≤τp+(1−κˉ)/2R_p=\tau_p+\|(I-P_p)D\|_F^2/L\leq\tau_p+(1-\bar\kappa)/2, where τp\tau_p is the discarded spectral energy of MM. The common spectrum and the Q/K relational mismatch jointly govern the shared reconstruction bound.

For unit-normalized anchor trajectories cQ(a),cK(a)c_Q^{(a)},c_K^{(a)}, let ma=(cQ(a)+cK(a))/2m_a=(c_Q^{(a)}+c_K^{(a)})/2 and da=(cQ(a)−cK(a))/2d_a=(c_Q^{(a)}-c_K^{(a)})/2. Their energies satisfy ∥ma∥22=(1+κa)/2\|m_a\|_2^2=(1+\kappa_a)/2 and ∥da∥22=(1−κa)/2\|d_a\|_2^2=(1-\kappa_a)/2. The original training trajectory raises κˉ\bar\kappa from 0.496 to 0.952, reducing differential relational energy from 0.252 to 0.024 (90.48%). The observed increase thus quantifies stronger common relational organization; Theorem 1 additionally accounts for the common component's spectral tail.

The trainable, within-layer design is Z=XASZ=XA^S, QS=ZBQQ^S=ZB_Q, and KS=ZBKK^S=ZB_K (Eq. 10): a shared input projection with separate Q/K readouts. Proposition 1 independently characterizes the best joint channel reconstruction at shared rank rsr_s as ∑j>rsσj2(Σ1/2[WQ  WK])\sum_{j>r_s}\sigma_j^2(\Sigma^{1/2}[W_Q\;W_K]), where Σ\Sigma is the input second moment. Together, the relational analysis motivates common-feature reuse, and the input-weighted channel result characterizes the approximation capacity of its implementation.

Private corrections complement this factorization. Theorem 3 identifies their first-order score directions, δS=δQ(KS)⊤+QS(δK)⊤\delta S=\delta Q(K^S)^\top+Q^S(\delta K)^\top; in MLA, the compressed key-content support also shapes the directions accessible to query-side corrections. Sharing avoids duplicated input computation, independent readouts retain asymmetric Q/K roles, and the private branches provide additional addressing directions. The following matched-budget results test all three choices.

AuthorsOpenReview ↗

Rebuttal by Authors (part 2)

(2) Cross-model controls support the breadth and training-related nature of the observation.

We use 100 common text batches (eight 4,096-token sequences each) with SmolLM-135M, Qwen2.5-3B, Llama-3-8B, and Qwen3-30B-A3B. Controls include position centering, per-token trajectories, identical-architecture random initialization, and shuffled K-layer correspondence.

Table R3-1. Q/K coupling κ\kappa under different statistical treatments and random controls.

ModelControl-protocol κ\kappaPosition-centeredPer-tokenRandom init.Shuffled K-layer correspondence
SmolLM-135M0.814440.359040.588290.016100.01870
Qwen2.5-3B0.431870.410630.259850.02455−0.00181
Llama-3-8B0.802430.525400.493440.032040.01100
Qwen3-30B-A3B0.843450.818150.781080.027380.01930

All four models retain positive coupling after centering and per-token calculation, while random initialization and K-layer shuffling yield near-zero scores. The relationship therefore occurs within content tokens and depends on the trained depth correspondence, rather than on batch-averaged means alone.

We also test whether the first token or a short prefix drives the signal.

Table R3-2. Q/K coupling κ\kappa after prefix exclusion.

ModelControl-protocol κ\kappaExclude first tokenExclude first 16 tokens
SmolLM-135M0.814440.814420.81441
Qwen2.5-3B0.431870.431840.43185
Llama-3-8B0.802430.802420.80228
Qwen3-30B-A3B0.843450.843360.84334

Excluding the first 16 tokens changes κ\kappa by at most about 0.00015, versus trained–shuffled gaps of about 0.43–0.82. The observed cross-layer relationship is stable under prefix exclusion.

The anchor-wide pattern is similarly broad: 26/30 SmolLM, 30/32 Llama, and 39/48 Qwen3 anchors exceed 0.7; for Qwen2.5, 17/36 exceed 0.5 (median 0.4823). Thus, the shared relational signal extends across many reference depths, consistent with applying the shared mechanism throughout the network.

(3) Matched-budget attention reconstruction tests the usefulness of the common structure.

To test whether the common structure is useful, we fit alternative low-rank Q/K maps to frozen pre-trained hidden states on calibration text and evaluate on disjoint text, at matched QK matrix-parameter budgets. All controls retain native output widths, positional encoding, and causal attention.

We measure DKL(At∥A^t)D_{\mathrm{KL}}(A_t\Vert\widehat A_t) between original and reconstructed attention probabilities, aggregated across tokens, heads, and examples. Lower KL directly indicates more faithful token-addressing distributions. Independent low-rank, shared-only, and shared–private alternatives distinguish the value of common features from that of private corrections.

Each calibration setting uses 800 sequences and 4,096 sampled tokens; evaluation uses 100 separate batches. SmolLM, Qwen2.5, and Llama cover all layers. Qwen3 uses 12 preselected full-context GQA layers spanning its 48-layer depth (1, 5, 10, 14, 18, 22, 27, 31, 35, 39, 44, 48).

Table R3-3. Attention KL on a causal submatrix of 256 common sampled positions, retaining 25% of QK matrix parameters (lower is better).

ModelIndependent low-rankShared-onlyQ-privateK-privateJoint-private
SmolLM-135M0.0993350.0981500.0942240.0986340.091870
Qwen2.5-3B0.0556630.0522970.0415440.0462600.061981
Llama-3-8B0.0437070.0445270.0434420.0441810.049122
Qwen3-30B-A3B0.0643330.0647590.0518020.0539440.076005

At 25% QK budget, Q-private correction lowers KL relative to independent low-rank by 5.15%, 25.37%, 0.61%, and 19.48% across the four models; joint correction is best on SmolLM (KL 0.091870, a 7.51% reduction). Matching the same budget to common features and appropriate private directions preserves addressing more faithfully. The preferred private allocation varies by model, motivating independent readouts and flexible corrections.

Table R3-4. Reconstruction KL for the full 4,096-token causal attention distribution (lower is better).

ModelQK parameters retainedIndependent low-rankShared-onlyQ-privateK-privateJoint-private
Llama-3-8B12.5%0.1234110.1423410.1220480.1354560.130952
Llama-3-8B25%0.0706990.0947820.0712980.0865000.078858
Llama-3-8B50%0.0345640.0549000.0273080.0413300.038056
Qwen3-30B-A3B25%1.0933012.0646391.0436791.5214501.261645
AuthorsOpenReview ↗

Rebuttal by Authors (part 3)

(3) Matched-budget attention reconstruction tests the usefulness of the common structure. (continue)

Full-4K attention reconstruction gives complementary evidence. Q-private is best for Llama at 12.5% and 50% budget and for Qwen3 at 25%: at Llama 50% and Qwen3 25%, KL falls by 20.99% and 4.54% versus independent low-rank, and by 50.26% and 49.45% versus shared-only. At Llama 25%, independent low-rank (0.070699) and Q-private (0.071298) are close, with Q-private still 24.78% below shared-only. Private directions improve full-context attention fidelity under restricted capacity, complementing the cross-layer diagnostic with functional evidence.

Standard-attention/GQA reconstruction and MLA pre-training evaluate private allocation in their respective architectures. Both retain the same design principle: reuse common Q/K inputs, preserve operator-specific flexibility, and choose its allocation under the target budget.

(4) Full training tests common-feature reuse and private supplementation separately.

Revised Tables 19, 39, and 40 compare 350M QK parameterizations over the same 40K steps and dataset. Efficient variants use approximately 50% QK matrix parameters; the independent low-rank baseline has a tuned learning rate.

Table R3-5. 350M pre-training from scratch with a common 40K-step horizon and a tuned independent low-rank baseline.

ParameterizationQK matrix parameters retainedFinal validation PPL ↓
Full-rank100%18.0500
Independent-LR (tuned)50%18.1900
Shared-only49.9512%17.7677
QK-coupling50%17.6400

Shared-only improves PPL from 18.0500 (full rank) and 18.1900 (independent low rank) to 17.7677; adding private corrections reaches 17.6400, a further 0.1277 improvement. The two comparisons isolate the contributions of shared reuse and private expression in end-to-end training.

The 130M control independently tunes learning rates over {10−4,5×10−4,10−3,5×10−3,10−2}\{10^{-4},5\times10^{-4},10^{-3},5\times10^{-3},10^{-2}\}, with three training seeds per configuration; matched variants retain 25% of the QK matrix parameters across 12 layers.

Table R3-6. Capacity and parameterization comparisons at 130M. PPL is the three-run mean (sample SD).

MethodQK matrix parametersFraction retainedValidation PPL: mean (sample SD)
Independent-LR, rank 963.538944M25%22.5760 (0.0977)
Independent-LR, rank 1927.077888M50%22.1410 (0.0789)
Shared-only, rank 1283.538944M25%22.3279 (0.0944)
Direct-narrow, head dimension 163.538944M25%22.9725 (0.1207)
QK-coupling3.538944M25%22.0955 (0.0740)

QK-coupling has the lowest mean PPL at equal 25% QK budget and wins all three runs against independent low-rank, shared-only, and direct narrowing. Doubling the independent low-rank budget brings its mean PPL only close to QK-coupling (22.1410 versus 22.0955). The 350M and repeated 130M results therefore directly support the parameter efficiency of shared inputs with private corrections.

Placement is separately tested within each architecture: the original MLA ablation favors joint correction (PPL 10.5827 versus 10.7193 Q-private and 10.6736 K-private); three-run standard-attention ablations favor Q-private at 60M and 350M (detailed in R3-3). The architecture-specific comparisons determine where to allocate private capacity.

(5) Rank selection has a reusable starting point and direct sensitivity results.

Ranks follow an explicit resource budget. For equal-width standard attention, full-rank Q/K uses 2d22d^2 matrix parameters and the shared–private design uses d(3rs+2rq+2rk)d(3r_s+2r_q+2r_k). Setting rs=d/8r_s=d/8, rq=d/16r_q=d/16, rk=0r_k=0 retains 25% of the QK parameters (75% saving); rs=d/4r_s=d/4, rq=d/8r_q=d/8, rk=0r_k=0 retains 50%. These width-relative settings are directly evaluated in the pre-training and matched-budget comparisons.

At approximately 50% QK budget, default, shared-heavy, shared-only, and private-heavy 350M allocations give PPL 17.6400, 17.7048, 17.7677, and 17.8854, respectively; full rank gives 18.0500. Several allocations work, with the default best under the tested protocol.

As in CoLA [1], CR-Net [2], and MLRA [3], a structural observation motivates a trainable computation; the architecture and budget determine its realized capacity. Here, multi-anchor coupling motivates Q/K sharing, while the channel approximation result, attention reconstruction, and matched-budget training evaluate the chosen ranks and corrections.

In sum, the contribution is a Q/K-specific relational diagnostic, an implementable shared-input/separate-readout/private-correction parameterization, and direct evidence that both sharing and private capacity improve quality per QK parameter. The theory characterizes the relevant relational and channel errors; the ablations establish the component roles.

Reviewer_tcTROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies multi-anchor QK-coupling, an anchor-relative geometry in which query and key trajectories remain coupled across layers, revealing shared routing structure in trained LLMs. The authors analyze this observation from three complementary perspectives. Guided by these results, they propose a QK-coupled parameterization with a shared routing trunk and lightweight query-private and key-private residual branches. The trunk captures the common routing geometry once, reducing QK projection cost, while query-private and key-private corrections preserve asymmetric score formation. The resulting defaults use query-private correction for standard attention and LoRA-style fine-tuning, and joint query-private and key-private correction for MLA-style attention. The design supports standard pre-training, MLA-style compressed-KV pre-training, and LoRA-style fine-tuning, reducing QK projection parameters, projection FLOPs, and training-state storage. They validate it across these three regimes, with matched-budget ablations on private-branch placement, comparisons against other QK-efficient parameterizations, multi-seed runs, and measured training efficiency.

Critical strengths
  • The method is simple, drop-in, and tested across three regimes. The matched-budget comparison against grouped QK and a tuned independent low-rank QK, the shared-only control, and the three-seed MLA run with consistent paired gains all support the claim that a shared input trunk for Q and K costs little or nothing in perplexity.

  • The measured throughput and memory also show that the analytical savings are real.

  • The private-branch placement analysis gives a clean, checkable criterion. The block decomposition in Theorem 3 is easy to follow, and the matched-budget ablations agree with it.

Critical weaknesses
  • The headline discovery is not shown to measure routing geometry, and it does not directly motivate the method. The score is a Frobenius cosine between raw activations from different layers, computed after averaging over the batch. That measure is not invariant to per-layer reparameterizations that leave attention scores unchanged, and batch averaging leaves mostly mean and position-wise components.

  • The method shares an input subspace inside one layer, while the score compares cross-layer profiles. No experiment shows that the score predicts where the method helps.

  • The efficiency story is narrower than the abstract suggests, and the evaluation cannot catch the most likely failure mode. The 75% figure applies to the QK projections only; total parameter savings are 5 to 11% at 60M to 1.3B, and the 1.3B step time drops by under 2%. The 27.7% time-to-quality gain comes mostly from reaching a perplexity target in fewer steps. In the 3B MLA run, the conservative setting that beats the baseline needs a larger per-token cache than the baseline: 480 plus 80 plus 64 against 512 plus 64, and the 9B setting is also larger. Only the aggressive setting saves cache, and it is worse than the baseline. Cache savings are shown only on 130M and 350M SmolLM models with a latent of 32.

  • Pre-training is judged by perplexity alone, with sequence length 256 in the standard-attention study and no downstream or long-context retrieval tests. A shared low-rank QK subspace is most likely to hurt exactly on sharp retrieval, which perplexity at short context will not reveal.

Questions for authors
  • Are the profiled activations taken before or after RoPE, and do they include the RoPE branch in MLA?

  • How do the coupling scores change with a rotation-invariant measure such as linear CKA, without batch averaging, and for a randomly initialized model of the same architecture?

  • Please confirm the per-token cache of the conservative 3B and 9B settings. Is there a configuration that both beats the MLA baseline and uses no more cache?

  • The learnable per-head mixing scalars act partly as a per-head score scale. Does a full-rank baseline with QK-norm or a learnable per-head scale close the perplexity gap at small scale?

  • For the 3B MLA models, can you report downstream zero-shot results and a long-context retrieval test such as RULER or needle-in-a-haystack?

13 comments
AuthorsOpenReview ↗

Rebuttal by Authors (part 1)

Response to Reviewer tcTR

Thank you for your detailed review. Below, we address your comments and questions using the original results and the additional experiments.

Response overview

  • Geometric interpretation: Coordinate transformations, CKA, and random controls clarify the directional information captured by the original κ\kappa (R2-1, R2-5, R2-6).
  • Functional validation: Matched-budget reconstruction and ablations connect cross-layer observations to within-layer sharing (R2-1, R2-2).
  • Quality and efficiency: Equal-cache and compact configurations improve PPL, alongside measured system benefits (R2-3).
  • Retrieval capability: The aggressive configuration saves cache while exceeding the baseline on overall downstream and RULER scores (R2-4).
  • Enhanced baselines: Compare QK normalization, per-head score scales, and branch-mixing configurations (R2-7).

Statistical convention. For the newly added repeated experiments, we report the mean across three independent random seeds, with the sample standard deviation (SD) in parentheses. SD is measured in PPL units for perplexity and in percentage points for percentage-valued task scores. We use s=∑i(xi−xˉ)2/(n−1)s=\sqrt{\sum_i(x_i-\bar x)^2/(n-1)} with n=3n=3; results cited from the original submission retain their original statistical definitions.

R2-1. Routing geometry, coordinate freedom, and the functional value of sharing (W1)

Thank you for raising this important question. We study the formation of shared Q/K relationships in trained representations and their use for efficient computation. The relational metric identifies structure, the error analysis characterizes the potential of joint representation, and attention-distribution reconstruction and full pre-training directly test functional value. Below, we explain these roles and provide controls for batch averaging and coordinate freedom.

(1) Shared-reconstruction analysis connects the relational observation to the parameterization.

Self-attention uses Q/K scores to determine where a token reads information. By “routing geometry,” we mean the organization of the representations that form these matching relationships. Equations (7)–(8) compare the cross-layer similarity trajectories of Q and K relative to the same anchor, rather than requiring Q and K within a layer to be identical. A high κa\kappa_a therefore means that Q and K organize their relationships across network depth similarly when referenced to layer aa. Observing this pattern across multiple anchors motivates applying within-layer reuse of common Q/K features at multiple depths.

For the normalized relational matrices CQ,CKC_Q,C_K, define M=(CQ+CK)/2M=(C_Q+C_K)/2 and D=(CQ−CK)/2D=(C_Q-C_K)/2. Theorem 1 in the original submission bounds the mean squared error of reconstructing both trajectory families in the same pp-dimensional relational space: Rp=τp+∥(I−Pp)D∥F2/L≤τp+(1−κˉ)/2R_p=\tau_p+\|(I-P_p)D\|_F^2/L\leq\tau_p+(1-\bar\kappa)/2. Here, PpP_p retains the leading pp singular directions of the common component MM, and τp\tau_p is the normalized energy in its discarded directions.

This result explains the role of coupling in joint low-rank modeling: the common spectrum determines the dimensionality needed to represent shared information, while coupling quantifies the relational differences that remain beyond it. At a fixed common spectral tail, stronger coupling yields a smaller upper bound on joint reconstruction error. This motivates representing the common component once and supplementing it with side-specific corrections.

Write the two unit-normalized trajectories for anchor aa as cQ(a)c_Q^{(a)} and cK(a)c_K^{(a)}, and let ma=(cQ(a)+cK(a))/2m_a=(c_Q^{(a)}+c_K^{(a)})/2 and da=(cQ(a)−cK(a))/2d_a=(c_Q^{(a)}-c_K^{(a)})/2. Then ∥ma∥22=(1+κa)/2\|m_a\|_2^2=(1+\kappa_a)/2 and ∥da∥22=(1−κa)/2\|d_a\|_2^2=(1-\kappa_a)/2. The two terms sum to 1: they are the common/differential energy decomposition of the original relational representation, rather than an additional fitted metric. Along the training trajectory in the paper, κˉ\bar\kappa rises from 0.496 to 0.952, corresponding to a reduction in differential energy from 0.252 to 0.024, approximately 90.48%. Stronger coupling during training thus corresponds to less relational variation requiring separate representation. Together with the common spectral tail in Theorem 1, this supports sharing common features and using private branches to supplement their differences.

Equation (10) implements this idea as Z=XAS, QS=ZBQ, KS=ZBKZ=XA^S,\ Q^S=ZB_Q,\ K^S=ZB_K. Proposition 1 shows that, for input second moment Σ\Sigma, the optimal joint output-reconstruction error is ∑j>rsσj2(Σ1/2[WQ  WK])\sum_{j>r_s}\sigma_j^2(\Sigma^{1/2}[W_Q\;W_K]). Common input features are therefore computed once and used through separate Q/K readouts; the information retained at a given channel capacity is characterized by the joint spectrum weighted by the actual input distribution. The relational analysis motivates sharing, while the within-layer parameterization provides a trainable implementation.

AuthorsOpenReview ↗

Rebuttal by Authors (part 2)

(2) Centering, per-token computation, and random controls examine the source of the observation.

We extend the analysis to public pre-trained checkpoints of SmolLM-135M, Qwen2.5-3B, Llama-3-8B, and Qwen3-30B-A3B. All controls use the same 100 sampled text batches, each containing eight 4,096-token sequences. Position centering removes each channel's mean over positions. The per-token analysis first constructs a relational trajectory for each example and token position, and then aggregates the results.

Table R2-1. Q/K coupling κ\kappa under different statistical treatments and random controls.

ModelControl-protocol κ\kappaPosition-centeredPer-tokenRandom init.Shuffled K-layer correspondence
SmolLM-135M0.814440.359040.588290.016100.01870
Qwen2.5-3B0.431870.410630.259850.02455−0.00181
Llama-3-8B0.802430.525400.493440.032040.01100
Qwen3-30B-A3B0.843450.818150.781080.027380.01930

All four models retain positive coordination after centering and under per-token computation, showing that the structure is present beyond the mean component and within individual tokens. Random initialization tests whether the architecture alone produces the relationship and yields near-zero scores. Layer shuffling preserves the trained activations while breaking the depth correspondence between Q and K, and also yields near-zero scores. These controls indicate that both trained representations and their cross-layer correspondence contribute to the observed coupling.

To examine prefix and attention-sink sensitivity, we additionally exclude the first token and the first 16 tokens, reporting the results before and after exclusion.

Table R2-2. Q/K coupling κ\kappa after prefix exclusion.

ModelControl-protocol κ\kappaExclude first tokenExclude first 16 tokens
SmolLM-135M0.814440.814420.81441
Qwen2.5-3B0.431870.431840.43185
Llama-3-8B0.802430.802420.80228
Qwen3-30B-A3B0.843450.843360.84334

Excluding the first 16 tokens changes the reported scores by at most approximately 0.00015, compared with a trained–shuffled gap of approximately 0.43–0.82. The prefix effect is much smaller than the separation between the trained models and the random controls, supporting a relationship that is stable under prefix exclusion. Together with centering, per-token computation, and random controls, this supports the presence of coordination broadly across content positions rather than its concentration in a few prefix positions.

(3) Score-preserving coordinate transformations distinguish cross-layer organization from the within-layer attention function.

We apply the same orthogonal signed permutation to post-RoPE Q and K within each layer, so that (QR)(KR)⊤=QK⊤(QR)(KR)^\top=QK^\top, choosing the permutation independently across layers. Here, S=QK⊤/dhS=QK^\top/\sqrt{d_h} is the pre-softmax attention-score matrix, whose entries are query–key matching scores. The maximum relative attention-score error is obtained by computing ∥S−S~∥F/∥S∥F\|S-\widetilde S\|_F/\|S\|_F for each tested matrix and taking the maximum. This checks that the actual attention-score computation is preserved.

Table R2-3. Coupling and attention scores before and after independent layer-wise coordinate transformations.

ModelPre-transform κ\kappa at sampled positionsTransformed κ\kappaMaximum relative attention-score error
SmolLM-135M0.815150.337611.15×10−71.15\times10^{-7}
Qwen2.5-3B0.43026−0.006362.05×10−72.05\times10^{-7}
Llama-3-8B0.801110.367911.64×10−71.64\times10^{-7}
Qwen3-30B-A3B0.843210.036741.43×10−71.43\times10^{-7}

The score error remains on the order of 10−710^{-7}, confirming that the within-layer addressing function is preserved. Meanwhile, κ\kappa decreases in all four models, showing that independent changes to each layer's coordinates alter the cross-layer correspondence in native representations. This specifies the metric's object: depth-wise relational organization in trained representations. Its value for efficient computation is assessed through the functional comparisons below. This transformation differs from applying the same RoPE rotation at a given position across all layers: the latter preserves cross-layer inner products, whereas independent layer-wise permutations change directional correspondence across layers.

AuthorsOpenReview ↗

Rebuttal by Authors (part 3)

(4) Direct addressing experiments and full training test the usefulness of the structure.

We hold fixed the actual hidden states produced by each pre-trained checkpoint, fit alternative low-rank Q/K maps on separate calibration text, and evaluate them on disjoint text. The methods use the same QK matrix-parameter budget and retain the native output widths, positional encoding, and causal attention. We fit the Q/K maps and use KL divergence to evaluate the resulting attention distributions.

Specifically, let AtA_t denote the original model's attention probabilities over visible keys at query position tt, and let A^t\widehat A_t denote the probabilities after replacing the Q/K maps. We evaluate DKL(At∥A^t)D_{\mathrm{KL}}(A_t\Vert\widehat A_t) and aggregate over positions, heads, and examples. Lower KL means that the compressed model more faithfully preserves where the original model reads information and the weights assigned to those locations. Comparing against independent low-rank maps tests the value of sharing; comparing shared-only and shared–private maps tests the value of retaining side-specific directions.

Each of the two calibration settings uses 800 sequences and 4,096 sampled calibration tokens, with evaluation on another 100 batches. We cover all layers of SmolLM, Qwen2.5, and Llama. For the 48-layer Qwen3 model, we preselect layers 1, 5, 10, 14, 18, 22, 27, 31, 35, 39, 44, and 48 across depth, all of which are full-context GQA layers.

Table R2-4. Attention KL on a causal submatrix of 256 common sampled positions, retaining 25% of QK matrix parameters (lower is better).

ModelIndependent low-rankShared-onlyQ-privateK-privateJoint-private
SmolLM-135M0.0993350.0981500.0942240.0986340.091870
Qwen2.5-3B0.0556630.0522970.0415440.0462600.061981
Llama-3-8B0.0437070.0445270.0434420.0441810.049122
Qwen3-30B-A3B0.0643330.0647590.0518020.0539440.076005

The Q-private configuration achieves lower mean KL than independent low-rank maps on all four models. Joint private correction further improves SmolLM to 0.091870, a 7.51% reduction relative to independent low-rank maps. Allocating the same QK budget to common features and appropriate private corrections therefore preserves the original token-reading distribution more accurately. Q-private correction gives the best result on the three larger public models, while joint correction is best on SmolLM. These model-dependent preferences support retaining independent Q/K readouts and private branches to accommodate side-specific directions.

In the full 4K evaluation, Q-private correction reduces KL from 0.034564 to 0.027308 at Llama's 50% matrix budget, a 20.99% improvement, and from 1.093301 to 1.043679 at Qwen3's 25% budget, a 4.54% improvement. Reusing common input features with a query-private correction therefore more accurately recovers the token-reading distribution across the full context, demonstrating a directly measurable addressing benefit beyond a high representation cosine.

Tables 19, 39, and 40 of the revised manuscript compare QK parameterizations on a 350M standard-attention model using the same training data and a 40K-step horizon. Apart from the full-rank reference, the low-rank configurations retain approximately 50% of the QK matrix parameters, and the independent low-rank baseline uses a tuned learning rate. Shared-only allocates all low-rank capacity to common input features; the full method assigns part of this capacity to private corrections.

Table R2-5. 350M pre-training from scratch with a common 40K-step horizon and a tuned independent low-rank baseline.

ParameterizationQK matrix parameters retainedFinal validation PPL ↓
Full-rank100%18.0500
Independent-LR (tuned)50%18.1900
Shared-only49.9512%17.7677
QK-coupling50%17.6400

Adding private corrections further reduces PPL to 17.6400, an improvement of 0.1277. These two steps separately establish the value of reusing common input features and retaining side-specific capacity, translating the relational analysis into gains in training from scratch.

We additionally conduct three independent 130M standard-attention pre-training runs per configuration, holding the C4 data, training budget, and 25% QK matrix-parameter budget fixed. Learning rates are selected from {10−4,5×10−4,10−3,5×10−3,10−2}\{10^{-4},5\times10^{-4},10^{-3},5\times10^{-3},10^{-2}\}. Independent low-rank, shared-only, and QK-coupling obtain mean PPL (sample SD) of 22.5760 (0.0977), 22.3279 (0.0944), and 22.0955 (0.0740), respectively. These repeated results agree with the original 350M ablation: common input features are useful to reuse, and independent readouts with private corrections make the limited capacity more effective for language modeling.

Reviewer_oMM6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This submission observes that Query and Key activations in trained LLMs show aligned anchor-relative cross-layer similarity trajectories (multi-anchor QK-coupling, quantified by kappa). It exploits this by replacing independent Q/K projections with a shared low-rank trunk plus lightweight private residual branches.The design is evaluated in three settings:

  1. standard pre-training (60M–1.3B), claiming ~75 % QK-path FLOP and optimizer-state savings with improved perplexity;

  2. MLA-style pre-training (3B-scale), claiming ~50 % projection-FLOP reduction and better final perplexity; and

  3. LoRA-style fine-tuning on large models, claiming equal or better scores with fewer adapter parameters.

Supporting theory links the coupling metric to low-rank relational approximation and score-space recovery.

Critical strengths
  1. Broad, multi-regime empirical coverage (dense, GQA/MoE, MLA, and PEFT) that demonstrates tangible efficiency gains while preserving or improving quality. This is consequential because efficiency claims for a core operator are only credible when shown across pre-training and fine-tuning at multiple scales.

  2. Clean, easy-to-adopt architectural pattern (shared trunk + private residuals) that retains Q/K asymmetry instead of simply tying the projections. This is consequential because it offers a practical, drop-in route to reducing a major cost center without forcing full symmetry.

Critical weaknesses
  1. The central diagnostic κQK\kappa_{QK} is not shown to measure shared routing geometry, nor is it causally linked to the proposed architecture. Similarity of smooth cross-layer trajectories can arise for generic reasons (position, sinks, gauge freedom, batch averaging); the metric is never used to set ranks or placement, and key controls (centering, first-token exclusion, null models, per-token computation) are missing. This is consequential because the paper’s novelty claim rests on the geometry–method connection.

  2. Efficiency and quality claims rest on imperfectly matched baselines and limited statistical support. MLA savings partly reflect changes in latent dimensions and RoPE width rather than coupling alone; several headline pre-training and fine-tuning numbers are single-seed or within sampling noise; wall-clock / peak-memory results are sparse. This is consequential because the practical value of the method is not yet cleanly isolated from confounding factors and optimization effects.

  3. Recent concurrent works such as Kayyam et al. (“Do Transformers Need Three Projections? Systematic Study of QKV Variants”) and Gao & Xu (“Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers”) are not discussed or compared against. A dedicated related-work section in the main text would help situate the proposed multi-anchor QK-coupling approach more clearly within the broader landscape of QKV-efficiency methods.

Questions for authors

Questions for authors:

  1. Does κQK\kappa_{QK} remain high after centering, excluding the first token, computing per-token rather than batch-mean activations, and comparing against random-init / shuffled-layer nulls?

  2. Does a conventional low-rank factorization of [WQ;WK][W_Q; W_K] (or simply reducing dqkd_{qk}) at matched parameter and FLOP budget match or exceed the proposed method?

  3. For the 3B MLA experiment, what is the result against a baseline with matched cache size, latent dimensions, and properly sized query latent?

  4. Can multi-seed results (with confidence intervals) be provided for the main standard-pretraining, MLA, and fine-tuning tables?

14 comments
AuthorsOpenReview ↗

Rebuttal by Authors (part 1)

Response to Reviewer oMM6

Thank you for your detailed review. Below, we address your comments and questions using the original results and the additional experiments.

Response overview

  • Geometric reliability: Centering, prefix exclusion, per-token analysis, and random controls test the coupling observation (R1-1).
  • Structural effectiveness: Error analysis and matched-budget comparisons evaluate independent low-rank, shared-only, and directly narrowed alternatives (R1-1, R1-3).
  • Controlled comparison: Match KV cache, RoPE, and output widths while using a tighter query-latent budget (R1-4).
  • Statistics and efficiency: Report three-seed results, confidence intervals, and measured training benefits (R1-2).
  • Related work: Position the method relative to projection-sharing and Keyless Attention, clarifying the structural distinctions (R1-5).

Statistical convention. For the newly added repeated experiments, we report the mean across three independent random seeds, with the sample standard deviation (SD) in parentheses. SD is measured in PPL units for perplexity and in percentage points for percentage-valued task scores. We use s=∑i(xi−xˉ)2/(n−1)s=\sqrt{\sum_i(x_i-\bar x)^2/(n-1)} with n=3n=3; results cited from the original submission retain their original statistical definitions.

AuthorsOpenReview ↗

Rebuttal by Authors (part 2)

R1-1. From shared Q/K relationships to a directly tested architecture (W1, Q1)

Thank you for focusing on the basis of the design. We start from the multi-anchor Q/K relationship, use shared-reconstruction error analysis to motivate a shared–private computation, and test it directly through attention reconstruction and full training. The statistical controls you suggest examine the origin of the observation; matched-budget attention reconstruction and full training then assess the resulting design.

(1) From multi-anchor relationships to shared computation.

Self-attention uses Q/K scores to determine where a token reads information. By “routing geometry,” we mean the organization of the representations that form these matching relationships. Equations (7)–(8) compare the cross-layer similarity trajectories of Q and K relative to the same anchor, rather than requiring Q and K within a layer to be identical. A high κa\kappa_a therefore means that Q and K organize their relationships across network depth similarly when referenced to layer aa. Observing this pattern across multiple anchors motivates applying within-layer reuse of common Q/K features at multiple depths.

For the normalized relational matrices CQ,CKC_Q,C_K, define M=(CQ+CK)/2M=(C_Q+C_K)/2 and D=(CQ−CK)/2D=(C_Q-C_K)/2. Theorem 1 in the original submission bounds the mean squared error of reconstructing both trajectory families in the same pp-dimensional relational space: Rp=τp+∥(I−Pp)D∥F2/L≤τp+(1−κˉ)/2R_p=\tau_p+\|(I-P_p)D\|_F^2/L\leq\tau_p+(1-\bar\kappa)/2. Here, PpP_p retains the leading pp singular directions of the common component MM, and τp\tau_p is the normalized energy in its discarded directions.

This result explains the role of coupling in joint low-rank modeling: the common spectrum determines the dimensionality needed to represent shared information, while coupling quantifies the relational differences that remain beyond it. At a fixed common spectral tail, stronger coupling yields a smaller upper bound on joint reconstruction error. This motivates representing the common component once and supplementing it with side-specific corrections.

Write the two unit-normalized trajectories for anchor aa as cQ(a)c_Q^{(a)} and cK(a)c_K^{(a)}, and let ma=(cQ(a)+cK(a))/2m_a=(c_Q^{(a)}+c_K^{(a)})/2 and da=(cQ(a)−cK(a))/2d_a=(c_Q^{(a)}-c_K^{(a)})/2. Then ∥ma∥22=(1+κa)/2\|m_a\|_2^2=(1+\kappa_a)/2 and ∥da∥22=(1−κa)/2\|d_a\|_2^2=(1-\kappa_a)/2. The two terms sum to 1: they are the common/differential energy decomposition of the original relational representation, rather than an additional fitted metric. Along the training trajectory in the paper, κˉ\bar\kappa rises from 0.496 to 0.952, corresponding to a reduction in differential energy from 0.252 to 0.024, approximately 90.48%. Stronger coupling during training thus corresponds to less relational variation requiring separate representation. Together with the common spectral tail in Theorem 1, this supports sharing common features and using private branches to supplement their differences.

Equation (10) implements this idea as Z=XAS, QS=ZBQ, KS=ZBKZ=XA^S,\ Q^S=ZB_Q,\ K^S=ZB_K. Proposition 1 shows that, for input second moment Σ\Sigma, the optimal joint output-reconstruction error is ∑j>rsσj2(Σ1/2[WQ  WK])\sum_{j>r_s}\sigma_j^2(\Sigma^{1/2}[W_Q\;W_K]). Common input features are therefore computed once and used through separate Q/K readouts; the information retained at a given channel capacity is characterized by the joint spectrum weighted by the actual input distribution. The relational analysis motivates sharing, while the within-layer parameterization provides a trainable implementation.

AuthorsOpenReview ↗

Rebuttal by Authors (part 3)

(2) Centering, per-token computation, and random controls support the structural observation.

We extend the analysis to public pre-trained checkpoints of SmolLM-135M, Qwen2.5-3B, Llama-3-8B, and Qwen3-30B-A3B. All controls use the same 100 sampled text batches, each containing eight 4,096-token sequences. Position centering removes each channel's mean over positions. The per-token analysis first constructs a relational trajectory for each example and token position, and then aggregates the results.

Table R1-1. Q/K coupling κ\kappa under different statistical treatments and random controls.

ModelControl-protocol κ\kappaPosition-centeredPer-tokenRandom init.Shuffled K-layer correspondence
SmolLM-135M0.814440.359040.588290.016100.01870
Qwen2.5-3B0.431870.410630.259850.02455−0.00181
Llama-3-8B0.802430.525400.493440.032040.01100
Qwen3-30B-A3B0.843450.818150.781080.027380.01930

All four models retain positive coordination after centering and under per-token computation, showing that the structure is present beyond the mean component and within individual tokens. Random initialization tests whether the architecture alone produces the relationship and yields near-zero scores. Layer shuffling preserves the trained activations while breaking the depth correspondence between Q and K, and also yields near-zero scores. These controls indicate that both trained representations and their cross-layer correspondence contribute to the observed coupling.

To examine prefix and attention-sink sensitivity, we additionally exclude the first token and the first 16 tokens, reporting the results before and after exclusion.

Table R1-2. Q/K coupling κ\kappa after prefix exclusion.

ModelControl-protocol κ\kappaExclude first tokenExclude first 16 tokens
SmolLM-135M0.814440.814420.81441
Qwen2.5-3B0.431870.431840.43185
Llama-3-8B0.802430.802420.80228
Qwen3-30B-A3B0.843450.843360.84334

Excluding the first 16 tokens changes the reported scores by at most approximately 0.00015, compared with a trained–shuffled gap of approximately 0.43–0.82. The prefix effect is much smaller than the separation between the trained models and the random controls, supporting a relationship that is stable under prefix exclusion. Together with centering, per-token computation, and random controls, this supports the presence of coordination broadly across content positions rather than its concentration in a few prefix positions.

(3) Matched-budget functional experiments test whether the shared features are useful.

We hold fixed the actual hidden states produced by each pre-trained checkpoint, fit alternative low-rank Q/K maps on separate calibration text, and evaluate them on disjoint text. The methods use the same QK matrix-parameter budget and retain the native output widths, positional encoding, and causal attention. We fit the Q/K maps and use KL divergence to evaluate the resulting attention distributions.

Specifically, let AtA_t denote the original model's attention probabilities over visible keys at query position tt, and let A^t\widehat A_t denote the probabilities after replacing the Q/K maps. We evaluate DKL(At∥A^t)D_{\mathrm{KL}}(A_t\Vert\widehat A_t) and aggregate over positions, heads, and examples. Lower KL means that the compressed model more faithfully preserves where the original model reads information and the weights assigned to those locations. Comparing against independent low-rank maps tests the value of sharing; comparing shared-only and shared–private maps tests the value of retaining side-specific directions.

Each of the two calibration settings uses 800 sequences and 4,096 sampled calibration tokens, with evaluation on another 100 batches. We cover all layers of SmolLM, Qwen2.5, and Llama. For the 48-layer Qwen3 model, we preselect layers 1, 5, 10, 14, 18, 22, 27, 31, 35, 39, 44, and 48 across depth, all of which are full-context GQA layers.

Table R1-3. Attention KL on a causal submatrix of 256 common sampled positions, retaining 25% of QK matrix parameters (lower is better).

ModelIndependent low-rankShared-onlyQ-privateK-privateJoint-private
SmolLM-135M0.0993350.0981500.0942240.0986340.091870
Qwen2.5-3B0.0556630.0522970.0415440.0462600.061981
Llama-3-8B0.0437070.0445270.0434420.0441810.049122
Qwen3-30B-A3B0.0643330.0647590.0518020.0539440.076005

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response to All Reviewers

We thank the reviewers. This single response addresses their shared and individual concerns. New results are blue-highlighted in the revised manuscript, with locations below.

1. Geometric reliability and coordinate controls (oMM6 W1/Q1; tcTR W1/Q1–Q2; pe1j W1/Q1).

Across four models, our 100-batch protocol gives trained κ=0.432\kappa=0.432–0.8430.843, versus 0.0160.016–0.0320.032 at random initialization. Coupling remains positive after centering and per-token computation; excluding 16 prefix tokens changes it by at most 0.000150.00015, and shuffled K-layer correspondence gives near-zero values. Pre-/post-RoPE scores differ by at most 2.6×10−52.6\times10^{-5} under the common orthogonal rotation. Linear CKA gives high alignment in trained and random models; original κ\kappa retains native directional information. Fixed random maps across layers can produce high κ\kappa on one residual stream; independent layer-wise maps do not. Locations: Sec. 4; App. C.1.3–C.1.5, C.1.7 (Tables 8–12, 15).

2. Mechanism and matched-budget alternatives (oMM6 W1/Q2; tcTR W2/Q4; pe1j W1/W3).

Theorem 1 bounds cross-layer relational reconstruction by Rp≤τp+(1−κˉ)/2\mathcal R_p\leq\tau_p+(1-\bar\kappa)/2; Proposition 1 separately characterizes within-layer channel approximation through the spectral tail of Σ1/2[WQ  WK]\Sigma^{1/2}[W_Q\;W_K]. They motivate shared input, separate Q/K readouts, and private corrections; Theorem 3 identifies complementary score directions. At equal 25% QK budget, Qwen2.5-3B attention KL improves 0.055663→0.041544 versus independent low-rank maps. At 50% budget, 350M PPL is 17.6400, versus 17.7677 (shared-only) and 18.1900 (tuned independent low-rank). Controls include output narrowing, QK-norm, per-head scaling, and repeated placement runs: Q-private is favored in standard attention, joint correction in compressed-key MLA. Locations: Sec. 4.3–5.2; App. C.1.6, C.2.3–C.2.5, C.4.2, C.6.2–C.6.3; App. D.1/D.3.

3. MLA fairness and cache accounting (oMM6 W2/Q3; tcTR W3/Q3; pe1j W2/Q3).

We separate original SmolLM content/positional cache savings. New 3B/100.663B-token runs match 576 KV-cache elements/token/layer, 64 RoPE dimensions, and 2048-dimensional Q/K content outputs. At equal cache, conservative QK-coupling improves PPL 9.829509→9.803333, saving 52.84% of QK content-path projection cost. Baseline MLA uses a 1536-dimensional query latent; our 384 shared + 80 query-private features impose a tighter Q-side budget (not an equal latent). Aggressive QK-coupling reduces cache 576→480 (16.67%), saves 64.43% of projection cost, and improves PPL to 9.819822. Three-seed 200B-token runs improve PPL 10.5549±0.0095→10.5148±0.0078. Locations: App. C.3, C.4.1/C.4.4, C.6.1 (Tables 22–24, 27–28, 38).

4. Downstream capability and precise retrieval (tcTR W4/Q5; oMM6 W2; pe1j W2).

We add six downstream tasks at 200B tokens, seven at 100B, and all 13 RULER4K tasks using a common 4,096-token window. At equal cache, conservative QK-coupling improves eight-task NIAH 70.18→75.38 and overall RULER 58.48→63.01. With 16.67% less cache, the aggressive variant still exceeds baseline on both (70.93 NIAH; 58.84 RULER), and its seven-task mean improves 32.87→32.99. Full-4K attention reconstruction also improves Llama KL 0.034564→0.027308. These results test token addressing beyond PPL. Locations: App. C.1.6 and C.4.3–C.4.6 (Tables 14, 26, 28, 30).

5. Statistics and real training efficiency (oMM6 W2/Q4; tcTR W3; pe1j W2/W3).

We add three-seed means, sample SDs, and 90% confidence intervals for key pre-training, placement, and fine-tuning results. Matched-rank repeats at 60M/350M support Q-private placement; the 3B/200B conservative gains are positive across all three runs. The 75% saving is QK-projection-specific. Measured 3B throughput improves 4.89% and peak GPU memory falls 4.79%; at 1.3B, step time falls 1.62% and time-to-target-quality 27.70% through fewer steps and lower step cost. Locations: App. C.2.3, C.4.2, C.6.1/C.6.4 (Tables 18, 25, 38, 42–43).

6. Adaptation baselines and related work (oMM6 W2–W3; pe1j Q2).

We add matched-budget MFA, ThinKeys, and shared-only baselines on Qwen3.5-35B-A3B, extend R2 to R8, and report three-seed statistics with 90% intervals. QK-coupling leads four of five R2 and all five R8 task means, using 0.61M QK-adapter parameters versus 0.84M for LoRA/ReLoRA/ThinKeys at R2. Main-text Related Work covers QKV variants and Keyless Attention, clarifying the shared–private distinction. Locations: Sec. 2, 5.1; App. C.5.2 (Tables 33–37), D.1.

Together, the evidence links observation, mechanism, functional performance, and efficiency. We sincerely appreciate the reviewers’ constructive feedback and welcome any further questions or concerns. We will promptly provide additional clarification as needed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.