acceptodds
Under review as a conference paper at ICLR 2027

Low-Rank Key Value Attention

Abstract

The key-value (KV) cache is a primary memory bottleneck in Transformers. We propose Low-Rank Key-Value (LRKV) attention, which reduces KV-cache memory by exploiting redundancy across attention heads. Each layer pairs a shared KV projection with low-rank, head-specific residuals, interpolating continuously between full sharing and per-head independence. Across pretrained models from 128M to 6.3B parameters, LRKV has the lowest observed held-out loss among standard MHA, MQA/GQA, and MLA at each method's operating point while using 48–67% of MHA's cache, and it remains below full-cache MHA with as little as 8.3% of the cache at 128M, 2.5B, and 6.3B. At 128M and 100B tokens, three-seed means give LRKV lower held-out loss than MLA at every matched cache budget (8.3%, 25%, and 50%). At 2.5B, LRKV reaches each baseline's final quality in 18–30% fewer optimizer steps. Trained at 8K context, 512M LRKV models attain the lowest long-context loss and lead every baseline on RULER at 32K, four times the training length. Its exact decode kernel evaluates the factored representation end to end without storing reconstructed per-head K/V. On a B200 at 2.5B, LRKV serves 8.5–8.9 MHA's resident tokens at every context and decodes at 103–111% of MHA's batch-1 throughput from 1.2B to 30B. After fixed-recipe midtraining, LRKV has the highest observed combined downstream accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
low-rank key value attention, attention mechanism, KV cache reduction, language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_En5JOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

LRKV splits the difference. Each head's K/V projection becomes a shared full-rank matrix plus a small head-specific rank-r correction: W_h = W_shared + U_h B_hᵀ. The cache then holds one shared K/V state per layer plus a rank-r latent per head, costing 1/H + r/d_h of what MHA costs. They pretrain from scratch at 128M, 1.2B, 2.5B and 6.3B on FineWeb-Edu against MHA, two GQA configs, MQA and MLA, with the learning rate grid-searched separately for every architecture.

Critical strengths

1- The problem of KV-cache compression for Transformer-based LLMs is practical. 2- Including the models at different scales in the experiment is an upside. The learning rate is searched per scale and per attention type.

Critical weaknesses

1- The solution for KV cache compression is heavier than many baselines that do upcycling instead of pre-training (e.g. X-EcoMLA) or the methods avoid any training (e.g. PALU).
2- I am not convinced that the experimental setting of the paper is practical and the results of the experiments conclusive. Pre-training a B parameter model with 50B token is undertrained and comparing their results might not be conclusive.

3- Result flags: Table 1 at 6.3B reads: MQA gets 2.304 CE using 3% of MHA's cache, while LRKV's default gets 2.288 using 53.1%. That's 0.016 CE, or 0.005 BPB, bought with 17.7x more memory. LRKV's loss advantage over MQA shrinks at every scale: 0.067 at 128M, 0.064 at 1.2B, 0.059 at 2.5B, 0.016 at 6.3B.

4- Comparing models only against CE loss will not necessarily reflect the task behavior of the models. Table 2 reporting the GSM8K results of 0-10% are signaling the performance of the model (clear evidence of undertrained model).

Missing baselines: X-EcoMLA, PALU and ....

Questions for authors

1- At 6.3B, MQA reaches 2.304 CE with 3% cache against your 2.288 with 53.1%, and the gap has narrowed at every scale you tested. Do you expect it to keep narrowing? If your answer is that loss converges but downstream doesn't (Table 2: MQA 33.4 vs LRKV 40.4, mostly HumanEval), please make that argument directly and support it. It's the strongest version of your case and right now it's a parenthetical.

2- Is one extra seed at 2.5B and 6.3B feasible for the rebuttal, even just for LRKV and MHA?

3- Which configuration does each abstract number come from? "8.5–8.9× MHA's resident tokens" is r=8 at 8.3% cache (l.363–365, Table 5: 566 vs 66 at 8K). The loss-leadership claim is r=64 at 48–67% cache, whose measured capacity is 118 vs 66, so 1.8×, and which OOMs at batch 128/8K and batch 32/32K where MQA, GQA and LRKV r=8 all run. The conclusion's "reduces the KV cache from 20.29 to 2.40 GB and peak memory from 27.19 to 9.20 GB" is also r=8. Tagging each number with its rank would fix this.

1 comment
AuthorsOpenReview ↗

Replication, training scope, and practical trade-offs

Thank you for asking whether LRKV is a practical architectural choice. The revision adds a second 2.5B seed, an upcycling experiment, and matched-tier evaluations. Each abstract number now identifies its configuration.

1. Training budget and the extra seed (Q2).

The common-protocol 2.5B comparison now has two LRKV/MHA seeds. Their BPB values are 0.7191/0.7230 and 0.7195/0.7222, giving LRKV-minus-MHA gaps of -0.0039 and -0.0027 (Table 3). LRKV r=64 uses 55.56% of MHA's cache. This replicates a quality improvement at that operating point and training budget.

We agree that 50B-token training cannot establish the eventual large-model ordering at convergence. An equal-horizon comparison still tests the architecture under a fixed training budget. The 128M, 100B-token three-seed comparison and separately tuned 512M/8K suite complement that evidence (Tables 1 and 2). Longer large-model training remains open; we report no additional 6.3B seed.

2. The 6.3B/MQA question (Q1).

Our audit found unequal horizons among the archived 6.3B runs and derived rather than directly measured CE entries. We withdraw the controlled ordering and the cross-scale narrowing trend based on them. We therefore do not predict whether the gap will continue to shrink or use the archived downstream scores to claim that a persistent task gap compensates for a converging loss gap.

The revised claim is a range of operating points. For example, in E0, 2.5B MQA uses 5.56% cache and reaches 0.7290 BPB, while LRKV r=64 uses 55.56% and reaches 0.7191. That is a meaningful quality-memory choice, not universal dominance over MQA. The controlled case against other compression methods comes from the matched-budget experiments, not this unequal-budget pair.

3. Upcycling and post-training alternatives.

Starting from an existing checkpoint can be much cheaper than pretraining. We test this route by converting a trained 2.5B MHA model to r=64 using SVD, then adapting for 2B tokens (Table 12). The MHA source is 0.7226 BPB, conversion initially gives approximately 0.78, and adaptation recovers to 0.7290 at 55.56% cache. The from-scratch LRKV reference is 0.7187 in this experiment.

This demonstrates an adaptation route and quantifies its remaining cost in quality: 0.0064 BPB above the source MHA and 0.0103 above the LRKV reference. It is not a direct Palu/X-EcoMLA benchmark or a matched-total-cost comparison. We now discuss both as practical alternatives. LRKV's contribution is the additive parameterization and its measured trade-off; superiority over existing upcycling methods is not part of the acceptance case.

4. Task behavior and low absolute performance.

The new nominal-50% checkpoints receive identical midtraining and five-task evaluation: LRKV scores 19.4, MLA-DS 18.9, GQA-3 18.6, and MHA 19.1 (Table 10). This tests task behavior at the relevant budget. Low absolute performance and sub-one-point gaps limit practical claims; the scores do not establish significant downstream gains or deployment-ready reasoning.

5. Abstract numbers identify their configurations (Q3).

The revised abstract identifies the operating point for every numerical claim:

  • Quality: 128M LRKV r=43, 50.26% cache, 0.8931 BPB versus MHA's 0.8920.
  • Throughput: 2.5B full-width r=64, 55.56% cache, batch 8/32K; 420 versus 250 tok/s for MHA (1.68x).
  • Capacity: 2.5B full-width r=8, 11.81% cache, 32K; maximum batch 145 versus 17 for MHA (8.53x). This is distinct from the reduced-width 8.33% quality configuration.

The new optimized comparison also shows rank-8 LRKV at 1250 tok/s versus FlashInfer MLA-DS's 1300 at batch 32/32K, and the batch-1 slowdown is explicit. We removed the untraceable absolute-memory numbers and unsupported training-step savings. Rank-64 batch 32 succeeds at 32K; the conflicting maximum of 31 is replaced with a lower bound of 32, without inventing a corrected maximum.

The additional seed, conversion experiment, and matched-budget results support a useful architectural contribution within the stated scope. We would appreciate reconsideration of the recommendation. If longer training or a specific comparison remains essential to acceptance, please identify that remaining requirement.

Reviewer_29oXOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Summary

The paper proposes LRKV, an attention variant that reduces KV-cache memory. Each head's K/V projection is a shared full-rank matrix plus a head-specific rank-rr residual. The cache stores the shared K/V once per layer, plus per-head rank-rr latents.

Method

  • Wh=Wshared+UhBh⊤W_h = W_{\text{shared}} + U_h B_h^\top, with r≪dhr \ll d_h, trained from scratch.
  • r=0r = 0 gives MQA, and r=dhr = d_h gives the same function class as MHA.
  • RoPE stays exact, and decoding does not materialize per-head K/V.

Results

(Comparison protocol)

  • Cache-matched. Same cached elements per token per layer, under one accounting rule.
  • Function-class matched. Same set of attention functions, different parameterization. LRKV with ds=r=dhd_s = r = d_h and MHA form such a pair.

(R.1) Pretraining loss. LRKV beats cache-matched MLA at every tier and scale (e.g., −0.011-0.011 CE at 50% cache, 128M; Table 1). At 128M, the function-class-matched LRKV (r=dhr = d_h) beats MHA by 0.044 CE (Table 13).

(R.2) Long context. 512M models are trained from scratch at 8K. LRKV (48.2% cache) has the lowest loss at 8K and the best RULER score at 32K. No cache-matched or function-class-matched reference is given.

(R.3) Cost. At 2.5B, LRKV (r=8r = 8) holds 2.8×2.8\times the batch of GQA-6 on one B200, at similar loss. A cache-matched GQA is not benchmarked.

(R.4) Downstream. After midtraining, LRKV has the best combined score at all three scales (+2.6 over MHA at 2.5B). No comparison is cache-matched.

(R.5) Analysis. LRKV's head diversity is close to MHA's (93.5% vs. 94.0% at 2.5B). No comparison is cache-matched, and the function-class-matched pair (r=dhr = d_h vs. MHA) is not analyzed.

Critical strengths

S1. The idea is simple. The rank rr interpolates between MQA and MHA, RoPE stays exact, and decoding is exact.

S2. LRKV beats MLA at matched cache across all four scales, with three seeds at 128M and 1.2B.

S3. The cost side is tested end-to-end. Measured capacity matches the analytic cache formula.

Critical weaknesses

The paper shows that LRKV beats cache-matched MLA in pretraining loss. However, the comparison setting changes across experiments, and the gain over MHA is not explained by the paper’s motivation. Key baselines and decoding costs are also not compared at matched settings. As a result, it is unclear whether LRKV improves the memory–quality trade-off beyond pretraining loss (W1, W3, W4), and where its gain comes from (W2).

W1. The comparison setting is inconsistent across experiments, so the main claim is tested only on pretraining loss.

  • Cache-matched evidence exists only for pretraining loss (Table 1). No cache-matched MLA is trained for long context, and the cache-matched checkpoints are not evaluated on downstream tasks.
  • These evaluations compare mainly with MHA, while MLA stays at ~8% cache and LRKV uses 48–67%. Beating MHA shows that LRKV loses little from compression, not that it beats other compression methods at the same budget.

W2. The gain over MHA may come from the parameterization, not from cache reduction.

  • At 128M, LRKV with r=dhr = d_h has the same function class as MHA, yet it is 0.044 CE lower (Table 13). So the gain over MHA appears even without any cache reduction.
  • The paper's motivation, to reduce KV memory "by exploiting cross-head redundancy without collapsing useful head specialization" (Sec. 1), does not explain this gap. In [1], compressed attention that beats MHA moves the saved parameters to the FFN. LRKV with r=dhr = d_h keeps the same FFN and even has more K/V parameters than MHA, so this explanation does not apply.
  • Part of the reported gain may therefore come from optimization, e.g., Muon on factored weights or initialization, rather than from sharing across heads. A function-class-matched reference at each scale would isolate the source.

W3. Key baselines are not matched or carefully configured.

  • MLA's quality depends on its configuration. It beats MHA in [1, 2] but has higher validation loss in [3]. This paper uses a non-standard MLA (partial RoPE on the up-projected key, unlike the decoupled RoPE key of [2]) and does not describe its initialization or any configuration search beyond the learning rate.
  • GQA is near-matched only for pretraining loss, not in the long-context, downstream, or cost results. MLRA [1] and TPA [3] target the same goal but are not compared.

W4. The decoding cost is not compared at a matched cache size.

  • No baseline is benchmarked at LRKV's cache size, so throughput mixes the effect of cache size with that of the kernel. It is unclear how much overhead the decomposition adds, and how LRKV's kernel compares with optimized kernels for other variants.

References

[1] Liu et al. Multi-Head Low-Rank Attention. ICLR, 2026.

[2] DeepSeek-AI. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv:2405.04434, 2024.

[3] Zhang et al. Tensor Product Attention Is All You Need. NeurIPS, 2025.

Questions for authors

Q1. 8K experiment. At 8K, MHA has the highest loss, even above MQA at 8.3% cache (Table 14). At 2K, MQA is the worst at 128M–2.5B (Table 1). Was the learning rate searched separately for the 512M 8K setting? How do the authors explain this difference?

Q2. Reduced shared width. For the lowest tier (ds<dhd_s < d_h), how does a dsd_s-dimensional shared feature form a dhd_h-dimensional key, and how is RoPE applied to it?

3 comments
AuthorsOpenReview ↗

(1/2): Matched-budget quality and missing baselines

Your main question is whether LRKV improves the memory-quality trade-off beyond the original pretraining comparison, and whether an optimization effect is being mistaken for a compression benefit. We have added experiments for both questions. The strongest new result is a lower loss at exactly the same cache size in the separately tuned 8K study, alongside the expanded 100B-token matched-tier suite.

W1: The comparison now extends beyond the original loss table.

At 128M and 100B tokens, three-seed means give LRKV 0.9080 BPB versus latent-only MLA's 0.9120 and MLA-DS's 0.9135 at exactly 8.33% cache. At the nominal-25% tier, LRKV reaches 0.8958 versus 0.8996 for MLA-DS and 0.9004 for latent-only MLA; the actual fractions are 25.26% versus 25.00%. At the nominal-50% tier, LRKV reaches 0.8931 versus MLA-DS's 0.8955 and GQA-3's 0.8968, at 50.26% versus 50.00% cache (Table 1). Full-cache MHA is 0.8920, so the latter LRKV point retains most of MHA's loss performance with approximately half its cache.

At 512M, 8K training context, and 50B tokens, LRKV r=32 and GQA-4 both use exactly 33.33% cache and reach 0.9640 and 0.9655 BPB. LRKV r=21 reaches 0.9672 at 24.74%, versus MLA-DS's 0.9690 at 25.00% (Table 2). The replacement retrieval probe also includes the compressed baselines: accuracies are 0.85 versus 0.80 for r=32/GQA-4 and 0.83 versus 0.78 for r=21/MLA-DS. These probe differences are descriptive. Against MHA's 0.86, r=32 has reported McNemar p=0.4; this is not an equivalence test. The old 32K RULER claim is withdrawn.

The nominal-50% checkpoints now receive identical midtraining, approximately 408M tokens, followed by five-task evaluation. Combined scores are LRKV 19.4, MLA-DS 18.9, GQA-3 18.6, and MHA 19.1 (Table 10). These add task-level evidence at the relevant budget, while their sub-one-point margins warrant a restrained interpretation. The main evidence for the trade-off is the controlled loss comparison.

W3: Standard-style MLA and additional factorized baselines.

The new MLA-DS configurations use a separately cached shared RoPE key: 64+64, 320+64, and 704+64 content-plus-RoPE widths at the three 128M tiers. Both components are counted. Thus the 8.33% and nominal-25% ordering holds for both MLA forms tested. Appendix A distinguishes their positional treatments and corrects the unsupported universal initialization/LR-search descriptions. We do not claim an exhaustive MLA configuration search.

The four-seed, 5B-token diagnostic adds TPA and MLRA at the nominal-25% target: LRKV-minus-TPA is -0.005 +/- 0.002 CE and LRKV-minus-MLRA is -0.004 +/- 0.002 CE, with LRKV lower on all four paired seeds in each case (Table 9). These are short-horizon results; the reported dispersions are not treated as confidence intervals, and exact baseline factor shapes are still needed to verify exact byte matching.

These comparisons address W1 and strengthen the baseline evidence for W3. Our companion comment, "29oX (2/2): Controls, decoding, tuning, and reduced width," addresses W2, W4, Q1, and Q2.

AuthorsOpenReview ↗

(2/2): Controls, decoding, tuning, and reduced width

This reply covers W2, W4, Q1, and Q2. The matched-budget quality and additional-baseline results are in "29oX (1/2): Matched-budget quality and missing baselines."

W2: The revised claim separates compression performance from its cause.

At 128M, full-rank LRKV r=128 gives 0.8912 BPB versus MHA's 0.8920; MHA with K/V initialization scaled by 1/sqrt(2) gives 0.8923 (Table 8). At 512M the full-rank control gives 0.9605 versus 0.9610. Full-rank LRKV uses more cache than MHA: 116.67% and 108.33%, respectively. These controls replace the original 0.044 CE argument.

We agree that they do not causally isolate sharing from optimizer-factorization interactions. The empirical claim is that this trainable parameterization delivers the measured quality at the stated cache budget. That claim can be evaluated directly even while its optimization mechanism remains open. We have removed causal interpretations of head geometry; full-rank controls at the largest scales and crossed optimizer controls remain additional work.

W4: Optimized implementations and explicit cache fractions.

At batch 32/32K, LRKV r=8 reaches 1250 tok/s versus FlashInfer MLA-DS's 1300 and GQA-2's 1280. Rank-64 LRKV reaches 520 versus its materialized path's 310 (Table 4). The GQA comparisons are labeled operating points: 11.11% versus 11.81% cache, and 50.00% versus 55.56%. Exact matched-cache systems comparisons remain a limitation; quality comparisons are reported separately.

Q1: The original 8K ordering does not survive the revised tuning.

The new suite separately tunes the compared families. Final MHA BPB is 0.9610 versus MQA's 0.9760. We withdraw the original anomalous ordering and the unsupported universal search statement. Appendix A.2 distinguishes intermediate sweep values from final 50B-token endpoints.

Q2: The reduced shared feature has an explicit learned expansion.

Z=XAZ=XA is 16-dimensional, and a learned 16-by-128 matrix C gives Kh=ZC+RhBh⊤K_h=ZC+R_hB_h^\top. The cache retains Z and residual latents. Full 128-dimensional RoPE is applied after expansion and summation, within the decode computation (Eqs. 7-8). The cache fraction is (16+6×8)/(6×128)=8.33%(16+6\times8)/(6\times128)=8.33\%. MQA at this shape is 16.67%, and is labeled accordingly.

Taken together, the new results extend the trade-off evidence to long-context training, introduce the requested standard-style MLA and parameterization controls, and provide a clear answer to the reduced-width mechanism. We would appreciate reconsideration of the recommendation on that revised empirical claim. If a concern still blocks acceptance, identifying it would help us address the remaining point directly.

Thank you for the detailed reply and the many new experiments within the short rebuttal period.

W1 & W3 & W4 Thank you for the matched experiments with DeepSeek-style MLA, the additional TPA and MLRA baselines, and the optimized serving comparisons. LRKV now stays slightly better than MLA-DS at matched cache. However, the gap shrinks from 128M to 512M (0.0038 → 0.0018 BPB at ~25%), so it is unclear whether it holds at larger scales.

I also could not find a serving advantage at matched cache. In the rebuttal, LRKV is slightly slower than both GQA-2 at near-matched cache and FlashInfer MLA-DS (1250 vs. 1280 and 1300 tok/s), and the MLA-DS cache fraction is not stated. The gains in Figure 5 and Tables 4–5 come from an unoptimized MLA implementation or from the launch-bound batch-1 setting, so they do not compare the designs. If quality and throughput are both close to MLA, the reason to choose LRKV is unclear to me. If I missed a setting where LRKV is faster at matched cache, please point me to it.

W2 Thank you for the full-rank and initialization controls. With the new numbers, full-rank LRKV is close to MHA (0.8912 vs. 0.8920 BPB at 128M), so the large gain at the same function class no longer appears. The revised claim, quality at a stated cache budget without a causal mechanism, is consistent with the evidence, and this resolves my concern. One point remains unclear. The same comparison gave a much larger gap in the original Table 13 (0.860 vs. 0.878 BPB), and the rebuttal does not explain the change. Could the authors state what caused it (e.g., re-training or retuning)?

Reviewer_Rs98OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Transformer architecture with standard SDPA attention is well known to be memory hungry, especially for the task that requires large contexts. This paper proposes to reduce the KV cache size by 1) sharing projection for all the heads 2) have a low rank component for every head. During decoding, similar to MLA attetnion in Deepseek LRKV does not materialise K and V matrices. Also LRKV compared to MLA does not require RoPE decoupling. Proposed approach is compared to MLA, MQA, GQA, and baseline MHA on pertaining 128M/1.2B/2.5B/6.3B model sizes.

Critical strengths

The method is simple, to its credit. The results are promising: the proposed approach not only reduces KV-cache memory during decoding compared with standard MHA, but also reaches comparable quality in fewer training steps, with improvements observed even at the 6.3B model scale.

Critical weaknesses

The fairness of the MLA efficiency comparison is a concern. As the authors acknowledge, their MLA baseline does not use a specialized kernel, whereas LRKV uses an optimized implementation. Figure 5 reports lower throughput for MLA than for MHA, raising questions about how much the comparison reflects differences in implementation quality. This is particularly concerning because MLA is a direct baseline for the proposed approach. Furthermore, a better MLA implementation could reduce temporary memory allocations and increase the measured maximum batch size without changing MLA’s cache size. A comparison against an optimized MLA implementation is therefore needed to substantiate both the throughput and serving-capacity advantages over MLA.

Questions for authors
  1. Could the authors repeat the MLA throughput comparison using an optimized fused MLA kernel under the same hardware, precision, batch sizes, and context lengths?
2 comments
AuthorsOpenReview ↗

Optimized MLA comparison and serving capacity

Thanks for identifying about the MLA implementation, we have added the optimized baseline you requested and revised the interpretation accordingly. The new result is that LRKV is competitive with fused MLA at the tested operating point, while still providing useful throughput and capacity relative to MHA.

Optimized MLA comparison (Section 4.5, Table 4).

For the 2.5B shape on one B200 at batch 32 and 32K context, FlashInfer MLA-DS achieves 1300 tok/s and full-width LRKV r=8 achieves 1250 tok/s: LRKV is 3.8% slower at this model size (we expect at larger model sizes this % reduces). GQA-2 reaches 1280 tok/s, the absorbed MLA-DS path 900, and MHA is OOM. This directly changes the inference from the original unfused baseline: we make no throughput-superiority claim over optimized MLA.

At batch 8/32K, where MHA fits, full-width LRKV r=64 reaches 420 versus MHA's 250 tok/s, a 1.68x ratio at 55.56% of MHA's cache. At batch 32/32K, the rank-64 factored path reaches 520 versus 310 tok/s for its materialized implementation. These comparisons show where the implementation is useful. The reported batch-1 LRKV throughput is 79-80% of MHA's, so the speedup is explicitly limited to the measured larger-batch regime.

Serving capacity with the optimized reference (Table 14).

At 32K, the reported maximum batches are 145 for rank-8 LRKV, 136 for optimized MLA-DS, and 17 for MHA. At 128K they are 36, 34, and 4. LRKV therefore supports approximately 6.6% and 5.9% more sequences than that optimized MLA implementation. We agree that allocation and prefill transients affect these boundaries; the new measurements replace the old implementation-dependent MLA capacity comparison.

Exactness and fairness boundaries.

Cached decoding is checked against each method's own dense model, with reported greedy agreement 1.000 and maximum logit error approximately 2e-2 in bfloat16 and 1e-4 in fp32. This supports numerical agreement on the tested cases; algebraic exactness does not imply bitwise equality.

These are operating-point comparisons. GQA-2 uses 11.11% cache versus LRKV r=8's 11.81%; GQA-9 uses 50.00% versus r=64's 55.56%. We do not label the systems MLA comparison an exact cache match. Appendix D.2 also identifies the backend, timing-precision, and cache-layout disclosures still needed for full reproduction.

The revision thus replaces the decisive unfused baseline, reports the outcome even where MLA is faster, and retains the supported larger-batch benefits. We hope this strengthens your confidence in the contribution and addresses the central implementation concern you raised.

Thank you for addressing my comments. I will maintain my score since I am leaning towards acceptance and I believe the paper's empirical findings are interesting and should be presented at ICLR.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision overview: New evidence and corrected claims

Thanks to all reviewers who took the time to go through the paper and provide feedback. We believe we have addressed the main concerns of that have been expressed through new experiments. Specifically, the central case for LRKV as an additive shared-plus-residual attention architecture can improve quality at a fixed or nearly fixed KV-cache budget, while retaining an implementable full-dimensional RoPE decode path. Thank you for identifying the comparisons needed to establish that case more clearly. The revision adds the following evidence.

1. Quality against compression baselines, with actual cache budgets reported.

At 128M and 100B tokens, three-seed means favor LRKV over both latent-only MLA and DeepSeek-style decoupled-RoPE MLA at the tested 8.33% and nominal-25% tiers. At the nominal-50% tier, LRKV reaches 0.8931 BPB versus 0.8955 for MLA-DS and 0.8968 for GQA-3; full-cache MHA is 0.8920. The LRKV point uses 50.26% cache. These results put the quality benefit at a defined compression budget (Table 1).

2. Evidence beyond the original setting.

At 512M, 8K training context, and 50B tokens, LRKV r=32 reaches 0.9640 BPB versus GQA-4's 0.9655 at exactly 33.33% cache. At a second tier, LRKV uses slightly less cache than MLA-DS and reaches 0.9672 versus 0.9690 (Table 2). A second 2.5B LRKV/MHA seed preserves the common-protocol loss ordering (Table 3). Full-rank and initialization controls, matched-tier downstream evaluation, and a TPA/MLRA diagnostic further clarify the scope of the result (Tables 8, 10, and 9).

3. A serving comparison that includes optimized MLA.

At batch 32/32K on one B200, rank-8 LRKV achieves 1250 tok/s versus FlashInfer MLA-DS's 1300. At batch 8/32K, rank-64 LRKV achieves 420 tok/s versus MHA's 250. The first comparison establishes competitive throughput at the tested operating points; the second establishes a 1.68x gain where MHA fits. Capacity and quality claims are tied to their respective ranks (Tables 4 and 14).

Corrections and scope. Our main evidence now comes from the matched-budget experiments, the second 2.5B seed, and comparisons with optimized decoding baselines. We have removed the earlier claims about the 6.3B loss ordering, 32K RULER performance and extrapolation. The new retrieval test is a controlled probe. We also specify the configuration behind each quality and serving result and have corrected the descriptions of learning-rate tuning and numerical agreement. We explain these changes in our replies to the relevant reviewers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.