- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
LRKV splits the difference. Each head's K/V projection becomes a shared full-rank matrix plus a small head-specific rank-r correction: W_h = W_shared + U_h B_hᵀ. The cache then holds one shared K/V state per layer plus a rank-r latent per head, costing 1/H + r/d_h of what MHA costs. They pretrain from scratch at 128M, 1.2B, 2.5B and 6.3B on FineWeb-Edu against MHA, two GQA configs, MQA and MLA, with the learning rate grid-searched separately for every architecture.
Critical strengths
1- The problem of KV-cache compression for Transformer-based LLMs is practical. 2- Including the models at different scales in the experiment is an upside. The learning rate is searched per scale and per attention type.
Critical weaknesses
1- The solution for KV cache compression is heavier than many baselines that do upcycling instead of pre-training (e.g. X-EcoMLA) or the methods avoid any training (e.g. PALU).
2- I am not convinced that the experimental setting of the paper is practical and the results of the experiments conclusive. Pre-training a B parameter model with 50B token is undertrained and comparing their results might not be conclusive.
3- Result flags: Table 1 at 6.3B reads: MQA gets 2.304 CE using 3% of MHA's cache, while LRKV's default gets 2.288 using 53.1%. That's 0.016 CE, or 0.005 BPB, bought with 17.7x more memory. LRKV's loss advantage over MQA shrinks at every scale: 0.067 at 128M, 0.064 at 1.2B, 0.059 at 2.5B, 0.016 at 6.3B.
4- Comparing models only against CE loss will not necessarily reflect the task behavior of the models. Table 2 reporting the GSM8K results of 0-10% are signaling the performance of the model (clear evidence of undertrained model).
Missing baselines: X-EcoMLA, PALU and ....
Questions for authors
1- At 6.3B, MQA reaches 2.304 CE with 3% cache against your 2.288 with 53.1%, and the gap has narrowed at every scale you tested. Do you expect it to keep narrowing? If your answer is that loss converges but downstream doesn't (Table 2: MQA 33.4 vs LRKV 40.4, mostly HumanEval), please make that argument directly and support it. It's the strongest version of your case and right now it's a parenthetical.
2- Is one extra seed at 2.5B and 6.3B feasible for the rebuttal, even just for LRKV and MHA?
3- Which configuration does each abstract number come from? "8.5–8.9× MHA's resident tokens" is r=8 at 8.3% cache (l.363–365, Table 5: 566 vs 66 at 8K). The loss-leadership claim is r=64 at 48–67% cache, whose measured capacity is 118 vs 66, so 1.8×, and which OOMs at batch 128/8K and batch 32/32K where MQA, GQA and LRKV r=8 all run. The conclusion's "reduces the KV cache from 20.29 to 2.40 GB and peak memory from 27.19 to 9.20 GB" is also r=8. Tagging each number with its rank would fix this.
Replication, training scope, and practical trade-offs
Thank you for asking whether LRKV is a practical architectural choice. The revision adds a second 2.5B seed, an upcycling experiment, and matched-tier evaluations. Each abstract number now identifies its configuration.
1. Training budget and the extra seed (Q2).
The common-protocol 2.5B comparison now has two LRKV/MHA seeds. Their BPB values are 0.7191/0.7230 and 0.7195/0.7222, giving LRKV-minus-MHA gaps of -0.0039 and -0.0027 (Table 3). LRKV r=64 uses 55.56% of MHA's cache. This replicates a quality improvement at that operating point and training budget.
We agree that 50B-token training cannot establish the eventual large-model ordering at convergence. An equal-horizon comparison still tests the architecture under a fixed training budget. The 128M, 100B-token three-seed comparison and separately tuned 512M/8K suite complement that evidence (Tables 1 and 2). Longer large-model training remains open; we report no additional 6.3B seed.
2. The 6.3B/MQA question (Q1).
Our audit found unequal horizons among the archived 6.3B runs and derived rather than directly measured CE entries. We withdraw the controlled ordering and the cross-scale narrowing trend based on them. We therefore do not predict whether the gap will continue to shrink or use the archived downstream scores to claim that a persistent task gap compensates for a converging loss gap.
The revised claim is a range of operating points. For example, in E0, 2.5B MQA uses 5.56% cache and reaches 0.7290 BPB, while LRKV r=64 uses 55.56% and reaches 0.7191. That is a meaningful quality-memory choice, not universal dominance over MQA. The controlled case against other compression methods comes from the matched-budget experiments, not this unequal-budget pair.
3. Upcycling and post-training alternatives.
Starting from an existing checkpoint can be much cheaper than pretraining. We test this route by converting a trained 2.5B MHA model to r=64 using SVD, then adapting for 2B tokens (Table 12). The MHA source is 0.7226 BPB, conversion initially gives approximately 0.78, and adaptation recovers to 0.7290 at 55.56% cache. The from-scratch LRKV reference is 0.7187 in this experiment.
This demonstrates an adaptation route and quantifies its remaining cost in quality: 0.0064 BPB above the source MHA and 0.0103 above the LRKV reference. It is not a direct Palu/X-EcoMLA benchmark or a matched-total-cost comparison. We now discuss both as practical alternatives. LRKV's contribution is the additive parameterization and its measured trade-off; superiority over existing upcycling methods is not part of the acceptance case.
4. Task behavior and low absolute performance.
The new nominal-50% checkpoints receive identical midtraining and five-task evaluation: LRKV scores 19.4, MLA-DS 18.9, GQA-3 18.6, and MHA 19.1 (Table 10). This tests task behavior at the relevant budget. Low absolute performance and sub-one-point gaps limit practical claims; the scores do not establish significant downstream gains or deployment-ready reasoning.
5. Abstract numbers identify their configurations (Q3).
The revised abstract identifies the operating point for every numerical claim:
The new optimized comparison also shows rank-8 LRKV at 1250 tok/s versus FlashInfer MLA-DS's 1300 at batch 32/32K, and the batch-1 slowdown is explicit. We removed the untraceable absolute-memory numbers and unsupported training-step savings. Rank-64 batch 32 succeeds at 32K; the conflicting maximum of 31 is replaced with a lower bound of 32, without inventing a corrected maximum.
The additional seed, conversion experiment, and matched-budget results support a useful architectural contribution within the stated scope. We would appreciate reconsideration of the recommendation. If longer training or a specific comparison remains essential to acceptance, please identify that remaining requirement.