acceptodds
Under review as a conference paper at ICLR 2027

DoseRank: Measured Rank Allocation for Training-Free MLA Conversion

Abstract

The key-value (KV) cache is a major memory bottleneck in large language model serving. Multi-head latent attention (MLA) reduces this footprint by caching a low-rank latent representation per token instead of separate keys and values. However, converting existing multi-head attention (MHA) and grouped-query attention (GQA) checkpoints to MLA without expensive retraining remains challenging. Prior methods allocate rank uniformly across layers or use spectral information to guide allocation, yet a layer's spectrum can be a weak predictor of its contribution to accuracy. We introduce DoseRank, an accuracy-guided rank allocator that measures each layer's sensitivity to compression using a smoothed accuracy metric on a small calibration set. At matched KV-cache sizes, DoseRank improves accuracy over uniform rank allocation by up to 5.5, 2.9, 5.6, and 7.9 percentage points on gpt-oss-120B, gpt-oss-20B, Grok-2.5, and Llama-3.1-8B, respectively. On our CARE-basis port, evaluated without recovery training, DoseRank outperforms the CARE-E spectral scheduler by up to 3.4, 2.5, 5.5, and 5.8 percentage points, respectively. Among evaluated budgets where compression leaves room for accuracy recovery, item-paired 95% confidence intervals exclude zero at most budgets for uniform and spectrum-guided allocation. Calibration on a strict subset of tasks retains most of the gains on held-out tasks, suggesting that DoseRank captures compression sensitivity that transfers beyond the calibration tasks. These results demonstrate the value of allocating rank according to measured accuracy sensitivity rather than spectral information alone. Code will be released upon acceptance.

Primary area:
optimization
Keywords:
kv-cache, inference, training-free

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6qqmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DoseRank, a training-free rank allocation method for converting existing attention models to MLA. Instead of assigning the same latent rank to every layer or relying on spectral energy, DoseRank directly measures each layer's accuracy sensitivity around a compressed operating point and uses these measurements to allocate a fixed rank budget. Across four models, DoseRank substantially improves accuracy over uniform allocation at matched KV-cache budgets and also consistently improves over CARE's spectral scheduler when applied to the CARE decomposition. The paper further studies task-transfer of the calibration signal and provides statistical tests and failure diagnostics.

Critical strengths
  1. Clean and consequential empirical result: The paper isolates rank allocation from decomposition and compares methods at matched KV budgets. DoseRank improves over uniform allocation at 24/26 tested budgets and also improves the CARE decomposition's spectral scheduler. This makes a convincing case that measured downstream sensitivity is substantially more useful than spectral energy for this allocation problem.
  2. Careful experimental diagnosis: The paper does more than report aggregate gains: it includes paired confidence intervals, multiple-comparison correction, alternative sensitivity proxies, held-out calibration-task experiments, and explicit analysis of regimes where the probe signal becomes unreliable. The held-out-task results are particularly useful in reducing the concern that DoseRank simply memorizes the calibration items.
Critical weaknesses
  1. Generalization beyond the evaluation suite remains unclear: DoseRank's default calibration data come from the same seven task families used for evaluation, and the sensitivity statistic is closely aligned with multiple-choice accuracy. The held-out-task ablation is helpful, but it still stays within the same benchmark suite. More importantly, generation and long-context behavior are not evaluated, despite long-context serving being a central motivation.
  2. The baseline set does not fully resolve how much novelty comes from the compressed operating point itself: The related-work section discusses other calibration/sensitivity-based layer-allocation approaches such as ASVD and MatryoshkaKV, but the main comparisons are against uniform allocation, spectral schedulers, and proxy ablations. An adapted task-aware allocation/search baseline under the same MLA decomposition and compute budget would make the central methodological claim substantially sharper.

These issues limit my enthusiasm for a clear accept, but I do not think they invalidate the main empirical finding.

Questions for authors
  1. Can the authors evaluate an allocation calibrated on the current multiple-choice suite on a genuinely different target distribution, preferably a generative or long-context task? This would clarify whether DoseRank identifies broadly important layers or mainly benchmark-specific sensitivity.
  2. Can the authors compare against an adapted calibration-driven allocation/search baseline of the type discussed in Section 2, under the same decomposition, KV budget, and comparable calibration compute? This would help isolate the benefit of measuring sensitivity specifically at the compressed operating point.
1 comment
AuthorsOpenReview ↗

We thank you for your review

Major Weakness 1 / Question 1, generalization beyond the suite. Direct answer: an allocation calibrated on the multiple-choice suite transfers to held-out multiple-choice task families and, on TransMLA's decomposition, to needle retrieval. It does not transfer to GSM8K or LongBench.

Held-out families, gpt-oss-120B: calibrating on knowledge QA raises the held-out commonsense tasks by +2.21 pp at r̄=128 and +1.63 pp at r̄=256. MMLU alone raises the other six tasks by +2.16 pp (Section 5, Table 3b). Llama-3.1-8B, TransMLA, r̄=768 (Table 2): the suite-calibrated allocation improves the needle by 15 points over uniform ranks and leaves LongBench (9.1) and GSM8K (46.1 against 45.9) where they were. Calibrating on long-context items is what moves generation: GSM8K +5.8 [+3.2, +8.5] and LongBench +3.1. Calibrating on both sets keeps an accuracy gain of +1.3 pp and gives GSM8K +7.2. Our reading: the sensitivity DoseRank measures follows the calibration items, so they should match the target. The abstract and contribution 2 now say this, in place of the submission's general transfer claim. Caveat: the suite-calibrated allocation was solved with a rank cap of 1,536, the other two with 1,984 (Section 5). Major Weakness 2 / Question 2, a calibration-driven search baseline. The two baselines under Reviewer 4Gqb, Major Weakness 1, share our decompositions and budgets: ASVD's rule on all 52 cells of Table 1 (Table 15), and a MatryoshkaKV-style greedy search on Llama-3.1-8B (Table 16). Compute is not matched. ASVD's probes are 2.5 to 12 times cheaper than ours. The greedy search takes 1.9 to 3.8 hours against our 1.4 to 1.6, on 175 calibration items against our 2,044. The comparison that isolates the operating point is the full-rank-probe ablation, with the same statistic, items and solver (Table 3a). The operating point matters on TransMLA's decomposition and at r̄=192 on CARE's, and not at 256 or 384.

Reviewer_TgMYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors propose a training-free allocator for latent ranks across layers, which can be used for transform MHA to MLA with a fixed KV-cache budget. To compute the optimal layer allocation, the method first evaluates the model at a operating point. It iteratively adds or removes one layer within a grid of small ranks, measuring how each change affects a logistic-smoothed accuracy metric. With information for these per-layer dose–response curves, a convex solver efficiently allocates ranks under the compute budget. In the experiments accross different model size, DoseRank outperforms uniform and CARE-E allocations with the same KV-cache size.

Critical strengths
  1. It proposed a framework to test the rank that should be allocated for each layer when transfer from MHA to MLA. And it use extensive experiments to show that the previous spectrum-based methods are not theoretically sound.
  2. The problem is important and should be of interest for many researcher. Reduce KV-cache memory for long-context learning is especially useful not only for llm, but for agents. The author demonstrate the DoseRank’s usefulness with extensive experiments, while the model size is up to 120/270B.
Critical weaknesses
  1. While the authors argued that DoseRank is proposed to solve the long-context problem, they did not include benchmarks on long context. I would be interested in the result for benchmarks like Longbench. Even under the smallest LLaMA model would be enough.

[1] Bai, Yushi, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du et al. "Longbench: A bilingual, multitask benchmark for long context understanding." In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), pp. 3119-3137. 2024.

Questions for authors
  1. The author did not include the required AI use statement. I will leave this for AC to decide.
  2. Will the calibaration size affect the final accuracy? Please provide accuray vs calibration size table, even under the samllest 7B model, to give a sense of the effect of calibration size.
1 comment
AuthorsOpenReview ↗

We thank you for your review

Major Weakness 1, no long-context benchmarks. Added as Section 4.3 and Table 2: five LongBench tasks, GSM8K and a passkey needle test to 32K, on Llama-3.1-8B with TransMLA's decomposition at r̄=768.

The one-shot conversion itself loses most long-context ability. Uniform ranks drop LongBench from 37.6 to 9.1 and the needle from 100% to 58%. No allocation at this budget recovers the unconverted scores. DoseRank calibrated on the accuracy suite improves accuracy by 2.3 pp [+1.7, +2.8] and the needle by 15 points. LongBench (9.1) and GSM8K (46.1 against 45.9) do not move. Adding 400 long-context items to the calibration set improves every column over uniform ranks: accuracy +1.3 pp [+0.8, +1.9], GSM8K +7.2 [+4.5, +9.9], LongBench +2.7 and the needle +16. The last two are reported without intervals. The study covers one model, one decomposition and one budget (Appendix R). Question 1, AI use statement. We gave the disclosure on the submission form and misread the requirement as satisfied by that. The section is now in the PDF before the references.

Question 2, calibration size. New sweep from 25 to 584 items per task (Appendix Q, Table 17, Llama-3.1-8B, TransMLA, r̄=256, uniform ranks 41.3):

25 items per task already beat uniform ranks by 3.8 pp (45.1). 125 items per task give 47.6. The paper's 292 give 49.3, and 320, 380 and 584 give 48.6, 48.9 and 48.2. The gain saturates at a few hundred items per task. Draws of the same size differ by 0.1 to 1.3 pp (s.d.), and the paper's 292-item set is a single draw at the top of this range.

Reviewer_4GqbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies layerwise rank allocation for converting pretrained models to MLA without recovery training. Given a fixed low-rank decomposition, DoseRank probes individual layers around a uniform compressed configuration, measures changes in a smoothed task loss, and solves a separable budget allocation problem to obtain nonuniform ranks. The main claim is that these measurements preserve task performance better than uniform or spectrum-based allocation at the same KV-cache budget.

Critical strengths
  1. The problem is practically relevant. Reducing KV-cache memory without costly recovery training is useful, and deciding how to distribute a fixed rank budget across layers is a meaningful part of this problem.

  2. The comparisons help isolate the allocator’s contribution. Comparing allocation strategies under the same decomposition helps distinguish improvements from rank allocation from those due to the decomposition itself. Evaluating the allocator on two decompositions also supports its potential use as a separate component.

Critical weaknesses
  1. The novelty over existing sensitivity-based rank search is unclear ASVD already uses layerwise rank probing and budget-constrained search, while MatryoshkaKV evaluates further compression in the current compressed configuration. Thus, measuring sensitivity in a compressed model is not a clear distinction from prior work. arxiv.org DoseRank’s main changes appear to be the smoothed multiple-choice loss, the uniform operating point, and the one-shot allocation procedure. The paper does not sufficiently explain why this combination constitutes a substantial advance. Its smoothed loss is also a surrogate, weakening the distinction drawn between earlier proxies and “measured accuracy.”
  2. The fixed response curves do not address the state dependence emphasized by the motivation The paper argues that a layer’s sensitivity depends on how other layers are compressed, yet measures all responses at a uniform operating point and reuses them to select a nonuniform configuration. The final allocation therefore changes the conditions under which those responses were measured. Single-layer probes do not identify the additional interactions arising from simultaneous changes across layers. The checks in Appendix I provide some support for a local approximation, but do not establish its valid range or when remeasurement is needed. This remains a gap between the motivation and the allocation procedure.
  3. The nonconvex exactness claim is not valid as stated For nonconvex costs, common-price search can produce rank jumps. Filling the remaining budget using the original one-step marginal gains can then miss better allocations requiring several rank increments. The stated algorithm admits such counterexamples under the paper’s own curve-construction rules. Furthermore, selected ranks lying on their lower convex envelopes only establishes agreement between the original and convexified costs at those points; it does not prove optimality under the budget constraint. Rerunning the solver provides no independent optimality certificate. The algorithm and its claimed guarantee need to be reconciled.

Additional submission issue. The manuscript lacks the AI use statement required by ICLR 2027. This omission should be clarified by the authors and checked by the AC; it does not, by itself, establish undisclosed AI use. iclr.cc

Questions for authors

1.What is the substantive methodological contribution beyond existing sensitivity-based search methods such as ASVD and MatryoshkaKV? Please explain what previously unresolved problem is addressed by the uniform operating point, the smoothed loss, and the one-shot allocation procedure.

2.Why should single-layer responses measured at a uniform operating point remain reliable after the final allocation changes the compression state of multiple layers? What conditions preserve the relevant marginal-benefit ordering, and when would remeasurement be necessary?

3.What establishes optimality for the nonconvex costs used in practice? Please explain how the solver handles rank jumps and avoids suboptimal allocations when filling the remaining budget using the original marginal gains. An argument beyond envelope contact and reproducibility is needed.

1 comment
AuthorsOpenReview ↗

We thank you for your review

Major Weakness 1, novelty. We agree that probing layers and searching under a budget follows ASVD and MatryoshkaKV. ASVD compresses weights, and MatryoshkaKV trains projections with distillation. Ours is GQA-to-MLA conversion with no training, only calibration on a small set. Within that task we differ on two choices, each now tested alone (Section 5, Table 3):

Where we measure, the compressed operating point. On Llama-3.1-8B with TransMLA's decomposition, full-rank probes give an allocation 1.6 to 5.0 pp worse. On CARE's decomposition, operating-point probes lead at r̄=192 and tie at 256 and 384 (Table 3a). On gpt-oss-120B, full-rank probes carry no signal (Appendix M). What we measure, an accuracy-aligned statistic. With gold-NLL in its place, DoseRank loses 1.8 to 4.4 pp on gpt-oss-120B and 1.1 to 1.9 pp on Grok-2.5. On gpt-oss-20B the two are within about a point of each other. A KL-optimal allocator loses 2.8 pp to uniform ranks (Appendix H, Table 6). We ran both prior methods under our decompositions and budgets (Appendix Q):

ASVD's rule (Table 15). DoseRank is ahead in 47 of 52 cells. It is behind by 0.6 to 0.8 pp at the three loosest Grok-2.5 budgets on TransMLA's decomposition and at the loosest Llama-3.1-8B one, and tied in one cell. ASVD's probes are cheaper: 33 to 55 minutes against 1.4 to 11.3 hours. A MatryoshkaKV-style greedy search, without distillation, on Llama-3.1-8B with TransMLA's decomposition (Table 16). It beats uniform ranks by 1.8 to 4.5 pp and trails DoseRank by 2.8 to 3.5 pp. It re-measures at every step, with 4,066 to 8,070 probes on 175 items, against DoseRank's 288 probes on 2,044 items. Built from 25 items per task (175 in total), DoseRank averages 45.1 at r̄=256 (Table 17), against the search's 45.8. We therefore claim the advantage at this compute, not that one-shot allocation beats re-measurement in general. Major Weakness 2, state dependence. We agree that single-layer probes do not identify interactions between layers. Section 5 and Appendix N (Table 12) report what this costs:

Summing single-layer costs predicts the final allocation's damage within 0.84 to 1.09×. Curves measured six times below the target budget cost 0.48 pp and still beat uniform ranks by 1.6 pp. Re-measuring at the final allocation and solving again does not improve it: −0.14 pp on gpt-oss-120B (interval includes zero) and −5.15 pp on Llama-3.1-8B. On Llama-3.1-8B, re-probing at states 25, 50 and 75% of the way from the uniform point to the final allocation costs 0.10, 0.22 and 0.66 pp, and the layer ordering drifts (ρ 0.91, 0.86, 0.74). Probes taken far from the uniform point give a worse allocation. When to re-measure: we do not have a rule, and the paper says so. In every setting we tested, one measurement at the uniform operating point was enough, and we reuse it only for budgets within a factor of two of it (Section 4.1).

Major Weakness 3, solver. You were right. Common-price search can produce rank jumps, and filling the remaining budget with one-step marginal gains then misses allocations that need several rank increments. We replaced it with an exact dynamic program (Section 3.3, Algorithm 1, Proposition 1) and re-solved the allocations.

Outside Grok-2.5, 14 of the 42 DoseRank cells of Table 1 changed, by −0.54 to +0.28 pp (mean −0.07 pp). No win or loss in Table 1 changed. Eight cells of Table 5 have no paired interval, because their baseline has no item-level record. Table 5 marks them, and Section 4.2 counts them separately. AI use statement added. Method section rewritten for readability.

Comments on the paper

1 comment

We thank the reviewers for their insights. The revised PDF is uploaded, and the replies under each review give the details.

We gave the disclosure about AI use on the submission form and misread the requirement as satisfied by that. The section is now in the PDF before the references.

Grok-2.5 was re-evaluated in one environment, with vanilla and every arm scored on the same 6,500 items (Section 4.1, Appendix G). Its block of Table 1 therefore differs from the submission. No Grok-2.5 win or loss changed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.