acceptodds
Under review as a conference paper at ICLR 2027

When Cleaner Caches Break: Training Recipes Shape MLA Cache Quantizability

Abstract

Quantizing the latent cache of multi-head latent attention (MLA) changes both reconstruction error and final predictive quality. We study how these outcomes depend on the trained model and on which cache coordinates share a quantization scale. Our evaluation uses three production models and six matched AdamW/Muon checkpoint pairs across three sizes. A streaming temporal policy preserves causality by quantizing groups only when they close. Under streaming INT3, Muon has greater loss damage in every pair despite cleaner marginal statistics, while retaining better final quality. Under tokenwise INT3, Muon has worse final quality at 125M and 350M, but better quality at 600M. At INT4 Muon keeps better final quality in every pair under both policies, so the reversal is specific to the seven-level symmetric INT3 reference. The activation measurements motivate a geometry- specific interpretation: channel outliers can enlarge neighboring channels’ steps under tokenwise grouping, but not under per-channel temporal grouping. This grouping argument does not order the recipes: Muon has the lower outlier ratios yet the larger tokenwise damage at 125M and 350M. Controls that hold the closure schedule fixed while regrouping channels, and that tile the latent width evenly, preserve key recipe-dependent differences; an asymmetric INT3 quantizer, which stores one extra FP16 value per group, reduces Muon’s streaming damage by 51–77% across the six checkpoints. Achieved fit remains a possible mediator of the sensitivity differences. Token-local noise probes, full-prompt MMLU, FP8 controls, and independent engine tests through 32K complement the causal comparisons. We report all matched seeds and distinguish compression damage from final quality at the tested training budget.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Head Latent Attention, MLA, KV Cache, Latent Cache Quantization, KV Cache Quantization, Low-Bit Quantization, Post-Training Quantization, LLM Inference, Cache Compression, Muon, AdamW, Training Recipe, Quantization Robustness, Representation Sensitivity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over one review, above 27% of the others.
Reviewer_3f4nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether a training recipe's BF16 advantage survives quantization of the multi-head latent attention (MLA) latent cache. It uses six matched AdamW/Muon pairs (125M/350M/600M, 1B tokens) and three production MLA models. Under streaming per-channel INT3 (Stream3), Muon has more damage but still has lower final NLL. Under tokenwise INT3 (Token3), the final-NLL ranking flips to AdamW at 125M and 350M but not at 600M. At INT4 nothing reverses. In separate comparisons, lower reconstruction MSE does not order the loss damage: across recipes, and within Moonlight across policies. Evaluation uses NLL on 47 WikiText articles of 1K tokens, plus MMLU, FP8 and llama.cpp side checks.

Critical strengths

The causal streaming protocol is a useful baseline for evaluating MLA cache quantization. Quantization happens at group closure with a BF16 open tail.

Critical weaknesses
  1. The main finding is narrow and fragile. The reversal appears only with a weak 7-level symmetric INT3 quantizer, only under Token3, and only at ≤350M. It disappears at INT4. Asym3 halves the damage. The models are undertrained (1.7–8 tokens per parameter) and there are only 3/1/2 seeds. Token3 excess damage has SD 0.23 across the three 125M pairs, and the 600M AdamW runs are atypical by the authors' own account. So the result is a fixed-budget counterexample, not a property of the recipes.

  2. There is no mechanism, and the conclusion is weak. Optimizer, learning rate and weight decay vary together, and the equal-fit control failed. The outlier and grouping account does not predict the sign. The takeaway, "measure final quantized quality, not MSE or BF16 loss," is expected given HAWQ, BRECQ and Vlassis et al. (2025). The text is dense and hedged throughout, which makes the core result hard to extract.

Questions for authors

Does the Token3 reversal at 125M/350M survive asymmetric or 8-level INT3 (Asym3-style) with tokenwise grouping? If it does not, is the finding specific to this quantizer?

1 comment
AuthorsOpenReview ↗

Reply to Reviewer 3f4n

Short answer to your question: the Token3 reversal survives centering alone (asym7, all six pairs) and an eighth level alone in two of six (sym8), and it disappears under the asymmetric eight-level grid, where the recipes are within 0.007–0.042 nats. You were right that the reversal is narrow. What is not narrow is the result we now lead with: two Muon settings within 0.04 nats in BF16 differ by 0.12–0.58 nats after Token3 in five 125M seeds and at 350M, and in the same direction under all four tokenwise grids, including the asymmetric eight-level one (Tables 55, 57a).

The grid comparison (Table 44; same evaluator and 47 articles; H100 rerun with its own baselines). Tokenwise grouping, Muon minus AdamW, 125M seeds 0–4 (3 and 4 are new) then 350M:

gridexcess damagefinal-NLL gap (positive = AdamW better)
sym7 (Token3)+0.204 / +0.176 / +0.587 / +0.270 / +0.520 / +0.439+0.108 / +0.072 / +0.492 / +0.170 / +0.423 / +0.345
asym7+0.137 / +0.146 / +0.165 / +0.288 / +0.107 / +0.147+0.040 / +0.043 / +0.070 / +0.189 / +0.011 / +0.052
sym8+0.033 / +0.063 / +0.372 / −0.045 / −0.099 / +0.146−0.063 / −0.040 / +0.277 / −0.144 / −0.196 / +0.052
asym8+0.089 / +0.091 / +0.078 / +0.057 / +0.084 / +0.067−0.007 / −0.012 / −0.017 / −0.042 / −0.012 / −0.028

Under Token3 the excess is positive on 282 of 282 articles across the six pairs, with a smallest single-article excess of +0.060 nats (Table 46), so it is a shift of the whole evaluation set. Under temporal grouping the excess is −0.006 to +0.027 and Muon keeps the better final model in every cell; among the 1B-token pairs every ranking change is under tokenwise grouping. The matched closure schedule gives the same pattern (Table 45). On seeds: we trained two more per recipe at 125M. The reversal under Token3 holds in five of five seeds and at 350M.

"Not a property of the recipes." Our decay control agrees with you in one respect and adds one thing. Both recipes apply the same per-step shrinkage (lr × decay = 1e-4). With a tenfold larger Muon decay coefficient, closed-loop Token3 damage at 125M falls from 0.39 ± 0.19 to 0.07 ± 0.01 over five seeds and from +0.511 to +0.058 at 350M, and Muon then has the lower NLL than AdamW in all 48 grid cells (Tables 1, 57). So the tokenwise fragility belongs to this Muon setting, not to Muon. What we add is that validation loss does not identify it: BF16 differs by 0.020 ± 0.017 nats, about the seed spread, and in four of five seeds favors the fragile setting. This also answers the SD of 0.23 you pointed to: the dispersion belongs to the fragile setting (seed 2: +0.630 → +0.056 with the larger coefficient). A streaming excess of +0.008 to +0.017 remains. At half and twice Muon's learning rate (one seed; Tables 1, 56) the larger coefficient again cuts Token3 damage two- to fourfold, but there it also has the lower BF16 loss, so the selection failure is specific to the reference rate.

It is also not a fit artifact: all four pairs have a Muon stage within 0.04 nats of final AdamW's BF16 loss with 3.9–8.8× its tokenwise damage (Table 50; un-annealed checkpoints).

600M. The 600M tokenwise ordering is removed from the abstract. AdamW-600M at lr 3e-4 reaches a similar loss and its Token3 damage is +0.034 instead of +0.280; a 3B-token Muon run at unchanged learning rate is fragile from its first saved stage. We read both as learning-rate pathologies and draw no recipe conclusion at 600M.

"No mechanism." We add measurements, an intervention, and say where they stop (Section 4, Tables 47, 58). Subtracting a static per-channel mean before Token3 quantization (estimated on four held-out articles; no per-token cost) removes 74% of the fragile setting's damage at 125M (0.39 → 0.10, five seeds) and 89% at 350M, leaves AdamW unchanged at 125M, and restores Muon's lead in five of six runs; a twofold residual remains at 125M. The fragile checkpoints have 5.8–7.1× the latent→key gain and off-center latents; the loss change is not first order. Removing the gain removes most of the tokenwise excess but not the streaming excess.

Presentation. The introduction now ends with a three-sentence practical summary, and Section 4 is three short paragraphs around one table.

If a specific remaining check would change your assessment, please name it; we can still run evaluation-only checks this week.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General response

We thank the reviewers. The paper's claim after revision: full-precision loss does not identify a training setting whose INT3 latent cache is fragile. Two Muon settings that differ only in the decay coefficient are within 0.04 nats in BF16 and 0.12–0.58 nats apart after Token3 quantization, in five seeds at 125M and one run at 350M; in four of the five seeds the worse one has the lower BF16 loss, within seed variation (Tables 1, 55). Against AdamW, the fragile setting takes more INT3 damage at 125M and 350M, and the cache policy decides whether that overturns its BF16 lead.

The reviews converge on four points: no mechanism, a narrow reversal, 600M convergence, and missing generation and out-of-domain evaluation. We ran the requested controls with the paper's frozen causal evaluator on the same 47 articles. Changes are in blue; new material is Section 4 and Appendix O. Every submitted number is retained.

What stands. We trained two more 125M seeds of each recipe, so there are now five. At 125M and 350M Muon's INT3 damage exceeds AdamW's under the sym7, asym7 and asym8 tokenwise grids in all six pairs (Table 44), and under the paper's Token3 on 282 of 282 articles (Table 46). Muon has the lower BF16 loss and the higher Token3 loss in five of five 125M seeds. All four pairs have a Muon stage within 0.04 nats of final AdamW's BF16 loss with 2.2–4.5× its streaming and 3.9–8.8× its tokenwise damage (Table 50; un-annealed checkpoints).

What we narrowed. The final-quality reversal is a consequence of that excess, not the headline. It occurs with seven-level grids under tokenwise grouping in all six pairs, in two of six with a symmetric eight-level grid, and not with the asymmetric eight-level grid. Among the 1B-token pairs, every ranking change is under tokenwise grouping; under temporal grouping and at INT4 the quantized ordering follows BF16.

What we removed or corrected. (1) The 600M tokenwise ordering is removed from the abstract. The submission already declined to read it as a scale effect; AdamW-600M at lr 3e-4 reaches a similar loss (validation 3.147 vs 3.152; WikiText 3.742 vs 3.726) and its Token3 damage is +0.034 instead of +0.280. (2) The lower participation ratio reported for Muon is a mean-offset artifact; centered, it is not lower.

The decay control (closed-loop; five seeds at 125M, one run at 350M; Table 1). Both recipes apply the same per-step shrinkage (lr × decay = 1e-4). With a tenfold larger Muon coefficient, Token3 damage at 125M falls from 0.39 ± 0.19 to 0.07 ± 0.01 across seeds (AdamW 0.04 ± 0.002) and at 350M from +0.511 to +0.058, while BF16 changes by +0.020 ± 0.017 and −0.014. The seed dispersion disappears with the excess, including the seed-2 outlier (+0.630 → +0.056). Muon then has the lower NLL than AdamW in all 48 grid cells (Table 57). Correcting the latent offset alone (static mean subtraction, no per-token cost) removes 74% of the fragile setting's Token3 damage at 125M and 89% at 350M, with AdamW unchanged at 125M (Table 58). A streaming excess of +0.008 to +0.017 remains in all six runs, so the decay setting is not the whole recipe effect. The fragile checkpoints have 5.8–7.1× the latent→key gain and off-center latents (Table 47).

Also new. HumanEval with the policy applied to prompt and generated tokens; 24 code files; 4,096-token articles (Tables 59–60). Keeping the first 128 cache positions in BF16 removes 99% of Moonlight's streaming damage, and the first token alone 91% (Tables 48, 58).

Limits we state. Five seeds at 125M only; 350M and 600M have one and two. At half and twice Muon's learning rate (one seed; Tables 1, 56) the larger coefficient again cuts Token3 damage two- to fourfold, but there it also has the lower BF16 loss, so the selection failure is specific to the reference rate. Nothing above 350M supports a recipe conclusion. Gain and sensitivity measurements are open-loop and used for ratios; the centering intervention is closed-loop and fails on one mistuned run. Production comparisons are observational.

Per-article records, the table script and checkpoint hashes are linked in a comment visible to reviewers and area chairs; code and weights will be released on acceptance.

AuthorsOpenReview ↗

Reply to Reviewer qSQU

Your W2 was correct: optimizer, learning rate, decay coefficient and achieved loss moved together in the submission. The revision separates two of them, and the separation produced the paper's clearest result: within Muon, a tenfold change of the decay coefficient leaves BF16 loss within 0.04 nats and changes Token3 damage 2.6- to 11-fold, in five of five 125M seeds and at 350M (Table 1).

W2. Mechanism and confounds.

  • Decay. Both recipes apply the same per-step shrinkage (lr × decay = 1e-4). With a tenfold larger Muon coefficient, closed-loop Token3 damage at 125M falls from 0.39 ± 0.19 to 0.07 ± 0.01 over five seeds (per seed: Table 54) and from +0.511 to +0.058 at 350M; the latent→key gain falls from 7.1–8.5 to 0.5–0.7. AdamW with a tenfold smaller coefficient does not become fragile, in two seeds. A streaming excess of +0.008 to +0.017 over AdamW remains, which the decay setting does not explain. At half and twice Muon's learning rate (one seed; Tables 1, 56) the larger coefficient again cuts Token3 damage two- to fourfold, but there it also has the lower BF16 loss, so the selection failure is specific to the reference rate.
  • Achieved loss. All four 125M/350M pairs have a Muon stage within 0.04 nats of final AdamW's BF16 loss, with 2.2–4.5× its streaming and 3.9–8.8× its tokenwise damage (Table 50). At 600M the matched stage shows no excess against the paper's AdamW baseline. Step checkpoints are un-annealed.
  • Matched-norm sensitivity. With i.i.d. noise of the same per-token norm as the rounding error, the 125M Muon checkpoint loses 3.3× more than AdamW; the rounding error itself costs 1.3× that noise (Table 47).

W1. Novelty. We agree that "the optimizer affects quantizability" is established (Vlassis et al., 2025) and that training-time regularization changes rankings for standard K/V caches. Two results are not in those designs. First, a within-optimizer selection failure: two Muon settings whose BF16 loss differs by 0.020 ± 0.017 nats, about the seed spread, differ by 0.30 ± 0.20 nats after Token3 in five of five seeds (Table 55), and in the same direction under all four tokenwise grids (Table 57a). Second, on fixed checkpoints the final ordering changes sign with the grouping axis and never with temporal grouping (Table 44b).

W3. Generalization and reproducibility. Added: four INT3 grids on two grouping axes for all twelve checkpoints; per-article counts; code and 4,096-token panels on the production models; HumanEval with the policy applied during generation; damage along training. Added: two more seeds per recipe at 125M (five in total) and five seeds of the decay control. Not added: seeds at 350M or 600M, or twins beyond WikiText at 1,024 tokens. Per-article records, the table script and SHA256 manifests of all checkpoints are linked in a comment visible to reviewers and area chairs, so every table cell can be recomputed; code and weights follow on acceptance.

If a specific remaining check would change your assessment, please name it; we can still run evaluation-only checks this week.

AuthorsOpenReview ↗

Reply to Reviewer AWya

Each of your three questions now has an experiment in the revised PDF, and the first changed our claims: the 600M ordering does not survive a lower AdamW learning rate or a longer schedule, so we no longer draw a conclusion from it (Figure 3, Tables 52–53). For Q2 we measured the loss change along the actual quantization error and found a 5.8–7.1× larger latent-to-key gain in the fragile checkpoints, which a decay setting removes (Tables 47, 7); for Q3, HumanEval with the policy applied during generation loses 9 pass@1 points under Token3 on Moonlight and under one point under Asym3 (Table 59).

Q1. Convergence at 600M, loss curves, transience. Figure 3 shows the training loss of the 600M runs; all are still descending at the end of their schedules, so you were right to doubt convergence. (i) AdamW-600M at lr 3e-4 reaches a similar loss (validation 3.147 vs 3.152; WikiText 3.742 vs 3.726) with higher attention entropy (3.6 nats vs 0.5 and 1.5), and its closed-loop Token3 damage is +0.034 instead of +0.280. (ii) We trained both recipes from scratch for 3B tokens. AdamW's Token3 damage is +0.031. The Muon model is fragile from its first saved stage (streaming +0.857 at 0.75B tokens, where the 1B-budget run has +0.008 at the same token count), so the difference is the schedule, not the recipe. We read both as learning-rate pathologies. At 125M and 350M, where the result stands, the excess is not transient: it is present at every saved stage (Table 51) and at matched BF16 loss in all four pairs (Table 50).

Q2. Hessian condition number or attention-logit variance. We did not estimate a Hessian spectrum; a top eigenvalue would not tell us the curvature along the actual quantization error. We measured that directly instead, and we report attention entropy rather than logit variance (Table 47): the gradient with respect to each cached latent, the first-order term, the loss change at four error amplitudes, four matched-norm noise controls, and the gain of the latent→key map. The loss change is not first order. The fragile checkpoints have 5.8–7.1× the gain and off-center latents. Removing the gain with the decay setting removes most of the tokenwise excess in five of five 125M seeds (Table 1) but not the smaller streaming excess.

Q3. Does NLL damage translate to generation? Partly. The direction carries over (Token3 costs 7–9 pass@1 points on both models and Asym3 costs under one point), but the fine ordering does not: on Moonlight, Stream3 costs more NLL than Token3 and fewer HumanEval problems. Moonlight: 73.8% with a BF16 cache, 73.2% Asym3, 68.9% Stream3, 64.6% Token3. Coder: 72.0 / 73.2 / 70.1 / 64.6. After Holm correction over eight tests, Token3 lowers Moonlight's pass@1 significantly (p = 0.041) and no policy does for Coder. Problems are short (about 400 and 300 tokens), so the temporal policies quantize only a few blocks.

Downstream breadth. On 24 code files Coder's streaming damage is −0.0050 [−0.0082, −0.0017], so its negative damage in the paper was not an out-of-domain artifact. At 4,096 tokens Moonlight is again damaged more by streaming (+0.106) than tokenwise (+0.045) (Table 60).

If a specific remaining check would change your assessment, please name it; we can still run evaluation-only checks this week.

Thank you for the detailed response and the additional experiments. The new results and clarifications adequately address my concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.