- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether a training recipe's BF16 advantage survives quantization of the multi-head latent attention (MLA) latent cache. It uses six matched AdamW/Muon pairs (125M/350M/600M, 1B tokens) and three production MLA models. Under streaming per-channel INT3 (Stream3), Muon has more damage but still has lower final NLL. Under tokenwise INT3 (Token3), the final-NLL ranking flips to AdamW at 125M and 350M but not at 600M. At INT4 nothing reverses. In separate comparisons, lower reconstruction MSE does not order the loss damage: across recipes, and within Moonlight across policies. Evaluation uses NLL on 47 WikiText articles of 1K tokens, plus MMLU, FP8 and llama.cpp side checks.
Critical strengths
The causal streaming protocol is a useful baseline for evaluating MLA cache quantization. Quantization happens at group closure with a BF16 open tail.
Critical weaknesses
-
The main finding is narrow and fragile. The reversal appears only with a weak 7-level symmetric INT3 quantizer, only under Token3, and only at ≤350M. It disappears at INT4. Asym3 halves the damage. The models are undertrained (1.7–8 tokens per parameter) and there are only 3/1/2 seeds. Token3 excess damage has SD 0.23 across the three 125M pairs, and the 600M AdamW runs are atypical by the authors' own account. So the result is a fixed-budget counterexample, not a property of the recipes.
-
There is no mechanism, and the conclusion is weak. Optimizer, learning rate and weight decay vary together, and the equal-fit control failed. The outlier and grouping account does not predict the sign. The takeaway, "measure final quantized quality, not MSE or BF16 loss," is expected given HAWQ, BRECQ and Vlassis et al. (2025). The text is dense and hedged throughout, which makes the core result hard to extract.
Questions for authors
Does the Token3 reversal at 125M/350M survive asymmetric or 8-level INT3 (Asym3-style) with tokenwise grouping? If it does not, is the finding specific to this quantizer?
Reply to Reviewer 3f4n
Short answer to your question: the Token3 reversal survives centering alone (asym7, all six pairs) and an eighth level alone in two of six (sym8), and it disappears under the asymmetric eight-level grid, where the recipes are within 0.007–0.042 nats. You were right that the reversal is narrow. What is not narrow is the result we now lead with: two Muon settings within 0.04 nats in BF16 differ by 0.12–0.58 nats after Token3 in five 125M seeds and at 350M, and in the same direction under all four tokenwise grids, including the asymmetric eight-level one (Tables 55, 57a).
The grid comparison (Table 44; same evaluator and 47 articles; H100 rerun with its own baselines). Tokenwise grouping, Muon minus AdamW, 125M seeds 0–4 (3 and 4 are new) then 350M:
Under Token3 the excess is positive on 282 of 282 articles across the six pairs, with a smallest single-article excess of +0.060 nats (Table 46), so it is a shift of the whole evaluation set. Under temporal grouping the excess is −0.006 to +0.027 and Muon keeps the better final model in every cell; among the 1B-token pairs every ranking change is under tokenwise grouping. The matched closure schedule gives the same pattern (Table 45). On seeds: we trained two more per recipe at 125M. The reversal under Token3 holds in five of five seeds and at 350M.
"Not a property of the recipes." Our decay control agrees with you in one respect and adds one thing. Both recipes apply the same per-step shrinkage (lr × decay = 1e-4). With a tenfold larger Muon decay coefficient, closed-loop Token3 damage at 125M falls from 0.39 ± 0.19 to 0.07 ± 0.01 over five seeds and from +0.511 to +0.058 at 350M, and Muon then has the lower NLL than AdamW in all 48 grid cells (Tables 1, 57). So the tokenwise fragility belongs to this Muon setting, not to Muon. What we add is that validation loss does not identify it: BF16 differs by 0.020 ± 0.017 nats, about the seed spread, and in four of five seeds favors the fragile setting. This also answers the SD of 0.23 you pointed to: the dispersion belongs to the fragile setting (seed 2: +0.630 → +0.056 with the larger coefficient). A streaming excess of +0.008 to +0.017 remains. At half and twice Muon's learning rate (one seed; Tables 1, 56) the larger coefficient again cuts Token3 damage two- to fourfold, but there it also has the lower BF16 loss, so the selection failure is specific to the reference rate.
It is also not a fit artifact: all four pairs have a Muon stage within 0.04 nats of final AdamW's BF16 loss with 3.9–8.8× its tokenwise damage (Table 50; un-annealed checkpoints).
600M. The 600M tokenwise ordering is removed from the abstract. AdamW-600M at lr 3e-4 reaches a similar loss and its Token3 damage is +0.034 instead of +0.280; a 3B-token Muon run at unchanged learning rate is fragile from its first saved stage. We read both as learning-rate pathologies and draw no recipe conclusion at 600M.
"No mechanism." We add measurements, an intervention, and say where they stop (Section 4, Tables 47, 58). Subtracting a static per-channel mean before Token3 quantization (estimated on four held-out articles; no per-token cost) removes 74% of the fragile setting's damage at 125M (0.39 → 0.10, five seeds) and 89% at 350M, leaves AdamW unchanged at 125M, and restores Muon's lead in five of six runs; a twofold residual remains at 125M. The fragile checkpoints have 5.8–7.1× the latent→key gain and off-center latents; the loss change is not first order. Removing the gain removes most of the tokenwise excess but not the streaming excess.
Presentation. The introduction now ends with a three-sentence practical summary, and Section 4 is three short paragraphs around one table.
If a specific remaining check would change your assessment, please name it; we can still run evaluation-only checks this week.