- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies the local stability of practical Muon with five Newton–Schulz steps and stored momentum. By analyzing the joint weight–momentum Jacobian, it derives two stability boundaries for each real eigenvalue of , corresponding to multipliers and . Its main finding is that the finite Newton–Schulz map can produce lower-edge instability even under positive semidefinite curvature, unlike the differentiable exact polar map. Proposal-scaling interventions change local stability while preserving the next weight under scale-invariant normalization, and quadratic examples exhibit attracting states beyond the boundaries. Language-model experiments support the negative-response mechanism and show that successive Jacobians predict perturbation growth much better than a frozen initial Jacobian. The trajectory validation concerns smooth FP64 updates.
Critical strengths
-
S1. A stability mechanism missed by exact-polar models. The finite Newton–Schulz map can reverse perturbations and generate negative effective modes under PSD curvature. GGN and output-matched controls separate this mechanism from negative loss curvature and changes in the reference update. This is the paper's strongest contribution.
-
S2. Careful theoretical and numerical validation. The analysis handles noncommuting operators, and the quadratic constructions verify attraction in the full weight–momentum state. Independent derivatives, finite-difference checks, and explicit numerical resolution criteria support the results. The 58M replay also shows that frozen-Jacobian extrapolation fails even for identified eigenmodes.
Critical weaknesses
-
W1. The mechanism's finite-horizon contribution remains unclear. GGN and output-matched controls isolate one-step effects but are not extended to Jacobian products. The trajectory experiments use the full Hessian, so negative loss curvature may also contribute. In Table 1's fixed-batch conditional Muon replay, the lower-mode perturbation still grows , despite being far below the frozen prediction. Successive-Jacobian agreement validates propagation, but does not show whether replacing the finite-map response reduces amplification or explain the large discrepancy from the frozen prediction.
-
W2. Transfer to native mixed precision is not established. The perturbation experiments use smooth FP64 updates, whereas training uses mixed precision. Footnote 2 reports BF16–FP64 matrix-map output differences of 1.9%–11.7% and states that native BF16 finite-difference checks fail, but the corresponding evidence is not provided in Appendix J.11. The key question is whether perturbations that are accurately represented initially still follow the predicted dynamics under native arithmetic.
-
W3. The changing-batch residence measurements do not isolate state dynamics. In Figure 6, the median upper ratio is 0.988 and only 13/64 values fall within ±5% of the edge, failing the paper's residence criterion. As the authors acknowledge, these measurements do not separate batch variability from state changes. Six repeated-batch controls already demonstrate non-residence in small-model continuations. What remains unclear is how much the variation in the changing-batch run reflects state dynamics, and whether the finding extends to larger models with the complete optimizer.
Questions for authors
-
Q1 (W1). Does replacing the finite-map response reduce finite-horizon amplification? Could the output-matched control be re-centered at successive reference states, preserving the reference trajectory while comparing original and controlled Jacobian products in the same fixed norm? A corresponding GGN comparison would help distinguish map effects from negative loss curvature. Does the frozen-versus-ordered discrepancy arise mainly from weaker local amplification or changing perturbation directions?
-
Q2 (W2). Which predictions survive native precision? For identical checkpoints, batches, and perturbation directions, over what amplitudes and horizons do FP32 and native mixed-precision responses agree with the smooth prediction? Please provide the native BF16 protocol and results referenced in footnote 2, distinguishing perturbations lost at initialization from deviations caused by subsequent rounding or finite-amplitude effects.
-
Q3 (W3). What changes when the saved states from the changing-batch continuation are evaluated on a fixed probe batch or panel? Using a consistent positive-mode selection rule, please report temporal averages, dispersion, and residence counts. This would separate probe-batch variability from changes along the original trajectory, complementing the existing repeated-batch training controls. A similar diagnostic at selected 58M checkpoints would test whether the finding extends beyond the small models.
Required Corrections
-
Undefined polynomial (Appendix F.4, p. 29, L1512–L1519). The recurrence uses without identifying with the previously defined (p. 28, L1498–L1501). Define or use consistent notation. Also correct the incomplete sentence preceding the recurrence.
-
Missing native BF16 evidence (p. 6, L321–L323; Appendix J.11, pp. 39–40, L2095–L2142). Footnote 2 refers to failed native BF16 finite-difference checks, but Appendix J.11 does not report their protocol or results. Provide this evidence or correct the cross-reference.
-
Unspecified edge-distance test (Appendix J.6, p. 37, L1970–L1975). Define the edge-distance test and its numerical tolerance, including why only the smallest amplitude passes in the unresolved case behind the reported 11/12 upper crossings.
-
Imprecise scaling description (p. 1, L020–L021; p. 2, L082–L084; Eq. (5), p. 5, L253–L261). “Rescaling stored momentum” may suggest , whereas Eq. (5) defines an affine intervention that rescales the proposal. Clarify this distinction and the scale-invariance condition for exact next-weight preservation.
-
Copyediting (p. 5, L218). Replace “the Muon’s matrix map” with “Muon’s matrix map.”