- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper conducts comprehensive ablations and proposes ChromaMuon to adjust the weighting of singular components through path-dependent mixing of retained Newton–Schulz iterates.
Critical strengths
The ablations are carefully conducted, with multiple seeds that give readers a sense of the variance. The idea is interesting and directly addresses the core mechanism of Muon.
Critical weaknesses
- What are the wall-clock times for the experiments? Although ChromaMuon achieves lower loss for a given number of training steps, 12 Newton–Schulz iterations could be computationally expensive. Therefore, under a fixed training-time or FLOPs budget, it is unclear whether ChromaMuon has an advantage over the other baselines.
- The main takeaway seems to be that, for a specific modification, early improvement does not guarantee improvement at the end of training. However, it remains difficult to identify a guiding principle for improving Muon.
- Many tuning components are studied, but it is difficult to distinguish which findings are generally applicable from those specific to this NanoChat setting.
- Although I appreciate the comprehensive numerical analysis, the motivations for the parameters and functional forms selected in Sections 3.2–3.4 remain unclear.
Questions for authors
- What principles guide the design of a better Muon optimizer?
- Are the evaluations limited to pretraining, using loss and BPB as the metrics?
- How does the proposed method compare in practice with "The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm"?
- Vanilla Muon supports direct 8-bit quantization of its momentum state without loss (see "Effective Quantization of Muon Optimizer States"). Does the proposed recipe offer similar benefits, either for the best parameter choice in practice or more generally under the transformation ? I understand that this may be outside the paper's current scope; I am curious because memory efficiency and throughput have substantial implications for practical usefulness.
- Why use in the final schedule, with being the fixed coefficient vector fitted to this response, when has the lowest observed mean in the static sweep (Figure 2)?
- What motivates the two specific training paths discussed in the paper?
2 comments
Thanks for the response. My concern is partially resolved so i will maintain my score.
Thank you for recognizing the careful multi-seed ablations. We clarify the design guidance and scope below.
1. Wall-clock time and K=12
We report target crossing in optimizer steps. Section 5 states that earlier crossings establish no wall-clock advantage. Both configurations compute X0,...,X12 throughout; reducing selected depth changes the response, not executed computation (§2.4).
K=12 provides representation: for the fitted x0.025 target, observed-spectrum weighted RMSE falls from 0.26439 at K=5 to 0.00478 at K=12 across 54 archived spectra (Figure A1). This supports the dictionary, not compute-optimality.
Iteration count alone does not determine end-to-end cost. Polar Express reports little runtime change across 2–30 iterations in its GPT-2-Small workload because forward/backward computation dominates (§4.3, Figure 5) [1]. The computational overhead for more NS iterations is negligible. This is context, not a timing measurement or evidence of negligible overhead for our implementation.
2. Functional forms and design principles
The power family connects gradient-like f(x)=x with Muon-like f(x)=1, but changing its exponent changes the entire response. The bounded weak-tail targets instead probe promotion beyond Muon: their analytic response is at least one, whereas x0.05<1 for 0<x<1. Table A14 audits deviations of the finite fits from these targets, so the conclusions concern the realized interventions.
A signed X0 edit adds cx to the response and cx2 to its signal-relative contribution, providing a top-weighted perturbation. This is only partially decoupled: the edit affects the positive spectrum and norm matching couples scale. Selected depth is distinct: for q≥1 it preserves f(1)=1 and f′(1)=0 while changing the interior response and weak-end slope (§§2.3–2.4).
The experiments give concrete guidance:
3. The two paths and the 0.025 exponent
The paths combine the same depth schedule with either a fading signed correction or a mild fitted-power pulse. They remain close to the strong matched deep control, as assessed in complete runs and the schedule/rate comparisons (Tables A8/A13); a substantial gain over that control is not established.
The static sweep uses a fitted base throughout training under a shared signed envelope. Depth–Fitted-Power instead mixes toward the fitted x0.025 response with at most 25% weight and removes it by step 2000, without a signed correction. The static minimum at 0.05 therefore does not select the exponent of this temporary mixture. It is a representative construction, not a claimed optimum (Table A1; Appendix E.2) as we discussed in the paper.
4. Generality and pretraining evaluation
NanoChat adds a second stack and 537M–2.82B models. Each configuration improves all four paired endpoints in each of ten model–protocol settings without spectral retuning. This supports portability of complete configurations; differing wrappers prevent isolating spectral-path gains (§4). Held-out pretraining loss and BPB directly evaluate the objective studied here. We do not infer downstream-task improvements from these measurements.
5. Polar Express
Polar Express improves approximation to the polar factor; our study examines controlled departures through retained outputs of a fixed recurrence. Its positive language-model results [1] do not resolve our questions about signed edits and accumulated path effects. Other settings show smaller differences: Muon with Nesterov Momentum reports nearly identical nanoGPT curves for PolarExpress and empirical/theoretical quintic solvers in its randomized-Muon ablation (Appendix E.4) [2]. This does not establish equivalence in our setting. We have no direct comparison establishing superiority over Polar Express.
6. Eight-bit momentum
We have not evaluated this suggested extension. Bfloat16 matrix arithmetic is distinct from quantizing persistent optimizer states. Compatibility with eight-bit momentum is therefore unestablished; the paper makes no claim about it.
[1] Amsel et al. The Polar Express. ICLR 2026, §4.3. https://arxiv.org/html/2505.16932v5
[2] Muon with Nesterov Momentum: Heavy-Tailed Noise and (Randomized) Inexact Polar Decomposition, Appendix E.4. https://arxiv.org/html/2605.06884v2