acceptodds
Under review as a conference paper at ICLR 2027

ChromaMuon: Spectral Optimizer Diagnostics and Design with Retained Newton–Schulz Coordinates

Abstract

Practical Muon optimizers approximate a polar-like update with a short Newton–Schulz recurrence. Intermediate iterates preserve singular directions while applying distinct spectral responses. Building on prior iterate mixtures, ChromaMuon uses these responses as coordinates for optimizer diagnostics and design. We characterize coefficient-level controls for fitted maps, signed edits, and time-varying paths, and relate each response to its signal-relative contribution profile . For gradient inputs, this profile gives first-order modal descent up to scale. Matched language-model experiments show that finite-coordinate fits to bounded weak-tail targets worsen mean loss in the tested family, while static negative corrections have mean effects that depend on the base map. Under native cooldown, checkpoint continuations show a shrinking local advantage of deeper geometry; transient-edit probes show early loss gains followed by endpoint penalties. These observations motivate two representative paths that retain strong matched deep-control performance. On Modded-NanoGPT Track 3, their complete configurations first reach evaluated mean loss below 3.28 before the tested Muon-family references. Transferred to NanoChat without coefficient or schedule retuning, each configuration finishes below Tuned NS12 at all 40 paired endpoints across three separately analyzed token-budget protocols, spanning 537M–2.8B parameters. ChromaMuon connects spectral characterization, matched diagnostics, and complete-path evaluation for practical optimizer design.

Primary area:
optimization
Keywords:
neural network optimization, Muon optimizer, Newton–Schulz iteration, matrix preconditioning, spectral optimization, orthogonalized optimizers, polar decomposition, language model pretraining

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_3VZhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper conducts comprehensive ablations and proposes ChromaMuon to adjust the weighting of singular components through path-dependent mixing of retained Newton–Schulz iterates.

Critical strengths

The ablations are carefully conducted, with multiple seeds that give readers a sense of the variance. The idea is interesting and directly addresses the core mechanism of Muon.

Critical weaknesses
  1. What are the wall-clock times for the experiments? Although ChromaMuon achieves lower loss for a given number of training steps, 12 Newton–Schulz iterations could be computationally expensive. Therefore, under a fixed training-time or FLOPs budget, it is unclear whether ChromaMuon has an advantage over the other baselines.
  2. The main takeaway seems to be that, for a specific modification, early improvement does not guarantee improvement at the end of training. However, it remains difficult to identify a guiding principle for improving Muon.
  3. Many tuning components are studied, but it is difficult to distinguish which findings are generally applicable from those specific to this NanoChat setting.
  4. Although I appreciate the comprehensive numerical analysis, the motivations for the parameters and functional forms selected in Sections 3.2–3.4 remain unclear.
Questions for authors
  1. What principles guide the design of a better Muon optimizer?
  2. Are the evaluations limited to pretraining, using loss and BPB as the metrics?
  3. How does the proposed method compare in practice with "The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm"?
  4. Vanilla Muon supports direct 8-bit quantization of its momentum state without loss (see "Effective Quantization of Muon Optimizer States"). Does the proposed recipe offer similar benefits, either for the best parameter choice in practice or more generally under the transformation f(x)f(x)? I understand that this may be outside the paper's current scope; I am curious because memory efficiency and throughput have substantial implications for practical usefulness.
  5. Why use x0.025x^{0.025} in the final schedule, with p.025p_{.025} being the fixed K=12K=12 coefficient vector fitted to this response, when x0.05x^{0.05} has the lowest observed mean in the static sweep (Figure 2)?
  6. What motivates the two specific training paths discussed in the paper?
2 comments

Thank you for recognizing the careful multi-seed ablations. We clarify the design guidance and scope below.

1. Wall-clock time and K=12K=12

We report target crossing in optimizer steps. Section 5 states that earlier crossings establish no wall-clock advantage. Both configurations compute X0,...,X12X_0,...,X_{12} throughout; reducing selected depth changes the response, not executed computation (§2.4).

K=12K=12 provides representation: for the fitted x0.025x^{0.025} target, observed-spectrum weighted RMSE falls from 0.26439 at K=5K=5 to 0.00478 at K=12K=12 across 54 archived spectra (Figure A1). This supports the dictionary, not compute-optimality.

Iteration count alone does not determine end-to-end cost. Polar Express reports little runtime change across 2–30 iterations in its GPT-2-Small workload because forward/backward computation dominates (§4.3, Figure 5) [1]. The computational overhead for more NS iterations is negligible. This is context, not a timing measurement or evidence of negligible overhead for our implementation.

2. Functional forms and design principles

The power family connects gradient-like f(x)=xf(x)=x with Muon-like f(x)=1f(x)=1, but changing its exponent changes the entire response. The bounded weak-tail targets instead probe promotion beyond Muon: their analytic response is at least one, whereas x0.05<1x^{0.05}<1 for 0<x<10<x<1. Table A14 audits deviations of the finite fits from these targets, so the conclusions concern the realized interventions.

A signed X0X_0 edit adds cxcx to the response and cx2cx^2 to its signal-relative contribution, providing a top-weighted perturbation. This is only partially decoupled: the edit affects the positive spectrum and norm matching couples scale. Selected depth is distinct: for q≥1q\geq1 it preserves f(1)=1f(1)=1 and f′(1)=0f^{\prime}(1)=0 while changing the interior response and weak-end slope (§§2.3–2.4).

The experiments give concrete guidance:

  • Inspect implemented responses: all twelve bounded-tail fits worsen mean loss, and their responses violate intended target bounds (Tables A4/A14).
  • Evaluate an edit jointly with its base: the same negative correction improves mean loss on one tested base and worsens it on another (Table A5).
  • Evaluate timing under the actual schedule: depth 12's continuation advantage over 5 contracts from 17.879 milli-loss at step 1000 to 0.184 at 2875, conditional on cooldown and remaining nonzero-rate exposure (Table A6).
  • Test complete paths: transient edits improve early loss but leave a mean endpoint penalty of 0.789 milli-loss after the edits vanish (Table A7).

3. The two paths and the 0.025 exponent

The paths combine the same depth schedule with either a fading signed correction or a mild fitted-power pulse. They remain close to the strong matched deep control, as assessed in complete runs and the schedule/rate comparisons (Tables A8/A13); a substantial gain over that control is not established.

The static sweep uses a fitted base throughout training under a shared signed envelope. Depth–Fitted-Power instead mixes toward the fitted x0.025x^{0.025} response with at most 25% weight and removes it by step 2000, without a signed correction. The static minimum at 0.05 therefore does not select the exponent of this temporary mixture. It is a representative construction, not a claimed optimum (Table A1; Appendix E.2) as we discussed in the paper.

4. Generality and pretraining evaluation

NanoChat adds a second stack and 537M–2.82B models. Each configuration improves all four paired endpoints in each of ten model–protocol settings without spectral retuning. This supports portability of complete configurations; differing wrappers prevent isolating spectral-path gains (§4). Held-out pretraining loss and BPB directly evaluate the objective studied here. We do not infer downstream-task improvements from these measurements.

5. Polar Express

Polar Express improves approximation to the polar factor; our study examines controlled departures through retained outputs of a fixed recurrence. Its positive language-model results [1] do not resolve our questions about signed edits and accumulated path effects. Other settings show smaller differences: Muon with Nesterov Momentum reports nearly identical nanoGPT curves for PolarExpress and empirical/theoretical quintic solvers in its randomized-Muon ablation (Appendix E.4) [2]. This does not establish equivalence in our setting. We have no direct comparison establishing superiority over Polar Express.

6. Eight-bit momentum

We have not evaluated this suggested extension. Bfloat16 matrix arithmetic is distinct from quantizing persistent optimizer states. Compatibility with eight-bit momentum is therefore unestablished; the paper makes no claim about it.

[1] Amsel et al. The Polar Express. ICLR 2026, §4.3. https://arxiv.org/html/2505.16932v5

[2] Muon with Nesterov Momentum: Heavy-Tailed Noise and (Randomized) Inexact Polar Decomposition, Appendix E.4. https://arxiv.org/html/2605.06884v2

Thanks for the response. My concern is partially resolved so i will maintain my score.

Reviewer_eYSLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper uses intermediate Newton–Schulz iterates as spectral coordinates for optimizer diagnostics and proposes two phase-dependent ChromaMuon configurations. Both configurations perform consistently across the tested settings, but the diagnostic experiments do not clearly justify the final design choices.

Critical strengths
  • Retaining intermediate Newton–Schulz iterates provides a useful way to study spectral transformations without an explicit SVD. These iterates share the same singular directions but produce different singular-value responses, offering a practical framework for optimizer analysis and design.

  • As complete implementations, both Depth–X0X_0 and Depth–Fitted-Power outperform Tuned NS12 across the tested NanoChat model sizes and token-budget protocols.

Critical weaknesses
  • The final configurations are not well motivated by the diagnostic experiments. The tested transient negative X0X_0 corrections improve early loss but worsen final loss, while the fitted-power configuration differs from the best setting in the static sweep.

  • The 12 weak-singular-value experiments do not isolate the intended intervention. Their fitted responses deviate substantially from the targets and fail to preserve some of their properties, making it difficult to draw conclusions about weak-singular-value enhancement.

  • The proposed configurations rely on manually chosen phases, coefficients, and transition points. The paper identifies these values as empirical choices but does not explain how the particular values were selected or provide a theoretical rationale for them.

  • The NanoChat comparison changes more than the spectral paths, as ChromaMuon and Tuned NS12 also differ in their surrounding optimizer components. This makes it difficult to attribute the improvements specifically to the dynamic spectral paths.

Questions for authors
  • The tested transient negative X0X_0 corrections improve early loss but worsen final loss, and disabling X0X_0 gives a slightly better final mean. Why is this correction retained in the final configuration?

  • The static sweep achieves its lowest mean at β=0.05\beta=0.05. Why does the final configuration use x0.025x^{0.025}?

  • Given that the 12 weak-singular-value fits do not accurately reproduce their target responses, what conclusions can be drawn from these experiments?

  • How were the X0X_0 coefficient of −0.2-0.2, the mixture weight of 0.25, the transition points, and the 12→512\to 5 depth schedule selected?

  • The NanoChat schedules are rescaled only according to relative optimizer-step progress. Why should this rule apply across different batch sizes and token budgets, and extend to longer training horizons?

2 comments

Thank you for recognizing the usefulness of retained NS coordinates and the consistency of the NanoChat results. Our diagnostics motivate representative constructions whose behavior is tested in complete runs; they do not uniquely determine an optimal recipe.

1. From diagnostics to the two configurations. The static sweep identifies a favorable near-Muon region, while signed edits reveal dependence on the base response (§§3.2–3.3). The paired continuation benefit of depth 12 over 5 contracts from 17.879±0.544 milli-loss at source step 1000 to 0.184±0.008 at 2875 (Table A6). This motivates testing deep responses early and late contraction under the native cooldown. Depth–X0X_0 combines that depth path with a fading signed correction; Depth–Fitted-Power uses a temporary fitted-response blend without that correction.

2. Imperfect weak-tail fits. The interventions use finite-coordinate fits of analytic targets. Table A14 audits them on 16,384 points disjoint from the 512 fitting points: all twelve exceed the analytic upper bound, and four dip below response 1. Accordingly, §3.3 explicitly states that approximation effects and intended reweighting cannot be separated. All twelve implemented fits worsen mean loss by 4.36–15.01 milli-loss (Table A4). Thus increased target promotion did not yield improvements in the tested fitting-and-training pipeline; this does not reject ideal weak-tail enhancement generally. Inspecting the realized response is itself essential diagnostic information.

3. Depth, constants and breakpoints. Dictionary capacity K and selected depth q(t) are distinct. Across 54 archived spectra, weighted fitting RMSE for x0.025x^{0.025} falls from 0.26439 at K=5K=5 to 0.00478 at K=12K=12 (Figure A1); Table A2 reports a 0.00712 early continuation improvement of K=12K=12 over K=5K=5. This supports an expressive dictionary, without establishing a compute-optimal K.

Table A8 compares fixed depths, an abrupt switch, two linear transitions and removal of X0X_0. The reported transition's target-window mean differs from the deep control by only 0.033 milli-loss (paired SD 0.466), versus a 1.025-milli-loss penalty for the abrupt switch. These are comparisons of complete transitions; duration and depth exposure also differ. Both configurations remain near the deep control across five matrix-rate multipliers (Table A13).

The amplitude -0.2, blend weight 0.25 and exact breakpoints are empirical choices, not analytically selected optima. Appendix E.2 defines them: the signed edit fades by 2000; the power blend ramps over 500–1000, holds through 1500 and vanishes by 2000; depth then transitions over 2500–3010. The ablations evaluate their behavior, while the diagnostics do not uniquely select every numerical constant.

4. Negative X0X_0 and the endpoint. The final -0.2/2000-step fade is tested separately in Table A8; it is not one of Table A7's twelve envelopes. Its 125–1000 mean loss is 3.89406 versus 3.89621 without the edit, while endpoint means are 3.27882 versus 3.27859. Thus it is a representative intervention with an early benefit, not an established endpoint improvement. This directly illustrates why early gains cannot substitute for complete-path evaluation. Depth–Fitted-Power provides a construction without this signed edit.

5. Static 0.05 versus dynamic 0.025. The static sweep uses a fitted base throughout 6000 steps under a shared fading-X0X_0 envelope. The final configuration instead uses, at maximum blend, 0.75fdepth+0.25ffit,0.0250.75 f_{depth} + 0.25 f_{fit,0.025}, removes the mixture by step 2000 and has no signed correction. The lowest tested static mean therefore does not select the exponent of this different, temporary mixture (Table A1). The dynamic 0.025 construction is evaluated as specified, without claiming superiority over an untested dynamic 0.05 alternative.

6. NanoChat attribution. All three methods share model, data, paired seeds, evaluation, auxiliary optimization, momentum schedule, outer matrix-rate scaling, recurrence and input-normalization implementation. Chroma retains row equilibration, output norm rescaling, variance reduction and cautious decay; Tuned NS12 uses direct updates with ordinary decay (§4; Appendix E.1.2). Therefore, the 40/40 paired endpoint improvements per configuration across ten settings support complete-configuration transfer without spectral retuning (Table A11), not an isolated spectral-path effect. The local shared-pipeline controls supply the complementary component evidence.

7. Relative progress. Using tsrc=3040t/Tt_{src}=3040t/T preserves fractional-progress breakpoints; with constant batch size, t/Tt/T also equals the fraction of the token budget consumed. Tables A10–A11 test batches of 65,536–2,097,152 tokens and budgets of 196.6M–6.291B tokens. This supports frozen transfer across different allocations. All NanoChat runs use 3000 optimizer steps, so longer update horizons and optimal timing there remain untested.

Thank you for the detailed response. Many of the clarifications appear to narrow the scope of the paper’s claims, but they do not fully address my concern that the proposed designs rely heavily on empirical choices. I will therefore maintain my original score.

Reviewer_8JsZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies variants of the Muon optimizer where instead of keeping all singular values at +-1, there is instead a spectral response that attenuates certain singular vectors. This is implemented by linearly combining the intermediate NS iterates. The proposed ChromaMuon is evaluated on experiments from the Modded-NanoGPT setting. Results show that the optimal shape is close to flat. Another round of experiments examines whether the optimal shape changes throughout training, however, results are similar to simply using the flat Muon baseline.

Critical strengths

The framework of examining Muon's resulting spectral signature, and experimenting on whether there are more optimal shapes rather than mapping all singular values to 1, is an interesting one. The experiments at NanoGPT scale are conducted fairly and with honest reporting.

Critical weaknesses

The proposed method is not novel itself, as mentioned in the paper, Soft/Contra Muon examines this question in prior work.

The results of the paper are lacking. On NanoGPT the experiments essentially show a null result, and that the original Muon flat shape is close to optimal. This is an interesting finding and a theoretical explanation why this is the case would be interesting, however, no such explanations were provided.

In general the writing in this work is quite hard to understand. Many phrases are not defined and details are scattered throughout the work without a coherent structure. Most details are simply mentioned without motivation or rationale.

Questions for authors

N/A

2 comments

Thank you for recognizing the interest of studying Muon's spectral response and the fairness of the experiments. Our contribution combines coefficient-level characterization, controlled diagnostic findings, and evaluation of representative complete configurations. Below are the clarifications of the important points of our paper.

1. Contribution beyond existing iterate mixtures

The paper explicitly credits Soft/Contra Muon for mixtures, power-response approximations, signed corrections, signal-weighted contributions, and a time-dependent schedule (§1, p. 2). Our contribution builds on those ingredients to characterize distinct interventions and measure their effects under matched conditions.

For the reported recurrence, Proposition 1 (§2.1, p. 3) gives

f_\alpha(1)=\sum_k\alpha_k,\quad f_\alpha'(0)=\sum_k2^k\alpha_k,\quad f_\alpha'(1)=\alpha_0.$$ Selecting/interpolating depth at $q\geq1$ preserves the reference endpoint value and slope, whereas adding $cX_0$ changes both and adds $cx^2$ to the contribution profile (§§2.3–2.4). These controls probe effects that a single power exponent couples together. The resulting diagnostic findings include: - **Base dependence:** applying $c=-0.05$ increases mean endpoint loss from 3.19430 to 3.19550 on fixed $k=5$, but decreases it from 3.19073 to 3.18971 on the fitted $\beta=0.05$ base (Table A5). These are descriptive four-seed mean effects, not a universal sign rule. - **Training history:** all twelve transient negative-X_0 envelopes improve mean loss over steps 125–1000 but worsen the endpoint, although the corrections have disappeared by step 2500, before the endpoint at 3040 (Table A7). This separates early benefit from the accumulated effect of a complete path. - **Schedule dependence:** the paired continuation advantage of depth 12 over 5 contracts from 17.879 milli-loss at source step 1000 to 0.184 at 2875 (Table A6), conditional on the native cooldown and remaining nonzero-rate exposure. These are specific findings about how spectral interventions interact with the base map and training trajectory. **2. Near-flat responses and theoretical context** The static sweep identifies a favorable near-Muon region, not flat-spectrum optimality. Its lowest tested mean is at $\beta=0.05$: 3.18744±0.00052, versus 3.18950±0.00135 at the Muon-like $\beta=0$ endpoint (§3.2; Table A3). The paper explicitly states that four seeds do not identify a unique best exponent. Theoretical context is already provided in Appendix D.3 (p. 30). For the current gradient $G=U \operatorname{diag}(\sigma_i) V^T$ and $1<r<\infty$, the local linear minimizer under a unit Schatten-r budget is proportional to $-U\operatorname{diag}(\sigma_i^{1/(r-1)})V^\top.$ The $r=2$ case gives matrix SGD, while the spectral-norm endpoint gives the polar direction. This explains the geometric significance of the response endpoints. It does not imply an optimal spectrum throughout finite-step stochastic training with momentum, curvature, preprocessing, and normalization. **3. Complete-path results and interpretation** The matched deep control shares the surrounding optimizer pipeline and early signed correction; both paths remain close to this strong control. The paper reports preservation of its performance, not a substantial scheduling gain over it (§§3.1, 5). Separately, Table 1 reports first evaluated target crossings at step 3020 for both configurations, versus 3175 for the nearest external reference; these are descriptive step comparisons. NanoChat evaluates complete configurations: each improves all four paired endpoints in each of ten model–protocol settings (Table A11), with wrapper differences explicitly disclosed in §4. These results complement the matched diagnostics. **4. Definitions and organization** The core objects are defined in §§2.1–2.4: the recurrence input and normalization, retained responses, contribution profile and its momentum limitation, selected depth, and coefficient path versus complete configuration. Section 3.1 specifies the comparison conditions and controls. Table A1 and Appendix A.1 provide an evidence/scope summary and notation guide; Algorithms 1–2 give the application order. This progression develops a diagnostic account of which spectral changes help under matched conditions and whether local or early gains persist through training.

Thank you to the authors for the detailed response, which is acknowledged. At the same time, my main concerns about the contents of this paper remain, and I opt to maintain my current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.