acceptodds
Under review as a conference paper at ICLR 2027

Orthogonalization buys a learning rate, not robustness, in RL post-training

Abstract

Orthogonalized optimizers such as Muon are reported to collapse in RL with verifiable rewards (RLVR), and the spectral filter Pion to beat AdamW. Step size accounts for Muon's collapse and for Pion's stability. The public Pion code runs Muon at thirteen times AdamW's measured step, and Pion's filter drives toward zero every matrix whose leading direction holds less than 2/3 of the energy. With each matrix's step matched in size to AdamW's, in GRPO on Qwen3, the flat-spectrum orthogonalized step tolerates within 60 steps a 2.0–3.0× larger learning rate at 0.6B and 1.4–2.8× at 1.7B; at 4B it is not determined, and over 200 steps without a penalty it is gone. That margin is not robustness: a flat step moves the layer outputs, on the inputs they receive, about three times less than an AdamW step of the same size. Matched in that output displacement it collapses at least as often as AdamW in 14 of 15 settings, and earlier in 11 of the 12 that differ (p = 0.003 over settings, 0.062 over groups of model, task and horizon); matched at every step in how much it changes the policy, its collapse fraction is within the registered ±25 points of AdamW's, and at thirty-two seeds per arm within a tighter ±20 (+8, 90% interval −4 to +20). At each arm's best learning rate the two end within a point of each other in accuracy (75.7% and 75.6%; under a practitioner's recipe, 72.3% and 71.6%), the flat step getting there sooner. The key tests were registered in advance, and the ones that failed are reported with the rest.

Primary area:
optimization
Keywords:
Reinforcement Learning, Reinforcement Learning with Verifiable Rewards, GRPO, Muon, Pion, AdamW, Orthogonalized Optimizers, Spectral Methods, Training Stability, Optimization, Update Geometry, LLM Post-Training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_bFYFOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper asks why orthogonalized momentum optimizers like Muon often collapse in GRPO post-training whereas the polynomial spectral-filter optimizer Pion stays stable. The authors trace part of this to a scale mismatch in public Pion RLVR code, which runs Muon at κ=5\kappa=5 (25x the Moonlight scale and ~13x AdamW's step RMS). Proposition 1 proves that Pion's quintic map fs(s)=2.5s3−1.5s5f_s(s)=2.5s^3-1.5s^5 has a repelling fixed point at s0=2/3s_0=\sqrt{2/3}, so matrices with s12<2/3s_1^2 < 2/3 decay doubly exponentially toward zero as iterations grow (and empirically 47% of BF16 probes underflow at 5 steps), while surviving updates shrink to O(dm−1/2)O(d_m^{-1/2}) RMS. Matching per-matrix step RMS to AdamW (Eq. 3) gives flat polar updates NS(mt)\mathrm{NS}(m_t) a 1.4-3.0x LR headroom over 60 steps on 0.6B-1.7B models (4B is inconclusive) due to ~3x smaller input-aligned change ata_t. Yet matching ata_t alone still collapses as often or earlier than AdamW; only 1-step KL matching equalizes collapse.

Critical strengths
  • Proposition 1 cleanly analyzes the quintic iteration fs(s)=2.5s3−1.5s5f_s(s)=2.5s^3-1.5s^5 around s0=2/3s_0=\sqrt{2/3}, and combining this asymptotic result with 5-step BF16 measurements explains why 47% of probe matrices zero out and surviving steps shrink by O(dm−1/2)O(d_m^{-1/2}).
  • The three-tier matching experiments (Frobenius RMS in Eq. 3, input-aligned ata_t, and 1-step policy KL across 32 seeds in Table 15) and pre-registered falsification checks (Table 37) are carefully designed.
Critical weaknesses
  1. The input-aligned metric at=ηtr(UtΣtUt⊤)/tr(Σt)a_t = \eta\sqrt{\mathrm{tr}(U_t\Sigma_t U_t^\top)/\mathrm{tr}(\Sigma_t)} fails across scales and horizons. In Sec. 5.4 (Table 15), matching ata_t alone collapses as often or earlier than AdamW on 0.6B, while on Qwen3-4B (Table 25) a/ηa/\eta differs 4x between RMS-matched polar updates and AdamW without separating collapse LRs (and polar matched in ata_t is 3/3 stable at 5×10−65\times 10^{-6} vs. AdamW's 2/3 collapse). Over 200 unpenalized steps (Table 13), all runs collapse and ata_t fails to order survival times; across 9 families (Tables 26, 28), ata_t has worse log-loss (0.71) than LR (0.68) or op-norm (0.66) and underpredicts flat-step KL by 1.09-1.88x.
  2. In Eq. (3), RMS-matched polar updates must track Adam's second moment to rescale NS(mt)\mathrm{NS}(m_t) by ∥At∥F\Vert A_t \Vert_F each step, and on 4B matching ata_t estimates input norms from 256 rows of micro-batch 1 (2 sequences, App. E.6, Table 11).
  3. Sec. 2 omits key orthogonal and RL optimizers.
Questions for authors
  1. Does the 4B a/ηa/\eta mismatch (Table 25) persist if input norms use the full batch instead of 2 sequences in micro-batch 1?
  2. Could Sec. 2 discuss Learning from Streaming Video with Orthogonal Gradients, AdamO: A Collapse-Suppressed Optimizer for Offline RL, and the spectrum-preserving Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation?

Above all, my major concern is the stability metric breakdown, I'm open to discuss.

2 comments
AuthorsOpenReview ↗

Response to Reviewer bFYF

[Edited on 13 Nov 2026: updated to the final PDF; W1 rewritten; the 4B one-step KL result added.]

Thank you. Your analysis of a_t drove the main change in this revision.

Q1 first: 4B input norms. With 2048 rows spread over 64 of the 512 micro-batches, the AdamW/polar ratio of a/eta is 3.89, vs 3.93 from the first micro-batch.

W1: we agree; the robustness claim does not rest on a_t. Its cross-scale constant fails at 4B, and a_t fails for the public Pion filters, random rotations and the sign of the momentum. The one-step KL is not offered as a stability metric either, but as a control for one paired question: at equal policy change per step, does the flat direction collapse less than Adam's? The match can separate directions: under it the sign of the momentum (5/16, against our prediction) and the edge-truncated direction (0/8, pre-specified) collapse less often than AdamW (24/32 at 3e-6), while the flat step collapses in 26/32 in that cell. For those two directions we cannot tell robustness from policy change the KL misses. The paired test:

  • Result. The flat step collapses in 34 of 64 runs, AdamW in 29 of 64 (+8 points, nominal 90% CI -4 to +20, uncorrected for the six looks): as AdamW at a 3% higher rate, against a 100-200% headroom; collapse times agree (log-rank p=0.19). The arm grew in six pre-specified looks and a +-20 margin failed, so we claim no equivalence, only that no advantage is detected.
  • What the match fixes. Only the size of each step's policy change from the same state; directions and each arm's 60-step trajectory stay free. Per step the matched KL is 0.78-1.13 of the target (5-95%). With the shared non-matrix branch frozen, counts are near ceiling but collapse times separate the matches: at 3e-6 and 5e-6, policy-matched steps 51-57 and 29-39, AdamW 51-60 and 34-40, aligned-matched 35-47 and 20-31 (Tables 16-17). Unfrozen, that branch is not rescaled and carries a median 16-79% of AdamW's one-step KL (a two-point fit; 14-20% over 200 steps), which dilutes the match. Removing the per-run refit (scale fixed in advance, pre-specified): 12/16 collapse, more often than AdamW (p=0.0495; we had predicted no difference), as AdamW at a ~12% higher rate (closed loop: 3%), so the closed loop does not create the absence of an advantage and fit noise does not hide one (App. E.11).
  • a_t remains a partial descriptor of the polar/Adam pair: at matched a_t the one-step KL of eight directions spans 2.1-6.4x instead of 48-766x at matched RMS, and it orders collapse in-sample (Cox concordance 0.85; learning rate 0.70, operator norm 0.72); held out over 9 families its log-loss is 0.71 vs 0.68 (lr) and 0.66 (operator norm).
  • 4B. The fourfold displacement ratio is real (3.89 vs 3.93), and the registered cross-scale extrapolation did not transfer: it decided 7 of 8 runs correctly, but the constants differ fourfold while the thresholds of the pre-specified seeds do not. With all seeds our threshold rule brackets the 4B ratio in (2, 7.5] only post hoc (AdamW's lower end is one run; its counts are not monotone), so the 4B headroom stays not determined. A pre-specified one-step KL test from four saved 4B states has now been scored: the polar/Adam ratio is 0.024-0.085, at most 0.15 in all four states, so by the reading fixed in advance this counts against the one-step account at 4B: it predicts a 3.4-6.5x headroom, which the 4B runs neither show nor exclude (polar-match 1/3 at 2e-5, where AdamW is 3/3). We draw no 4B conclusion (App. E.11, Table 43).

Is either yardstick reliable? a_t is not; no claim uses it as one. The one-step KL is the trust-region quantity (TRPO, PPO), used only as a paired control, and it does separate other directions. Its precision is limited: the closed- and open-loop protocols differ by 22 points (+8 vs +30), yet neither shows an advantage. Without it, at matched RMS the flat step's collapse curve is AdamW's shifted 2.2x with equal slope (a weak test).

W2: Eq. 3 needs Adam's second moment. Correct, and intentional. Eq. 3 is a measurement control that holds the per-matrix step RMS equal to AdamW's; it is not a proposed optimizer. It reuses AdamW's moments (peak memory 14.5 vs 14.6 GB on 0.6B). For practice: use the moment-free Moonlight scale (our kappa=0.2 arm) and tune the rate; the 1.4-2x starting rate in our conclusion applies to Eq. 3 only.

Q1 details. In four pre-specified 4B runs the two estimates agree within 1% (ratio 0.991-1.002); at 0.6B the small micro-batch inflates the ratio by under 10% (new Table 36).

W3 / Q2: related work. The revised Sec. 2 adds:

  • AdamO and Learning from Streaming Video with Orthogonal Gradients: orthogonality-based updates outside LLM post-training.
  • Shi et al.'s spectrum-preserving Pion, a different method with the same name (it rotates each matrix and keeps its singular values); Sec. 2 disambiguates it from Fan et al.'s high-pass filter.

Thank you for your detailed rebuttal. It has addressed all my concerns, so I will raise my score to 4.

Reviewer_op93OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

In the RL setting, the paper tries out matching norms of Muon's, Pion's and polar of AdamW momentum updates to the norm of AdamW update. The norms they're matching include Frobenius/RMS, operator and input-aligned norm. They also show that Pion (a newly proposed change to Muon designed for RLVR) has lots of its updates 0 when using BF16, and that AdamW with tuned lr performs better. If I understand correctly, a practical takeaway of this norm comparison is: when doing RLVR, we should use smaller ratio of learning rates for Muon and AdamW than what is set as the default in the public libraries. The authors claim that this reduces instabilities in training RLVR with Muon.

Critical strengths

The paper questions benefits of Pion.

Critical weaknesses

The paper is unclear, including the abstract, motivation, key contributions, methodology and the takeaway. The sentences are too long and quite hard to follow (e.g. majority of the sentences in Contribution section). The notation and phrases are used before they are introduced.

The motive of studying Pion is not so clear: in the Pion paper, they claim it's better than AdamW, but haven't done the learning rate sweep. Doing a lr sweep a is only a small contribution to the field. The same thing holds for the proposition+experiments in the paper showing that most Pion updates are actually 0.

Perhaps the authors should change the phrasing 'RL collapse', because this usually means entropy collapse, not the reward going below certain threshold (while the entropy actually increases).

Proposition 2 is nowhere to be found in the main text, but it's cited there multiple times.

Questions for authors
  1. What happens if you do use KL regularization?
  2. How do you know that the collapse (i.e. reward falling below half of what it was at step 10) isn't just a consequence of the GSM8K being too hard at small scale?
  3. Apart from Pion is not great and Muon matrix lr should be smaller than default if one does RLVR, are there any more takeaways of the paper? Can you compactly explain the claims about robustness (My understanding is that the authors are saying that the claim 'Muon is more robust to lr changes' is unfair comparison, but I haven't seen people making/using that claim)?
  4. How does the paper findings fit in other public results on RLVR with Muon (e.g. [1], where they train 4B model with Muon on math without encountering instabilities)?
  5. Have you tried tuning κ\kappa for Muon and comparing to norm-matched updates?

[1] ISO: An RLVR-Native Optimization Stack

6 comments
AuthorsOpenReview ↗

Response to Reviewer op93

[Edited on 13 Nov 2026: rewritten to answer your summary and Q3 first, then every point in order.]

Thank you. Your summary is close to our conclusion, which the paper did not state plainly enough. Two refinements:

  • Our Q1 (the reported collapse) is a scale setting, yes. The public Pion RLVR code runs Muon at kappa=5, a step 13.5x AdamW's; at the Moonlight scale kappa=0.2 Muon is stable at the same rate. So "smaller than the default" holds for that code, not for Moonlight's.
  • Our Q2-Q3: at matched step size, only a rate ratio remains. We fix every Muon matrix step to the RMS of AdamW's step (the usual fairness control). Muon's direction then tolerates a 2-3x larger learning rate (0.6B; 1.4-2.8x at 1.7B; 60 steps). We find this gap is also only a learning-rate ratio: AdamW with its step scaled by ~0.4, the factor that gives it Muon's one-step policy change, gets the headroom too (at 6e-6: 3/16 collapses vs 6/16 for Muon), and that factor predicts a 2.5x headroom before any collapse is counted (observed 2-3x).

Q3: takeaways, compactly.

  1. The reported RLVR collapse of Muon (Fan et al., 2026), attributed there to its spectrum, is a step-size setting in our setup (0.6B, n=3; one seed at 1.7B).
  2. Pion's stability is consistent with a small step: its filter outputs an exactly zero bf16 update on 47% of probe matrices per step (Prop. 1), its step per unit learning rate is ~10x smaller than AdamW's, and tuned AdamW ends more accurate (75.6% vs 73.5%, Pion's best of three rates).
  3. A larger stable learning rate is not robustness. Rescaled to AdamW's one-step policy KL at every step, Muon's direction collapses in 34/64 runs, AdamW in 29/64: no advantage detected (we do not claim equivalence). Best accuracy is equal (75.7% vs 75.6%, 8 seeds each).
  4. For comparing RL optimizers: state the scale, and check stability claims at matched one-step KL (the TRPO/PPO trust-region quantity; adds 2-8% training time). It can detect a difference: under it the edge-truncated direction collapses in 0/8 vs AdamW's 24/32 (pre-specified).

Who claims Muon is more robust? You are right: we know of no paper stating "Muon is more robust to learning-rate changes" for RLVR, and attribute it to no one. Fan et al. claim the converse (Muon's spectrum causes the collapse, Pion's filter removes it): takeaways 1-2. Ruan et al. (2026) find that Muon tolerates a more aggressive effective step than AdamW in agentic RL, and Nguyen et al. (2026) analyze how spectral flattening sets the maximal stable step. And the standard RMS-matched comparison produces the "more robust" reading itself (Fig. 1a); takeaway 3 says how to read it.

W1, clarity. Agreed. Abstract rewritten; terms defined before the contributions; Table 1 marks which answers are replicated and which provisional.

W2, why Pion; "an lr sweep is a small contribution". Our contribution is not "Muon needs a smaller learning rate". (a) The published comparison changed direction, step size (kappa=5) and a spectrum-dependent effective rate (Prop. 1) at once; our Q1 separates these and finds the reported collapse is the scale setting. (b) The standard fair comparison, equal step size, makes Muon look 2-3x more robust. (c) That apparent robustness disappears once steps are matched in their effect on the policy. A sweep shows (b), not (c); (c) changes how optimizer robustness in RL should be measured.

W3, "RL collapse". Agreed; we now say "reward collapse" and define it at first use: the first step after step 10 with training reward below half its step 1-10 mean (Sec. 5.1); entropy and truncated responses rise with it, unlike entropy collapse.

W4. Proposition 2 is now stated in Sec. 4.

Q1, KL regularization. A per-token penalty (beta=0.04, 200 steps, eta=5e-6) delays the instability, not removes it: KL to the initial policy passes 1 at steps 62-73 for AdamW and 135-147 for Muon at AdamW's RMS; policy-matched, the two cross together (medians 62 vs 66, Table 16). With warmup, cosine decay, KL penalty and clipping, no run collapses (8 seeds per arm) and accuracy is equal (72.3% vs 71.6%, Table 31): in such a recipe, choose on cost, not stability.

Q2, is GSM8K too hard? No. Qwen3-0.6B solves 63-65% before training, and on the same model and task the outcome depends on the learning rate: AdamW gains 10.2 points at 2e-6 and collapses in 3/3 runs at 5e-6, losing 63.8 points. The same Muon/AdamW ordering appears on MATH and on Qwen3-1.7B.

Q4, ISO. ISO trains Muon on Qwen3-4B-Base for 400 updates without collapse, consistent with takeaway 1; its rates and scale convention do not map onto ours (Sec. 2).

Q5, tuning kappa. Done (Table 34, pre-specified): kappa in {0.5, 1, 2, 3, 4} at eta=1e-6, three seeds each, gives 0, 0, 0, 0 and 3 of 3 collapses. kappa scales only the matrix step, so this is a sweep of the matrix learning rate; the best value, kappa=1, ends at 73.8%, against 75.7% for RMS-matched Muon and 75.6% for AdamW at their best rates.

I thank the authors for the reply. The paper is progressing towards something clearer. To make sure I’m on the same page as authors, I’m restating the contributions:

  • Contribution A) The paper pushes back on Pion in 3 ways:
    1. showing Muon’s reward collapse in Pion paper is due to too high κ\kappa, not due to its direction (which is what Pion paper claimed)
    2. showing that with properly tuned learning rate, AdamW outperforms Pion (in 3 seeds by a small margin)
    3. showing theoretically and empirically that Pion’s update is of smaller norm and encourages rank 1 updates, or even 0 updates
  • From this the conclusion is that we need more control on ‘the size of the update’ when comparing optimizers.
  • Next, the paper investigates what’s the best way to ‘match the size of the update’ for a fairer comparison. The definition of best is that the reward collapse happens at the same base learning rate. Two main candidates are:
    1. Matching the RMS norm of Muon’s matrix update to what would AdamW update be (this is introduced in [2] and called lr grafting, which the authors should cite)
    2. Scaling Muon-updates step size to match KL between the policy pre and post optimizer step (again to the KL of AdamW update)
  • Contribution B) Experiments show that with 1. matched, Muon collapse happens at a larger lr than AdamW, and when we do 2. they collapse at the same/similar base learning rate. Thus, for fair comparison in RLVR, one should match 2.

I recognize the contribution A. However, I struggle to accept contribution B), i.e. the author’s takeaway 4. Why would one define the best control for fair comparison to be the one that has reward collapse at the same base lr? If I’m misunderstanding it, then what is the criterion used to conclude matching KL is the best? Could the authors clarify this?

Another clarification I need is the definition of ‘robustness’. For example, the authors state that from figure 1(a) Muon looks more robust. I disagree, I don’t think people say an optimizer is more robust if it is stable with higher learning rate (even if lr grafted).

These two are my biggest remaining concerns, and I’m willing to reconsider my score if they become clear to me.

[2] Disentangling Adaptive Gradient Methods from Learning Rates

AuthorsOpenReview ↗

Response to Reviewer op93 (follow-up)

Thank you; this pins down exactly where the paper was unclear. Contribution A is as you state it. On B, your reading follows from our own wording (the abstract's "Practical rule: compare RL optimizers at matched per-step policy change"), and that wording claimed more than we can support. We have removed it. Below is what B claims, then your two questions.

What B claims (no KL needed). With learning-rate grafting (Adam's per-matrix magnitude, Muon's direction; thank you for the reference, now cited at Eq. 3), the two optimizers differ in RLVR by a learning-rate factor of about 2-3 and, as far as we can detect, by nothing else:

  • Collapse starts at a 2-3x larger rate (0.6B, Sec. 5.3), and the collapse curve is AdamW's shifted by 2.2x in rate with the same slope (logistic slopes 15.5 vs 15.0; few transition points for Muon, so a weak test; Sec. 5.4).
  • Learning per unit rate is slower too: at 2e-6 the grafted Muon direction gains 6.8 points where AdamW gains 10.2; it needs 5e-6 to gain 11.4 (Tables 2 and 10, n=3).
  • Each at its best rate, final accuracy is equal: 75.7% vs 75.6% (8 seeds each, Table 30). So the usable learning-rate window is moved, not widened. Grafting does not equalize the effective learning rate in RLVR.

Q1. The criterion for the KL match. It is not "collapse at the same base rate", and we do not claim it is the best match or the fair one. It is a test of one explanation for the factor above. One step of grafted Muon changes the policy as much as an Adam step scaled by about 0.4 (one-step KL, measured without counting any collapse). That predicts a 2.5x shift; we observe 2-3x (Sec. 5.4, App. E.10). The KL match then removes this difference in every step and asks whether any stability gap is left.

  • Why KL and not a norm: collapse is an event of the policy, and the one-step KL measures the step's effect on the policy directly; a parameter norm does not (at equal RMS, and even at equal output displacement, the two steps' one-step KL differs, Sec. 5.4).
  • It was fixed before its results. The first test was registered before launch with the prediction that Muon would still collapse (Table 43, KP1). It could have failed: Muon could have collapsed less. The same match does separate another direction from AdamW (edge-truncated: 0/8 vs 24/32 collapses, p=0.0002, pre-specified, App. E.11).
  • Result: 34/64 collapses vs 29/64 for AdamW. No advantage is detected; we do not claim equivalence.
  • Its limits, which is why we no longer call it a rule: one-step KL is a local quantity. Two versions of the match give +8 and +30 points against Muon (App. E.11), and for directions that do differ under it we cannot tell whether the KL misses part of the policy change. It supports B together with the three KL-free facts above; B does not rest on it alone.

For choosing an optimizer, the comparison that matters is each one at its own tuned rate. There the two are equal.

Q2. Robustness. We agree with you: being stable at a larger learning rate is not robustness, grafted or not. That is the paper's conclusion, and our phrase "looks more robust" (Sec. 1) blurred it. We have deleted it. The paper now defines the term where Q3 is posed: a direction is more robust than AdamW's only if it has a stability advantage that rescaling AdamW's learning rate cannot give. Two observable forms: a wider stable window in log learning rate, or fewer collapses when both are tuned. We find neither (same slope; at their best rates 0/8 collapses each and equal accuracy, Table 30; with warmup, decay, KL penalty and clipping neither collapses, 72.3% vs 71.6%, Table 31). The KL match is a third, more direct check with the limits above. If you use the word differently, we would be glad to adopt your definition and state the result under it.

We also agree, as in our first reply, that no paper claims the opposite for RLVR. The reason to state the result is practical. A grafted (RMS-matched) comparison at one common learning rate shows AdamW collapsing in 3/3 runs at 5e-6 and Muon in 0/3 (Table 2). That gap is only the factor of 2-3.

Takeaway 4, revised. Grafting leaves a 2-3x difference in effective learning rate in RLVR, so tune each optimizer's rate before comparing stability. The one-step KL match is a diagnostic for whether a stability gap is more than such a factor; it is not required for a fair comparison.

Changes in the revised PDF. Abstract: the "practical rule" sentence replaced (tune each optimizer's rate). Sec. 1: "looks more robust" removed; robustness defined after Q3; the KL match called a diagnostic, not a standard of fairness. Sec. 3.2: Eq. 3 identified as learning-rate grafting (Agarwal et al., 2020) applied per matrix. Fig. 1a: "a shift, not a wider window". Sec. 5.4: the criterion (registered before the first run, KP1). Sec. 5.6 and 6: "wider stable range" corrected; "check stability claims at matched KL" replaced by "tune each optimizer's rate before comparing stability".

Reviewer_Y9XCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether the instability of orthogonalized optimizers in RL post-training comes from update direction or magnitude. Through spectral analysis and controlled GRPO experiments, it identifies scaling confounds in prior comparisons of Muon, Pion, and AdamW. Flat-spectrum updates tolerate larger learning rates in several settings, but show no clear robustness advantage once policy change is matched.

Critical strengths
  1. An important practical question. Understanding when orthogonalized optimizers help or fail in RL post-training is valuable, and the paper directly examines conflicting prior findings.
  2. Careful experimental controls. Comparisons under matched update magnitude, activation displacement, and policy change help separate scaling effects from directional effects. Reporting unsuccessful predictions also clarifies the limits of the findings.
Critical weaknesses
  1. Optimizer-history dependence remains unexplored. Pretraining and SFT may shape the geometry relevant to the observed stability differences. Without comparisons involving checkpoints previously trained with Muon, it remains unclear whether the empirical answers to Q1–Q3 generalize across optimizer histories.

  2. The robustness conclusion needs clearer qualification. Most evidence comes from short runs, the learning-rate headroom remains unresolved at Qwen3-4B, and it disappears over the longer unpenalized horizon. Policy-matched comparisons support the absence of a clear advantage, but their uncertainty does not establish identical robustness.

Questions for authors
  1. Can you evaluate checkpoints pretrained or fine-tuned with Muon, ideally against matched AdamW-trained checkpoints, to test whether optimizer history changes the main findings?
  2. What practical guidance follows for choosing between Muon and AdamW? In particular, how should practitioners calibrate the learning rate without expensive policy-matching measurements?
  3. Can you distinguish which answers to Q1–Q3 hold consistently across settings and which remain provisional?
  4. Please discuss the relationship to earlier fractional-spectrum methods, particularly Qi et al. (2026) and HTMuon.
  5. Please define D~t\tilde D_t before its first use in lines 148–150; its definition currently appears later in Eq. (3).
2 comments
AuthorsOpenReview ↗

Response to Reviewer Y9XC

[Edited on 13 Nov 2026: updated to the final PDF; adds the 1.7B 8-seed result; Q2 points 2-3 updated.]

W1 was an untested gap; we have now tested it at the fine-tuning stage (24 runs, all three pre-specified predictions hold). W2: we agree, and the wording is tightened.

W2 / Q3: which answers hold, which are provisional. The revision adds Table 1 with this split.

  • Replicated (n >= 3 at the primary setting). Q1: the kappa=5 collapse disappears at a moderate scale (0.6B, n=3; consistent at 1.7B, one seed; kappa sweep, n=3 per value). Q2: at matched RMS, within 60 steps, the flat step is stable where AdamW collapses (0.6B, 1.7B, MATH; Llama-3.2-1B exploratory), and concentrated directions collapse first (0.6B, MATH). Q3: matched in policy KL, no robustness advantage (0.6B, 32 seeds per arm and rate; consistent at 1.7B, 8 seeds at 7e-6: 4/8 vs 5/8). n=3 cells: MATH 1/3 vs 0/3 and 3/3 vs 3/3; Llama 3/3 vs 3/3. Equal best accuracy.
  • Provisional. The headroom at 4B (not determined); the headroom beyond 60 steps without a penalty (gone at 200 steps); a_t as a predictor (its cross-scale extrapolation fails at 4B; it also fails for the public Pion filters, random rotations and the sign of the momentum); why the sign of the momentum and the edge-truncated direction stay more stable than AdamW when policy-matched; optimizer history (tested at the SFT stage only, n=3 per cell). We agree this shows the absence of a clear advantage, not identical robustness. The abstract now says the flat step "collapses as AdamW would at a learning rate about 3% higher" and "With the scale fixed in advance it shows no advantage either" (App. E.11).

W1 / Q1: optimizer history. All our checkpoints were AdamW-pretrained, and we know of no public Muon-pretrained model with an AdamW twin. The policy-KL match is measured on the current model at every step, so it assumes nothing about pretraining geometry. Following your question, we ran a direct test that needs no public twin. From the same Qwen3-0.6B, two short SFT runs on MATH (150 steps) differ only in the optimizer: AdamW (ckpt A) or public-code Muon at kappa=0.2 (ckpt M). On each we repeated both GRPO comparisons (pre-specified; 3 seeds per cell, 24 runs; Table 35, App. E.5):

  • RMS-matched, eta=5e-6: AdamW collapses in 3/3 (A) and 3/3 (M), the flat step in 1/3 and 2/3, and on ckpt M about 20 steps later (steps 56, 56 against 31-35). The order is consistent across histories.
  • Policy-matched, eta=3e-6: the flat step collapses in 2/3 (A) and 3/3 (M), AdamW in 1/3 and 3/3; on ckpt M both fail in the same window (steps 43-50 against 42-51). No robustness advantage on either checkpoint. All three pre-specified predictions hold, and the order of the two arms is the same on both checkpoints. At n=3 per cell this is a consistency check; SFT on MATH lowers GSM8K accuracy (64.8% to 32-36%), so we compare arms only within one checkpoint.

Q2: practical guidance (points 1-2 in the conclusion).

  1. Compare optimizers only at a stated scale: RMS matching (Eq. 3) or the Moonlight 0.2*sqrt(d_max), never kappa=5.
  2. Starting rate. For the Eq. 3 control, start the flat step at 1.4-2x AdamW's tuned rate (conclusion; observed threshold ratios 2-3x at 0.6B and 1.4-2.8x at 1.7B within 60 steps; not determined at 4B; without a penalty gone by 200 steps, Sec. 5.7). In practice use the moment-free Moonlight scale and tune the rate. With a practitioner recipe neither arm collapses (72.3% vs 71.6%; Table 31): choose on speed or cost.
  3. Threshold predictions from a 10-step a/eta held for the public Muon and MATH, were wrong for both public Pion variants (Table 42) and failed at 4B and on Llama (Table 43); held out it is no better than the learning rate. Use it to start a short scan, not to replace one.

Q4: fractional-spectrum methods. The revised Sec. 2 discusses both. HTMuon (Pang et al., 2026), Qi et al. (2026) and Muon^p (Dong & Sawin) apply fractional powers U S^p V^T, changing spectrum and step size together; we hold the RMS fixed and vary only the spectrum. Our held-out direction U S^0.5 V^T is the p=1/2 member: at matched RMS it is flatter than AdamW but has a larger a/eta; it collapsed where flatness alone predicted stability (Table 22).

Q5. Done: D~_t is defined at first use (Sec. 3.1).

AuthorsOpenReview ↗

Q2 (calibrating the rate without policy matching): now an experiment

Your Q2 asked how practitioners should calibrate Muon's learning rate without policy-matching measurements. We now have a direct answer (revised PDF, App. E.4; Table 43, JK1–JK4, criteria written before launch). With the public Muon code at the Moonlight scale κ = 0.2 on Qwen3-0.6B, reusing AdamW's tuned rate (2e-6) ends 5.5 points below AdamW (70.1 vs 75.6, n = 3); a plain sweep up to about 4–5 times AdamW's rate finds the best cell at 7.5e-6, 76.0% (n = 8, no collapse), equal to AdamW (p = 0.44). The guidance is therefore: keep the Moonlight scale, and extend the learning-rate grid upward to about 5 times AdamW's best rather than centring it there; no KL measurement is needed. (0.6B, 60 steps.) Q1 (optimizer history) is answered in App. E.5; which answers are settled and which provisional (Q3) is in Table 1.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of responses and of the revision

[Edited on 13 Nov 2026: updated to the final PDF; 15 Nov: updated to the revised PDF; adds results since 11 Nov; "registered" now means externally timestamped only, other predictions are called "pre-specified" (defined below).]

We thank the reviewers. Table and section numbers in our replies refer to the revised PDF. For the numbers in the reviews: Table 13 is now 14, 15 is 16, 25 is 26, 26 is 27, 28 is 29, 37 is 42.

The claim in three sentences. (i) The reported RLVR collapse of Muon is a scale setting: the public Pion code runs Muon with a step 13.5x AdamW's; at a standard scale Muon is stable, and tuned AdamW ends more accurate than Pion (0.6B; one seed at 1.7B). (ii) At AdamW's per-matrix step size, Muon's direction tolerates a 1.4-3x larger learning rate (0.6B, 1.7B; 60 steps), because each step changes the policy less. (iii) Matched in one-step policy KL, it collapses in 34/64 runs vs 29/64 for AdamW, as AdamW would at a ~3% higher rate (-2 to +9%), not 100-200%: no advantage is detected (nominal 90% interval -4 to +20 points, uncorrected for the six looks of a growing arm; a +-20 margin failed, so we withdraw the submission's equivalence claim).

Two checks beyond the matched collapse counts. The match predicts the headroom before any collapse is counted: AdamW's step scaled to the flat step's one-step KL is about 0.4x, implying 2.5x headroom, observed 2-3x (0.6B). Without any KL, at matched RMS the flat step's collapse curve is AdamW's shifted 2.2x in learning rate with equal slope (15.5 vs 15.0), and best accuracy is equal.

What is new. RMS matching (Eq. 3; per-matrix learning-rate grafting, Agarwal et al., 2020, and a strict form of the scale conventions of Moonlight and Ruan et al., 2026) is a common way to compare orthogonalized optimizers. In RLVR it does not equalize the effective learning rate: it still leaves an about 3x difference in output displacement per unit learning rate (4x at 4B). Matching each step's one-step policy KL removes the headroom and, at 0.6B, accounts for its size (above).

New since the submission (pre-specified in local files; App. E.10, E.11):

  • Headroom moves with the policy step. AdamW scaled to the flat step's one-step KL (scale ~0.4) collapses no more often than the flat step at 6e-6: 3/16 vs 6/16.
  • Scale fixed in advance (no per-run refit): the flat step collapses more often than AdamW, not less: 12/16 (p=0.0495 against 29/64; we had predicted no difference; on the same seeds closed loop 10/16, AdamW 9/16, p=0.46). No advantage under either protocol.
  • The match does not force equal outcomes. Under it, the sign of the momentum collapses in 5/16 vs AdamW's 24/32 (against our prediction, read post hoc). In a pre-specified detection test, the edge-truncated direction, which collapses first at matched RMS, collapses in 0/8 vs AdamW's 24/32 at 3e-6 (p=0.0002; the policy-matched flat step: 26/32 in the same cell).
  • Caveat. For the flat step nothing survives the match, and the two checks above agree. For the two directions that do differ, we cannot tell robustness from policy change the one-step KL misses.
  • 1.7B at AdamW's boundary 7e-6, 8 seeds per arm: 4/8 vs 5/8 (p=1).

What the claim rests on. Three facts that need no match (the collapse curve is shifted 2.2x with equal slope, a weak test; learning per unit rate is slower; at each arm's best rate accuracy is equal with 0/8 collapses each) and the policy-KL match as a diagnostic, not a standard of fairness. Not the input-aligned norm a_t. a_t explains most of the gap (at matched a_t the one-step KL of eight directions spans 2-6x instead of 48-766x) but is not sufficient (Table 42).

Revision.

  • Abstract rewritten around one number (op93); its guidance is now: tune each optimizer's rate (15 Nov). The KL match is tied to the trust-region principle (TRPO, PPO; Sec. 3.2).
  • Introduction: why Pion, and why a learning-rate sweep cannot answer the question; terms defined before the contributions (op93).
  • "Collapse" defined at first use; Proposition 2 in Sec. 4 (op93); D~_t defined at first use (Y9XC).
  • New Table 1 (replicated vs provisional answers) and practical guidance in the conclusion (Y9XC).
  • Related work: the methods named by all three reviewers (Sec. 2).
  • Optimizer history: a new pre-specified test from AdamW- and Muon-SFT checkpoints, all predictions hold (Y9XC Q1); pretraining history stays a limitation (Sec. 5.7). 4B input statistics from 2048 spread rows (bFYF Q1).

On "pre-specified" and "registered". Pre-specified: the prediction and its refutation rule were written down before the outcome was known, in local files (timing details in App. E.11). Registered (dagger rows of Table 43): also externally timestamped before launch. Readings without such a rule are marked post hoc. All failed predictions are reported (Tables 42-43), and the step-by-step logs of all 611 submitted runs are in the supplementary material.

AuthorsOpenReview ↗

Revised PDF uploaded

We have uploaded the revised PDF and updated our replies in place (each marked "Edited on"). What changed, per reviewer:

  • Y9XC: optimizer history tested at the fine-tuning stage (Table 35, App. E.5); Table 1 separates replicated from provisional answers; practical guidance in the conclusion.
  • op93: abstract and introduction rewritten, terms defined before use, Proposition 2 in Sec. 4; our reply now answers your summary and Q3 first, with four short takeaways.
  • bFYF: the robustness claim does not rest on a_t; the one-step policy-KL match is a control, checked by a pre-specified test that it can detect a direction effect (App. E.11); 4B input statistics from spread rows (Table 36). New results since the first revision: Qwen3-1.7B policy-matched, 8 seeds per arm (4/8 vs 5/8); the edge-truncated direction under the policy match (0/8 vs 24/32); a 4B one-step KL test, scored against its pre-specified reading (Table 43). Thank you for your time.
AuthorsOpenReview ↗

Final revised PDF (15 Nov; updated 16, 18 and 19 Nov)

The PDF was updated after our notice of 14 Nov, in response to Reviewer op93's follow-up. Text only; no result or number changed.

  • The abstract's "practical rule" is replaced: RMS matching does not equalize the effective learning rate in RLVR, so tune each optimizer's rate.
  • The one-step KL match is called a diagnostic, not a standard of fairness (Sec. 1, 3.2); robustness is defined where Q3 is posed.
  • Sec. 5.4 and the conclusion state the three observations that need no match.
  • Eq. 3 is identified as per-matrix learning-rate grafting (Agarwal et al., 2020). The General summary above is updated accordingly.

16 Nov: Sec. 1 now defines robustness as Reviewer op93 does, and App. E.3 adds one pre-specified experiment (8 seeds per relative rate; no wider window detected). No table, figure or section number changed.

18 Nov: op93's Q5 (tune κ and η jointly) is now run (App. E.4; JK1–JK4 in Table 43): the best public-code Muon cell at κ = 0.2 ends at 76.0% against AdamW's 75.6% (p = 0.44), and reusing AdamW's rate at the Moonlight scale, as that recipe intends, costs 5.5 points; the practical guidance asked for in Y9XC's Q2 is therefore an experiment, not advice. Every experiment a reviewer requested that could be run in the period was run with criteria written beforehand; outcomes, including the failed predictions, are in Table 43. Open beyond this: Muon-pretrained histories and the 4B headroom (Table 1). App. E.4 and Table 43 (row JK1–JK4) are added; no table, figure or section number changed.

19 Nov: the abstract and Sec. 6 now state the Moonlight result (at AdamW's tuned rate, Moonlight-scaled Muon ends 5.5 points lower; App. E.4). Pages 4–9 and all appendices are unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.