acceptodds
Under review as a conference paper at ICLR 2027

Muon for RLVR: What Works, Why, and What It Learns

Abstract

Muon is widely adopted for pretraining large language models and has been extended to supervised fine-tuning, but it is rarely used for reinforcement learning (RL), where reports of its value are mixed. We ask whether Muon can be used for RL with verifiable rewards (RLVR), and if so, how does Muon make a difference. With the right recipe, Muon extends the useful training horizon: it keeps improving after Adam-series optimizers plateau and reaches higher accuracy on both an AdamW-pretrained base (Qwen3-4B-Base, 29.17% versus 24.47% macro-averaged Pass@1 against AdamW) and a Muon-pretrained one (Moonlight-16B-A3B-Instruct, 32.60% versus 31.29% macro-averaged Pass@1 against SignSGD). We identify two critical ingredients. First, the matrix passed to Newton–Schulz should be a single linear operator: orthogonalizing fused QKV projections per head rather than as one stored tensor is what lifts Muon above AdamW on Qwen3-4B-Base, because splitting keeps the cross-head agreement already present in the gradient, which fused orthogonalization whitens away. Second, momentum should be removed, because RLVR offers it no persistent direction to average, Newton–Schulz's scale invariance turns the resulting correlation into extra displacement that AdamW would damp, and on-policy sampling makes that displacement unrecoverable. Finally, Muon, like AdamW, learns off-the-principal directions of the base weights. Muon's endpoint, however, is less compressible and relies on the attention value projections for accuracy where Adam-series optimizers do not.

Primary area:
reinforcement learning
Keywords:
optimizer, muon, rlvr

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4CmeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper evaluates how to extend muon to RLVR with several modifactons to make it work. First, split qkv on adam ckpts. Second, remove momentum from the Muon. It performs better than adam on the tested experiments, though it is a multi-epoch setting. Also some anlysis between Muon and AdamW ckpts are provided.

Critical strengths
  • The paper identifies a useful and consequential implementation choice for Muon in RLVR.
  • The further study of Muon ckpts is interesting and rebute the wrong claims from recent rank-1 papers which i totally agree.
Critical weaknesses

Though, still many key weakness remains:

  • The split qkv contribution is trivial and natural when you do Muon on ckpts, given their way different learning dynamics
  • The author didn't compare with Adam with 0 momentum, which cannot fully ablate the source of benefit
  • Still, the baselines we are comparing I feel it is less optimized. E.g., Adam's lr need to retuned, and why we keep the update norm is smimilar can ensure it is a fair comparison
  • How should the claim that Muon relies on value projections be interpreted in light of Table 7?
Questions for authors

Some questions:

  • Can you compare with Adam with beta1=0?
  • Can you explain why the Adam is collapsing?
  • Why we have weight decay here for RL run? and How is it set for Muon?

One more big question would be: the current setup would epoch the dataset multiple times, I don't think it is generally good for RL

3 comments
AuthorsOpenReview ↗

Response to Reviewer 4Cme

Original review

Thank you for the detailed review and for highlighting the importance of the optimizer controls and the interpretation of our ablations.

1. Why the QKV partition matters

The split qkv contribution is trivial and natural when you do Muon on ckpts

We agree that per-head QKV splitting is a natural choice with precedent in pretraining. In our Qwen3-4B-Base RLVR comparison, changing the orthogonalization boundary increases Pass@1 from 22.8% with fused Muon to 29.2% with split Muon, with AdamW at 24.5% (Table 2). Fused orthogonalization couples heads through a shared Gram matrix, while splitting preserves cross-head alignment and changes the spectrum and norm allocation of the update (Table 4). We will clarify that our contribution is to measure this effect in RLVR and explain the geometry behind it.

2. AdamW with β₁ = 0

Can you compare with Adam with beta1=0?

Yes. AdamW with β₁ = 0 started more slowly and achieved similar average accuracy across the observed trajectory. We ran it on Qwen3-4B-Base with β₂ = 0.98 and the same learning rate (1e-6) and weight decay (0.1) as standard AdamW. We used the paper's 64k evaluation protocol and measured macro Pass@1 across AIME24, AIME25, and BeyondAIME every 100 steps (revised Appendix C.1, Figure 10).

ReferenceSteps 99–999Step 1099 to last shared stepAll shared steps
AdamW (β₁ = 0.9, β₂ = 0.98), through step 2999−1.9+0.6−0.2
SignSGD (β₁ = 0, β₂ = 0), through step 2599−1.5+1.1+0.1

Entries are the mean paired differences in macro Pass@1 (percentage points), computed as the β₁ = 0, β₂ = 0.98 arm minus each reference. The accuracy gap seen on the 16k in-training AIME24 evaluation was absent on AIME24 under the 64k protocol. Each arm has one run. The ablation used tensor parallelism of 1 (references used 2) and had about twice standard AdamW's update norm at the same learning rate.

3. Learning rate and update scale

Adam's lr need to retuned, and why we keep the update norm is smimilar can ensure it is a fair comparison

We used the standard AdamW learning rate of 1e-6 for our RLVR setup and scaled Muon's matrix updates so that the observed update norms were on the same order of magnitude. This controls for large differences in step size. A separate learning-rate sweep for each optimizer would address sensitivity to the chosen rates.

4. Interpreting Table 7

How should the claim that Muon relies on value projections be interpreted in light of Table 7?

Table 7 reports two interventions. Adding the value-projection update alone to the base model recovers 24% of split Muon's accuracy gain, the highest among the tested families. Removing it from the final checkpoint loses 3% of the gain, versus 11% for the query projection and 10% for the MLP down projection. We will revise the claim to say that value projections have the strongest single-family sufficiency for recovering split Muon's accuracy gains, while coordinate-wise optimizers show a different distribution of functional contributions.

5. Late AdamW evaluation decline

Can you explain why the Adam is collapsing?

AdamW's rollout reward largely plateaus late in training while held-out accuracy declines. One possible explanation is that the policy specializes to the training distribution as reward saturates. The observed curves leave the cause open.

6. Weight decay

Why we have weight decay here for RL run? and How is it set for Muon?

We used a weight decay coefficient of 0.1 for Muon and the other arms to keep this setting fixed across optimizers. We leave a comparison across coefficients to future work.

7. Repeated prompts across epochs

the current setup would epoch the dataset multiple times, I don't think it is generally good for RL

Yes, prompts recur over 3,000 steps of training on DAPO-Math-17k. Each rollout batch uses 32 prompts and eight sampled responses per prompt. When a prompt recurs, we sample fresh trajectories from the current rollout policy, which is updated during training.

Thank for the reviewer's detailed feedback. Some concerns remain:

  • Still, I feel AdamW's lr is properly untuned, which is unfair as baseline
  • Also, the baseline's collapse is still not explained. If it is due to overfitting as the multi-epoch setting, the author could try some other dataset instead of the DAPO.

So i would keep my assessment at this stage

We thank the reviewer for the follow-up.

1. Learning rate

We use the commonly adopted RLVR learning rate of 1e-6, following default settings used in common for RLVR post-training, eg, [1] - [3]. To make a fair comparison, we also calibrate Muon's scaling to keep the measured update norms on the same scale of magnitude as Adam's on each step. And further, additional learning-rate sweeps would further strengthen the comparison, and we leave this exploration to future work.

2. Additional datasets and tasks

We appreciate the suggestion to test other datasets. The remaining rebuttal time is insufficient to complete these experiments. We plan to explore additional datasets and tasks, such as coding and agentic tasks, to assess how well the recipe generalizes beyond mathematical reasoning.

References

[1] DAPO: An Open-Source LLM Reinforcement Learning System at Scale. https://arxiv.org/abs/2503.14476

[2] Do We Need Adam? Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMs. https://arxiv.org/abs/2602.07729

[3] Rethinking Muon Beyond Pretraining: Spectral Failures and High-Pass Remedies for VLA and RLVR. https://arxiv.org/abs/2605.19282

Reviewer_uDJFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors introduce additional recipes for using Muon in RLVR setups. Their ingredients include (i) splitting QKV weights per head before the NS iterations, (ii) using no momentum, and (iii) calibrating the learning rate to match the update norm of AdamW. Their experiments reveal that the modified Muon optimizer outperforms conventional baselines on both AdamW- and Muon-pretrained base models.

Critical strengths
  1. The proposed recipe provides a simple modification to the Muon optimizer without introducing additional computational overhead.
  2. The explanations of why per-head splitting and the absence of momentum are effective are well-organized and easy to follow.
Critical weaknesses
  1. The experiments are primarily based on two models: Qwen3-4B-Base and Moonlight-16B-A3B-Instruct. More extensive experiments on other base model families would further support the paper's claims.
  2. One concern is that the proposed recipe, including splitting fused QKV weights, using per-head mode, and removing momentum, has already been reported in language model training, which may limit the novelty of this paper [GLM-5 Team, 2026; Qwen Team, 2026; Mukherjee et al., 2026]. Although this paper provides some new analyses of each component of the recipe, the authors could more explicitly discuss the novelty of their contributions.
  • GLM-5 Team, GLM-5: from vibe coding to agentic engineering, 2026
  • Qwen Team, On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability, 2026
  • Mukherjee et al., Do We Need Adam? Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMs, ICML 2026
Questions for authors
  1. The performance gap between Muon-fused and Muon-split becomes smaller when using a Muon-pretrained base model as initialization. Do you have any intuition as to why the choice of initialization affects the training dynamics during RLVR? Why does Muon-fused achieve performance comparable to Muon-split when performing RLVR on Muon-pretrained models?
2 comments
AuthorsOpenReview ↗

Response to Reviewer uDJF

Original review

Thank you for the thoughtful review and for asking how the recipe relates to prior work and transfers across models.

1. Generalization across models and tasks

More extensive experiments on other base model families would further support the paper's claims.

We agree that testing more base models would help establish how broadly the recipe applies. To extend the momentum comparison beyond Qwen3, we added a momentum ablation on the Muon-pretrained Moonlight-16B-A3B-Instruct model. Figure 12 in revised Appendix C.3 shows the split-Muon and AdamW–SignSGD curves. The results below average the final five logged AIME24 evaluations for each momentum run and compare its reference over the same rollouts (30 problems, 8 samples per problem, 14,848-token cap).

ComparisonRolloutsAIME24 accuracyReference accuracyTruncation (run / reference)
Split Muon, μ = 0.9 vs. μ = 0699–7793.1%10.1%49% / 18%
Fused Muon, μ = 0.9 vs. μ = 0499–5795.0%8.7%46% / 16%
AdamW vs. SignSGD639–7196.3%6.0%43% / 19%

On Moonlight, the accuracy gap for Muon with momentum coincides with higher truncation. The split-Muon training reward recovered after a transient dip. An infrastructure interruption ended the momentum runs early, so these comparisons cover the observed rollouts under the in-training AIME24 protocol. Given the time and compute available for this revision, we leave a broader evaluation across model families and task domains to future work.

2. Relationship to prior work

the authors could more explicitly discuss the novelty of their contributions.

Our contribution is to establish a working Muon recipe for RLVR post-training and explain how its components change the applied update. GLM-5 and Qwen3.8-Next provide evidence for head-wise splitting in pretraining. We measure its effect on RLVR stability and accuracy and reconstruct split and fused updates from the same optimizer input to quantify the effect of the partition. Mukherjee et al. study momentum for Adam and SGD in RLVR. A new empirical finding of this work is that momentum can destabilize split Muon in RLVR, with early reward gains followed by collapse in our matched Qwen3-4B-Base comparison at μ = 0.9. We examine how momentum interacts with Muon's nonlinear Newton–Schulz transform. Momentum can correlate successive updates, while orthogonalization can preserve their scale even when gradients disagree.

3. The smaller split–fused gap on Moonlight

Why does Muon-fused achieve performance comparable to Muon-split when performing RLVR on Muon-pretrained models?

One possibility is that fused RLVR updates are better matched to a base pretrained with fused Muon. Whether Moonlight used head-wise splitting in pretraining is unspecified, and its attention architecture and training protocol also differ from Qwen3's. The observed gap therefore describes these checkpoints and recipes. Differences in the base models' capabilities may also help explain why splitting yields a smaller gain on Moonlight.

Thank you for addressing my questions. I will maintain my positive score.

Reviewer_Ek2tOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper talks about the best practice of using Muon to do post-training, and what structure of weight does Muon-postraining learn. They show that once a correct recipe, which means split-head Muon without momentum, is chosen, then Muon is better than Adam on both Muon-trained and Adam-trained based models. They argue theoretically about why momentum in Muon is potentially harmful in RLVR and found models learned by Muon to be harder to compress.

Critical strengths
  1. This paper studies a important empirical question and shed some light on the loss landscape of post-training.

  2. The empirical results are solid and convincing.

  3. The intuitive explaination of the algorithm choice is plausible.

Critical weaknesses
  1. The argument that why mometum underperforms are not fully supported with experiments. The near independence of gradient is shown but the argument that scale invariance of Muon hurts is not well grounded as te effect could potentially be addressed with a lower learning rate. Also it is observed that momentum leads to gain early, which is not explained.

  2. The authors didn't seem to discuss potential effect of consistent gradient bias caused by potential train inference mismatch, which may be another plausible explaination of why momentum hurts.

Questions for authors
  1. Could the authors find either more empirical evidence or some simple theory or synthetic examples to demonstrate their empirical argument that Muon's momentum will hurt in the on-policy RL setting?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Ek2t

Original review

Thank you for the constructive review and for pointing out where the evidence for our momentum explanation could be stronger.

1. Muon with momentum at a lower learning rate

the effect could potentially be addressed with a lower learning rate.

Also it is observed that momentum leads to gain early, which is not explained.

Reducing the learning rate from 1e-6 to 2.5e-7 stabilized training for split Muon with μ = 0.9 over the 3,000 steps we observed on Qwen3-4B-Base. At 1e-6, smoothed training reward fell below 0.3 around step 850. At 2.5e-7, the run tracked the momentum-free reference in training reward from around step 500 (revised Appendix C.2, Figure 11).

The lower-rate run still scored below momentum-free Muon on the in-training AIME24 evaluation with a 16k-token cap and 16 samples per problem. We report mean accuracy within each step window (revised Appendix C.2, Table 10).

Training stepsμ = 0, LR 1e-6μ = 0.9, LR 2.5e-7μ = 0.9, LR 1e-6
1–49915.3%11.2%19.7%
500–99925.5%18.1%14.5%
1000–149927.5%21.2%3.7% (through step 1099)
1500–199929.2%21.1%—
2000–249930.4%20.4%—
2500–299931.5%21.7%—

The lower-rate run was evaluated every 100 steps and the references every 20 steps. One possible explanation for the early gains at the original learning rate is that momentum reduces directional noise when the policy has a relatively stable direction of improvement. As the policy changes, useful directions may persist for fewer steps, and accumulated momentum may align less well with the current improvement direction.

2. Training–inference mismatch

potential effect of consistent gradient bias caused by potential train inference mismatch

We thank the reviewer for raising this possibility. Numerical differences between the rollout and training engines can arise even with fully on-policy rollouts. A persistent mismatch could bias the gradient estimates, and momentum could carry that bias across optimization steps.

Figure 5 shows that functional displacement scales approximately diffusively without momentum and that momentum adds short-range correlation between functional updates. In Figure 9, a model with no fitted parameters reproduces the observed displacement scaling by applying an exponential moving average to independent, zero-mean directions and normalizing each update. The measured scaling can therefore arise under unbiased gradient directions.

Mismatch-induced bias could still contribute to the eventual performance decline. We will clarify that the model accounts for the measured displacement scaling and explicitly acknowledge training–inference mismatch as a possible additional source of instability.

3. Further evidence for the proposed mechanism

Could the authors find either more empirical evidence or some simple theory or synthetic examples to demonstrate their empirical argument that Muon's momentum will hurt in the on-policy RL setting?

We thank the reviewer for suggesting a synthetic test. Appendix B.3 (Figure 9) presents a model with no fitted parameters. It applies an exponential moving average with momentum coefficient μ to independent, zero-mean gradient directions and then normalizes each update to unit norm, modeling the scale invariance of Muon's orthogonalization.

The moving average induces short-range correlation even for independent inputs. For stationary update increments, let ρ(k) be the lag-k autocorrelation and V(L) the expected squared displacement over L increments divided by the expected squared norm of one increment. Equation (4) gives

V(L)=L+2∑k=1L−1(L−k)ρ(k).V(L) = L + 2∑_{k=1}^{L-1}(L-k)ρ(k).

The positive correlations at short lags increase displacement above the diffusive baseline. In Figure 5, the measured functional displacement exponent is 0.507 without momentum and 0.611 with μ = 0.9. The normalized EMA model reproduces this change in Figure 9. Each measured increment spans 20 optimizer steps.

In on-policy RLVR, each policy update changes the distribution of subsequent rollouts. Momentum mixes gradient directions from earlier policies into the current update, and Muon's normalization can maintain the step scale as those directions become less useful. This feedback could help explain why momentum improves training reward early and degrades later. The model explains the extra displacement over the measured windows. Learning-rate calibration and training–inference mismatch may also affect whether it leads to a performance drop. We will make the connection between Appendix B.3 and the training behavior in Section 5 explicit in the revision.

I appreciate the detailed response. I suppose the acceptance of this paper and will keep my positive score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised manuscript and additional experiments

We thank the reviewers for their constructive feedback. We have uploaded a revised manuscript with three additional experiments in Appendix C.

  • AdamW without first-moment averaging (Appendix C.1, Figure 10). Setting β₁ = 0 while keeping β₂ = 0.98 led to a slower start and similar average macro Pass@1 across the training run under the 64k evaluation protocol.
  • Split Muon with momentum at a lower learning rate (Appendix C.2, Figure 11 and Table 10). We observed no training reward collapse over 3,000 steps at the lower learning rate. The run still scored below momentum-free split Muon on the 16k in-training AIME24 evaluation.
  • Momentum on Moonlight-16B-A3B-Instruct (Appendix C.3, Figure 12). On this Muon-pretrained model, split Muon with momentum showed lower in-training AIME24 accuracy and higher response truncation than its momentum-free counterpart over the same late rollout window.

We discuss these results in more detail in our individual replies.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.