- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper evaluates how to extend muon to RLVR with several modifactons to make it work. First, split qkv on adam ckpts. Second, remove momentum from the Muon. It performs better than adam on the tested experiments, though it is a multi-epoch setting. Also some anlysis between Muon and AdamW ckpts are provided.
Critical strengths
- The paper identifies a useful and consequential implementation choice for Muon in RLVR.
- The further study of Muon ckpts is interesting and rebute the wrong claims from recent rank-1 papers which i totally agree.
Critical weaknesses
Though, still many key weakness remains:
- The split qkv contribution is trivial and natural when you do Muon on ckpts, given their way different learning dynamics
- The author didn't compare with Adam with 0 momentum, which cannot fully ablate the source of benefit
- Still, the baselines we are comparing I feel it is less optimized. E.g., Adam's lr need to retuned, and why we keep the update norm is smimilar can ensure it is a fair comparison
- How should the claim that Muon relies on value projections be interpreted in light of Table 7?
Questions for authors
Some questions:
- Can you compare with Adam with beta1=0?
- Can you explain why the Adam is collapsing?
- Why we have weight decay here for RL run? and How is it set for Muon?
One more big question would be: the current setup would epoch the dataset multiple times, I don't think it is generally good for RL
3 comments
Thank for the reviewer's detailed feedback. Some concerns remain:
- Still, I feel AdamW's lr is properly untuned, which is unfair as baseline
- Also, the baseline's collapse is still not explained. If it is due to overfitting as the multi-epoch setting, the author could try some other dataset instead of the DAPO.
So i would keep my assessment at this stage
We thank the reviewer for the follow-up.
1. Learning rate
We use the commonly adopted RLVR learning rate of 1e-6, following default settings used in common for RLVR post-training, eg, [1] - [3]. To make a fair comparison, we also calibrate Muon's scaling to keep the measured update norms on the same scale of magnitude as Adam's on each step. And further, additional learning-rate sweeps would further strengthen the comparison, and we leave this exploration to future work.
2. Additional datasets and tasks
We appreciate the suggestion to test other datasets. The remaining rebuttal time is insufficient to complete these experiments. We plan to explore additional datasets and tasks, such as coding and agentic tasks, to assess how well the recipe generalizes beyond mathematical reasoning.
References
[1] DAPO: An Open-Source LLM Reinforcement Learning System at Scale. https://arxiv.org/abs/2503.14476
[2] Do We Need Adam? Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMs. https://arxiv.org/abs/2602.07729
[3] Rethinking Muon Beyond Pretraining: Spectral Failures and High-Pass Remedies for VLA and RLVR. https://arxiv.org/abs/2605.19282
Response to Reviewer 4Cme
Original review
Thank you for the detailed review and for highlighting the importance of the optimizer controls and the interpretation of our ablations.
1. Why the QKV partition matters
We agree that per-head QKV splitting is a natural choice with precedent in pretraining. In our Qwen3-4B-Base RLVR comparison, changing the orthogonalization boundary increases Pass@1 from 22.8% with fused Muon to 29.2% with split Muon, with AdamW at 24.5% (Table 2). Fused orthogonalization couples heads through a shared Gram matrix, while splitting preserves cross-head alignment and changes the spectrum and norm allocation of the update (Table 4). We will clarify that our contribution is to measure this effect in RLVR and explain the geometry behind it.
2. AdamW with β₁ = 0
Yes. AdamW with β₁ = 0 started more slowly and achieved similar average accuracy across the observed trajectory. We ran it on Qwen3-4B-Base with β₂ = 0.98 and the same learning rate (1e-6) and weight decay (0.1) as standard AdamW. We used the paper's 64k evaluation protocol and measured macro Pass@1 across AIME24, AIME25, and BeyondAIME every 100 steps (revised Appendix C.1, Figure 10).
Entries are the mean paired differences in macro Pass@1 (percentage points), computed as the β₁ = 0, β₂ = 0.98 arm minus each reference. The accuracy gap seen on the 16k in-training AIME24 evaluation was absent on AIME24 under the 64k protocol. Each arm has one run. The ablation used tensor parallelism of 1 (references used 2) and had about twice standard AdamW's update norm at the same learning rate.
3. Learning rate and update scale
We used the standard AdamW learning rate of 1e-6 for our RLVR setup and scaled Muon's matrix updates so that the observed update norms were on the same order of magnitude. This controls for large differences in step size. A separate learning-rate sweep for each optimizer would address sensitivity to the chosen rates.
4. Interpreting Table 7
Table 7 reports two interventions. Adding the value-projection update alone to the base model recovers 24% of split Muon's accuracy gain, the highest among the tested families. Removing it from the final checkpoint loses 3% of the gain, versus 11% for the query projection and 10% for the MLP down projection. We will revise the claim to say that value projections have the strongest single-family sufficiency for recovering split Muon's accuracy gains, while coordinate-wise optimizers show a different distribution of functional contributions.
5. Late AdamW evaluation decline
AdamW's rollout reward largely plateaus late in training while held-out accuracy declines. One possible explanation is that the policy specializes to the training distribution as reward saturates. The observed curves leave the cause open.
6. Weight decay
We used a weight decay coefficient of 0.1 for Muon and the other arms to keep this setting fixed across optimizers. We leave a comparison across coefficients to future work.
7. Repeated prompts across epochs
Yes, prompts recur over 3,000 steps of training on DAPO-Math-17k. Each rollout batch uses 32 prompts and eight sampled responses per prompt. When a prompt recurs, we sample fresh trajectories from the current rollout policy, which is updated during training.