- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The authors propose using the Inner Momentum (DP-Muon-IM) to reduce the adverse effects of per-sample gradient clipping in Muon-DP. They analyze how clipping affects Muon from both theoretical and empirical perspectives, and show that using the Inner Momentum can improve DP-Muon. Experiments show the advantages of DP-Muon-IM across multiple datasets.
Critical strengths
- The paper studies an important issue for differentially private Muon. The theoretical analysis is clear.
- The proposed method DP-Muon-IM is conceptually simple and shows better performance than DP-Muon.
Critical weaknesses
-
Novelty concern: as mentioned by the authors, the core mechanism (Inner Momentum) has already been proposed in prior work. This weakens the contribution.
-
A major concern is the practice of DP-Muon-IM. Previously, Muon is mainly adopted in the pretraining setting. However, DP-Muon-IM requires multiple additional gradient computations over historical model states. Such overhead ( than regular Muon) seems to be too expensive, especially in pre-training. The experiment is also limited to fine-tuning GPT-2-Small on two tasks, without evaluating the optimizer in pre-training settings with different scales and architectures (e.g. LLaMA). Moreover, the paper does not clearly specify the exact pretrained GPT-2-small checkpoint used in the experiments. This is important since if the initialization was pretrained using AdamW/Adam, subsequently fine-tuning it with Muon may introduce additional optimizer-mismatch considerations discussed in [1].
[1]Qu, Xingyu, Peigeng Huang, and Samuel Horvath. "Can Muon Fine-tune Adam-Pretrained Models?." arXiv preprint arXiv:2605.10468 (2026).
Questions for authors
See weaknesses.