acceptodds
Withdrawn from ICLR 2027

Inner Momentum for Differentially Private Muon

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Differentially private training clips each per-example gradient before adding noise. This clipping is radial for each example, yet unequal clipping factors can distort the relative singular-vector geometry of their average. Muon is particularly exposed to this effect, since its update is an approximate polar factor that depends only on the singular vectors that clipping can shift. To curb this degradation, we propose averaging each sampled example's Muon gradient over the current model and a short history of recent models before clipping. The clipped batch matrix then separates into a common rescaling and a covariance residual between sampled gradients and clipping values, with , bounding the clipping-induced distortion directly. We further show that a finite Newton–Schulz iteration preserves the polar factor of its input under these spectral conditions, confirming that our correction survives orthogonalization. In private GPT-2 fine-tuning on E2E and DART at 1,2,4,8, DP-Muon-IM improves BLEU and ROUGE-L over DP-Muon in every seed-matched comparison, and non-private diagnostics show 2-4% lower pre-noise polar error.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Differential privacy, private optimization, Muon, gradient clipping, inner momentum

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_w4aqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The authors propose using the Inner Momentum (DP-Muon-IM) to reduce the adverse effects of per-sample gradient clipping in Muon-DP. They analyze how clipping affects Muon from both theoretical and empirical perspectives, and show that using the Inner Momentum can improve DP-Muon. Experiments show the advantages of DP-Muon-IM across multiple datasets.

Critical strengths
  1. The paper studies an important issue for differentially private Muon. The theoretical analysis is clear.
  2. The proposed method DP-Muon-IM is conceptually simple and shows better performance than DP-Muon.
Critical weaknesses
  1. Novelty concern: as mentioned by the authors, the core mechanism (Inner Momentum) has already been proposed in prior work. This weakens the contribution.

  2. A major concern is the practice of DP-Muon-IM. Previously, Muon is mainly adopted in the pretraining setting. However, DP-Muon-IM requires multiple additional gradient computations over historical model states. Such overhead (K×K\times than regular Muon) seems to be too expensive, especially in pre-training. The experiment is also limited to fine-tuning GPT-2-Small on two tasks, without evaluating the optimizer in pre-training settings with different scales and architectures (e.g. LLaMA). Moreover, the paper does not clearly specify the exact pretrained GPT-2-small checkpoint used in the experiments. This is important since if the initialization was pretrained using AdamW/Adam, subsequently fine-tuning it with Muon may introduce additional optimizer-mismatch considerations discussed in [1].

[1]Qu, Xingyu, Peigeng Huang, and Samuel Horvath. "Can Muon Fine-tune Adam-Pretrained Models?." arXiv preprint arXiv:2605.10468 (2026).

Questions for authors

See weaknesses.

Reviewer_y8LDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper analyzes how per-example clipping changes the geometry used by DP-Muon and proposes DP-Muon-IM, which averages each sampled example's gradients over current and historical private model states before clipping. The analysis separates a common scaling term from a clipping residual and explains why subsequent Newton–Schulz processing cannot generally restore the polar-factor direction lost in its input. Experiments on private GPT-2 fine-tuning for E2E and DART report generation-quality improvements across four privacy budgets, together with geometry diagnostics and history-length costs.

Critical strengths
  1. The clipping-residual perspective is useful for understanding the interaction between per-example clipping and Muon's matrix update. The privacy argument clearly conditions on the previously released private states and retains clipping of the resulting per-example query. The paper distinguishes its residual bounds from a guarantee of lower realized error.

  2. Table 2 reports mean and standard deviation over three seeds for two datasets and four budgets, with positive mean BLEU/ROUGE-L differences throughout. Training time, memory, achieved privacy parameters, and the limitations of the pre-noise geometry probes are disclosed, which makes the tradeoffs assessable.

Critical weaknesses
  1. The experiments do not yet establish that preservation of Muon's polar-factor geometry explains the utility gains. In Table 5, BLEU is 63.66 without history, 63.76 with Muon-only history, 64.37 with AdamW-only history, and 65.30 with both. The large interaction prevents attributing the improvement simply to either group; it also means the Muon-only result alone cannot be used to conclude that the Muon branch is irrelevant. Enabling history only for the AdamW group improves BLEU substantially, but this does not localize the gain to those parameters: full-query clipping can also change the clipping factors applied to Muon blocks. No non-Muon optimizer with the same inner momentum is evaluated. The geometric probes also use a different batch size, history decay, optimizer-group configuration, and training trajectory from the main utility experiment, and stop before noise and outer momentum. They demonstrate a diagnostic effect, but do not identify the mechanism behind the reported utility gains.

  2. The incremental utility has not been assessed at comparable resource and tuning budgets. In the history-length sweep, runtime rises from 36.9 to 123.7 minutes for K=4, and peak allocation from 26.70 to 49.85 GiB; these are costs of the reported sweep, not a measured universal multiplier for every selected configuration. The main comparison lacks an equal-compute DP-Muon control at the same privacy budget, or a similarly evaluated non-Muon inner-momentum baseline. In addition, Table 1 reports per-metric maxima over seeds while Table 2 reports means; the maxima are disclosed in the text, but mixing these with published and single-seed numbers makes that table unsuitable as a clean superiority comparison. The method extends an existing inner-momentum technique, so these controls are important for determining the significance of the extension.

Overall assessment: I recommend rejection in its current form. The reported improvement is consistent and the analysis is informative, but the present evaluation does not isolate the proposed Muon-specific mechanism or establish the value of the extension at a comparable resource budget. These are consequential issues for the paper's central contribution. A convincing optimizer-group analysis and resource-matched comparison could change my assessment.

Questions for authors
  1. What evidence links the final noisy update geometry to utility in the actual utility-training regime, rather than only in the diagnostic regime?
  2. Can you clarify the Table 5 interaction across optimizer groups, provide the seed count, and compare a non-Muon optimizer using the same history averaging?
  3. At equal privacy and compute budgets, how does DP-Muon-IM compare with additional baseline optimization? Please report the cost of the actual selected configurations as well as the sweep.
  4. Please replace best-over-seeds comparisons in Table 1 with comparable means or clearly separate the protocols, and provide the per-seed results supporting all 24 paired improvements.
  5. Could averaging repeated current-state gradient queries with fresh dropout masks explain part of the gain? A same-state, equal-query control would isolate historical-state information from stochastic averaging. Please also state the tuning budgets per method.
Reviewer_Q4A6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper observes that clipping breaks an assumption that the Muon optimiser makes, and proposes a to average per-example gradients from several iterations to mitigate the issue and improve Muon with differential privacy. The paper proves that this averaging reduces an upper bound on the error from clipping, and contains experiments where the change improves DP-Muon on two GPT-2 finetuning tasks.

Critical strengths

None

Critical weaknesses

The main weakness of the paper is the experiments. The baselines do not include DP-Adam, DP-AdamW or DP-SGD, which are much more common variants than DP-Muon. Only one model is finetuned and the chosen model, GPT-2, is fairly old at this point.

The paper does not explain how the hyperparameters were chosen. The experiments use a fixed clip bound of 1, which is unlikely to be optimal. Optimising the clipping bound may lead to different behaviour, which is especially important to investigate since the paper's motivation is related to clipping. There are also some discrepancies with the batch size, stated in lines 682-886. Specifically, the text states that "the nominal logical batch size" is 1024, but also says that the subsampling rate gives different expected batch sizes, which round down to 1025 or 1027, depending on the dataset.

The difference between Tables 1 and 2 needs to be clarified. Both tables report results for E2E at epsilon = 8, but the numbers do not match.

As a minor point, Section 2.1 needs a citation for Muon's operation.

Questions for authors

Do the experiments use Poisson subsampling in practice, or fixed-size batches?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.