acceptodds
Under review as a conference paper at ICLR 2027

DP-Muon: Differentially Private Optimization via Matrix-Orthogonalized Momentum

Abstract

We study differentially private (DP) training with Muon, a matrix-valued optimizer that updates hidden-layer weights using momentum followed by Newton-Schulz orthogonalization. While DP-SGD is well understood, the interaction between per-example clipping, Gaussian noise, momentum, and nonlinear orthogonalization in Muon has not been systematically analyzed. We formulate DP-Muon, a private Muon procedure that clips per-example gradients globally, adds Gaussian noise to the clipped minibatch average, and then applies momentum and Newton-Schulz orthogonalization as post-processing. We prove that DP-Muon inherits the privacy guarantee certified by standard subsampled-Gaussian accounting, with no additional privacy cost from Muon-specific post-processing. On the optimization side, we establish finite-horizon and vanishing stationarity guarantees under global clipping, with bounds that separate optimization error, clipping residual, privacy noise, and Newton-Schulz approximation error. We further show that Gaussian privacy noise, although centered before the nonlinear Newton-Schulz map, induces a systematic bias in the expected update after this nonlinear transformation. This motivates DP-MuonBC, an adaptive bias-corrected variant designed to cancel the leading bias induced by the current Gaussian perturbation while preserving the same privacy guarantee. Experiments on E2E at four privacy budgets show that the tested Muon configurations improve test NLL over Adam baselines, with further gains for the selected DP-MuonBC configuration.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
DP-Muon, Differential privacy, optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XWM2OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a DP variant of the so called muon algorithm, that updates weight matrices of models (mainly applied to LLMs) with updates that are orthogonal products UVTUV^T, where UU and VV are orthogonal matrices.

The first algorithm, Alg. 1 is a plain / naive version of DP-muon, i.e. do DP-SGD gradient computation, then do the orthogonalization as post-processing (Netwon-Schultz iteration).

Algorithm 2 has certain post-processing on top of Alg. 1, where additional noise averaging is carried to remove the bias generated by the plain DP muon (Alg. 1).

Critical strengths
  • timely and interesting topic, to consider a DP variant of this popular algorithm
Critical weaknesses
  • The claim that "we prive the privacy guarantees" even given in the abstract is oversell and a very small contribution/novelty since the DP-SGD guarantees hold trivially since all the orthgonalization operations are just post-processing

  • there are no theoretical "DP utility results" since they are stated in terms of the noise parameter σ\sigma, not any privacy guarantee like DP ε\varepsilon. The DP guarantees can go arbitrarily large in those bounds.

  • There is only one model (GPT-2) and one dataset. I think a study of "DP muon" should include an extensive experimental comparison.

  • Looking at the metrics given in the appendix, the proposed muon method loses to DP-Adam in several metrics (ROUGE metrics e.g.).

  • In the Table 1, the main experimental result of the main text, does not have averaging over several seeds nor has any error bars. It is hard to judge the results (especially with that narrow margins).

Questions for authors

minor comments:

Alg. 1: kappa is defined in the appendix. moreover, it should be integer, right? since it is the order of the polynomial. The way it is written now makes the impression it is scalar.

line 10: equation not aligned with the for loop

line 10: isn't that squared Frobenius norm the same without that mathcal(T)? I.e., it is redundant

Alg. 2:

What is q, what is kappa?

Line 12: O_t^{(2})} is not inside the for loop (not aligned)

Line 259: that σ\sigma apparently refers to the sigma algebra, however is not defined (and clashes with DP sigma notation)

2 comments

We thank the reviewer for the direct feedback on the theoretical claims, empirical evidence, and presentation.

W1. Contribution of the Privacy Result

We agree that privacy follows from standard accounting and post-processing. Theorem 2.1 specifies the joint mechanism, including global clipping over all trainable parameters and a single Gaussian release shared by matrix and auxiliary updates. We will describe this result as inheritance of the underlying mechanism's privacy guarantee. The main theoretical contributions are the nonlinear conditional output-bias analysis, its correction, and the stationarity bounds, rather than a new privacy-accounting principle.

W2. Fixed Privacy Budgets and Stationarity

We agree that fixed-noise-multiplier asymptotics do not establish convergence at a fixed total privacy budget. This restriction is stated after Corollary 4.4, and we will emphasize it more clearly.

For fixed horizon TT, sampling probability γ\gamma, and target (ε⋆,δ)(\varepsilon_\star,\delta), choose σ⋆\sigma_\star such that a valid accountant for the actual mechanism certifies εacc(T,γ,σ⋆,δ)≤ε⋆\varepsilon_{\mathrm{acc}}(T,\gamma,\sigma_\star,\delta)\le\varepsilon_\star. The variance term in Theorem 4.1 then becomes

vC2(ε⋆,δ)=C2B+dC2σ⋆2B2.v_C^2(\varepsilon_\star,\delta)=\frac{C^2}{B}+\frac{dC^2\sigma_\star^2}{B^2}.

Substitution gives a finite-horizon stationarity bound at the certified privacy budget, with all trajectory-dependent quantities evaluated for that run. The same substitution applies to Theorem 4.2 when bλ<1−εqbcb_\lambda<1-\varepsilon_q^{\mathrm{bc}}. These statements retain the theorems' smooth-map assumptions. We do not claim a vanishing or optimal fixed-total-privacy rate, or that these results directly cover the practical Jordan-map implementation.

W3. Empirical Scope

We agree that GPT-2 on E2E provides a limited empirical evaluation. The current results support conclusions about this setting, while broader applicability requires additional evidence. We will make the limited scope explicit.

W4. Generation Metrics

We agree that lower NLL does not imply uniformly better generation metrics. Table 4 shows that DP-AdamBC outperforms both Muon variants in ROUGE-L at ε≈4,8\varepsilon\approx4,8. Table 3 also shows that DP-MuonBC trails DP-Muon in BLEU at ε≈1,2,8\varepsilon\approx1,2,8. All configurations were selected using validation NLL, and generation metrics were additional evaluations. We will limit any empirical advantage claim to the metrics and settings where it is observed and acknowledge the mixed generation-metric rankings.

W5. Three-Seed Comparison and Error Bars

We evaluated all five methods at ε≈1,4\varepsilon\approx1,4 using seeds 0,1,20,1,2. Test NLL (mean ±\pm sample SD) is:

Methodε≈1\varepsilon\approx1ε≈4\varepsilon\approx4
DP-SGD (extended LR search)0.586666±0.0016310.586666\pm0.0016310.516988±0.0012850.516988\pm0.001285
DP-Adam0.569400±0.0011250.569400\pm0.0011250.530977±0.0024830.530977\pm0.002483
DP-AdamBC0.570692±0.0007410.570692\pm0.0007410.528505±0.0021760.528505\pm0.002176
DP-Muon0.565070±0.0019560.565070\pm0.0019560.524436±0.0019030.524436\pm0.001903
DP-MuonBC0.564605±0.0018370.564605\pm0.0018370.523921±0.0019070.523921\pm0.001907

DP-MuonBC achieves the lowest mean test NLL at ε≈1\varepsilon\approx1, while DP-SGD performs best at ε≈4\varepsilon\approx4 after extending its learning-rate search. We will update the main comparison and revise our empirical claims accordingly. These results remain limited to one model/task and three seeds.

Minor Comments. Algorithm Notation and Presentation

  • qq and κ\kappa. We will declare q,κ∈Z≥1q,\kappa\in\mathbb Z_{\ge1} in both algorithms. Here qq is the number of Newton-Schulz iterations and κ\kappa is the degree of pκp_\kappa. We will place the polynomial definition at its first use.
  • Transpose in the normalization. The reviewer is correct that ∥TW(M)∥F=∥M∥F\|\mathcal T_W(M)\|_F=\|M\|_F. We will simplify the denominator while retaining the numerator's transpose convention.
  • Loop indentation. Algorithm 1, line 10, and Algorithm 2, line 12, are both inside the loop over matrix blocks. We will fix the displayed equations' indentation to make that scope clear.
  • Line 259. Here σ(⋅)\sigma(\cdot) denotes a generated sigma-algebra. We will instead define Gt\mathcal G_t in words as the sigma-algebra generated by the complete past, current sampled lot, and current gradient-evaluation randomness, excluding current privacy noise and probes, and reserve σ\sigma for the noise multiplier.

Thank you for the rebuttal. These address my questions though the limitation of experimental comparisons remains. That answer to W2, expressing utility results in terms of ε\varepsilon instead of σ\sigma and TT, is satisfactory.

Reviewer_Gb9GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies differentially private training with Muon, employing global per-example clipping and a joint Gaussian release. It analyzes the bias induced by passing privacy noise through the nonlinear Newton-Schulz map and introduces DP-MuonBC. Under the matched λ=1\lambda=1 setting, DP-MuonBC eliminates the leading O(ν2)O(\nu^2) conditional bias, resulting in an O(ν4)O(\nu^4) residual. The paper also derives stationarity bounds separating clipping, privacy noise, momentum, and Newton-Schulz approximation errors. Empirically, the tested Muon methods outperform Adam baselines in test NLL on GPT-2 fine-tuning on the E2E dataset. The selected DP-MuonBC configuration provides a modest further improvement.

Critical strengths
  • The paper identifies a nontrivial bias arising when centered DP noise passes through the nonlinear Newton-Schulz map, and shows that the λ=1\lambda=1 correction cancels the leading O(ν2)O(\nu^2) term.
  • The privacy formulation is clean: a single globally clipped Gaussian release is followed by post-processing.
  • The stationarity analysis distinguishes among clipping bias, privacy noise, momentum tracking, and Newton-Schulz approximation error, and explicitly states that the vanishing bounds are derived under the assumption of a fixed noise multiplier.
Critical weaknesses

The experiments do not cleanly isolate the effect of bias correction.

In the main experimental comparison, DP-Muon uses the practical Jordan map, while DP-MuonBC uses a Smooth-NS variant, so their comparison changes both the matrix map and the correction. Moreover, the fourth-order cancellation result applies to matched λ=1\lambda=1, whereas the selected configuration uses λ=1.25\lambda=1.25. A cleaner ablation would compare Smooth-NS with λ∈{0,1,1.25}\lambda\in\{0,1,1.25\} under matched settings.

The empirical evidence is limited.

The main comparison uses a single seed. The multi-seed DP-Muon vs. DP-MuonBC comparison uses only three seeds, while the reported gains are small relative to run-to-run variation. In addition, the evaluation is restricted to a single model (GPT-2) and a single dataset (E2E), limiting evidence of generalization across tasks or model settings.

Questions for authors
  • Can you provide a same-map Smooth-NS ablation with λ∈{0,1,1.25} under matched settings to isolate the effect of bias correction?

  • Can you provide stronger statistical evidence for the DP-MuonBC gains, e.g., more paired seeds and confidence intervals for the test-NLL differences?

  • Could you expand the DP-SGD learning-rate search, since the best reported learning rate lies at the upper boundary of the search range?

2 comments

We thank the reviewer for identifying the need for a controlled ablation and stronger statistical evidence.

W1/Q1. Matched Smooth-NS Ablation and Bias-Correction Guarantee

We compared λ∈0,1,1.25\lambda\in\\{0,1,1.25\\} on the same Smooth-NS map at ε≈4\varepsilon\approx4, with matrix learning rate 0.0080.008 and seeds 0,1,20,1,2. Their mean test NLLs are 0.5238740.523874, 0.5239010.523901, and 0.5239210.523921, respectively. For the paired test-NLL difference (λ=0\lambda=0 minus λ=1\lambda=1), the mean is −2.69×10−5-2.69\times10^{-5} and the nominal 95% paired tt-interval is [−7.90×10−5,2.52×10−5][-7.90\times10^{-5},2.52\times10^{-5}]. Thus, these runs do not establish a test-NLL benefit from correction. The common learning rate was selected for λ=1.25\lambda=1.25, and the interval is based on only three seeds.

The fourth-order cancellation guarantee applies to matched λ=1\lambda=1. For the validation-selected λ=1.25\lambda=1.25, the leading conditional bias is −ρt2ΔQ(Ht)/8-\rho_t^2\Delta Q(H_t)/8, which is generically second order, as already noted in Appendix B.7. We will distinguish this guarantee from the empirical configuration and avoid attributing the practical Jordan versus Smooth-NS performance difference solely to correction.

W2/Q2. Multi-Seed Uncertainty

At ε≈4\varepsilon\approx4, across seeds 0,1,20,1,2, DP-Muon and DP-MuonBC achieve test NLLs of 0.524436±0.0019030.524436\pm0.001903 and 0.523921±0.0019070.523921\pm0.001907 (mean ±\pm sample SD), respectively. The mean difference (DP-Muon minus DP-MuonBC) is 5.15×10−45.15\times10^{-4}, with a nominal 95% paired tt-interval of [2.43×10−4,7.87×10−4][2.43\times10^{-4},7.87\times10^{-4}]. However, since the two configurations differ in their matrix maps and learning rates, this comparison does not isolate the effect of bias correction.

Q3. Extended DP-SGD Tuning

We extended the DP-SGD learning-rate search at all four budgets and selected 0.88,0.52,1.04,1.920.88,0.52,1.04,1.92 for ε≈1,2,4,8\varepsilon\approx1,2,4,8, respectively, using validation NLL. The next tenfold candidates performed worse. At ε≈4\varepsilon\approx4, DP-SGD obtains 0.516988±0.0012850.516988\pm0.001285 test NLL over three seeds, below practical DP-Muon (0.524436±0.0019030.524436\pm0.001903) and DP-MuonBC (0.523921±0.0019070.523921\pm0.001907). At ε≈8\varepsilon\approx8, seed-0 DP-SGD also outperforms both Muon configurations. We will update the comparison and empirical claims accordingly.

W2. Generalization Across Settings

We agree that the present GPT-2/E2E evaluation does not establish generality across tasks or models. We will state this limitation explicitly and restrict the conclusions to the evaluated setting.

Thanks to the authors for their response !

I will briefly summarize how the authors' response addressed the weaknesses:

  • The results for matched Smooth-NS ablation do not show an improvement from bias correction. The uncorrected configuration (λ=0\lambda = 0) performs slightly better than λ=1\lambda = 1 and λ=1.25\lambda = 1.25. The practical DP-Muon and DP-MuonBC configurations also use different matrix maps and learning rates.

This weakness has not been resolved.

  • The multi-seed results and confidence intervals are helpful. However, the evaluation still uses only three seeds, and the observed gains cannot be attributed solely to bias correction.

This weakness has been partially addressed.

  • The authors extend the DP-SGD learning-rate search. The updated results show that DP-SGD outperforms both Muon configurations at ε≈4\varepsilon \approx 4 and ε≈8\varepsilon \approx 8 in the seed-0 comparison.

This concern has been resolved.

  • The authors acknowledge the limited experimental scope. However, the evaluation remains restricted to one model (GPT-2) and one dataset (E2E).

This weakness has not been resolved.

Overall, I appreciate the authors' efforts and clarifications. However, the main concerns regarding the empirical effectiveness of bias correction and limited generalization remain.

I am maintaining my original rating (2: Weak rejection).

Reviewer_G2zwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper formulates DP-Muon by applying global per-example clipping and a joint Gaussian release, followed by matrix momentum and finite-step Newton–Schulz orthogonalization as post-processing. It analyzes the conditional output bias induced when fresh Gaussian noise passes through the nonlinear Newton–Schulz map and proposes DP-MuonBC, an antithetic-probe correction intended to cancel the leading conditional bias without additional privacy cost. The paper derives stationarity bounds for a Muon-managed matrix block and a hybrid matrix/auxiliary-optimizer setting, and evaluates GPT-2 fine-tuning on E2E under several privacy budgets.

Critical strengths

(1) Analysis of nonlinear post-processing bias. The Gaussian heat-semigroup argument provides a principled explanation of why zero-mean input noise can induce nonzero conditional output bias after a nonlinear matrix map. (2) Decomposition of private optimization errors. The theoretical analysis separates clipping bias, privacy noise, momentum tracking, and finite-step Newton–Schulz approximation error.

Critical weaknesses

(1) The empirical comparison does not isolate the effect of bias correction. DP-Muon and DP-MuonBC use different matrix maps and different correction configurations. The missing uncorrected Smooth-NS (λ=0\lambda=0) baseline prevents attribution of the observed difference to the correction itself. The reported NLL gains are small, and the selected λ=1.25\lambda=1.25 is not the λ=1\lambda=1 setting for which the leading O(ρ2)O(\rho^2) term is cancelled. (2) The empirical scope and baseline tuning is not general. The evaluation uses one model/task combination, and the DP-SGD learning-rate selection reaches the upper boundary of the search grid. This weakens the strength of the claimed utility advantage.

Questions for authors

(1) The fourth-order residual-bias result is stated for λ=1\lambda=1. What is the precise bias order for the experimentally selected λ=1.25\lambda=1.25. (2) Could the DP-SGD learning-rate search be extended beyond the current upper boundary? (3) Would the method be tested on more additional tasks or models?

1 comment

We thank the reviewer for the careful assessment and constructive questions.

W1. Same-Map Ablation and Attribution of Bias-Correction Gains

We conducted a matched Smooth-NS ablation at ε≈4\varepsilon\approx4, keeping the matrix learning rate at 0.0080.008 and using seeds 0,1,20,1,2. The test NLL results are:

Correction strengthTest NLL (mean ±\pm sample SD)
λ=0\lambda=00.523874±0.0019330.523874\pm0.001933
λ=1\lambda=10.523901±0.0019120.523901\pm0.001912
λ=1.25\lambda=1.250.523921±0.0019070.523921\pm0.001907

The same-map comparison does not show a test-NLL improvement from bias correction. The reported difference between practical DP-Muon and DP-MuonBC also changes the matrix map and learning rate, so it cannot be attributed to correction alone. We will clarify this distinction and limit the empirical claim accordingly.

W2/Q2. Extended DP-SGD Learning-Rate Search

We extended the DP-SGD search beyond its previous upper boundaries. The validation-selected learning rates and seed-0 test NLLs are:

Privacy budgetSelected learning rateDP-SGD test NLL
ε≈1\varepsilon\approx10.880.880.5878060.587806
ε≈2\varepsilon\approx20.520.520.5449350.544935
ε≈4\varepsilon\approx41.041.040.5184490.518449
ε≈8\varepsilon\approx81.921.920.4988040.498804

For each budget, the next tenfold learning rate gave worse validation NLL, resolving the previous boundary issue within this coarse search. DP-SGD now outperforms both Muon configurations at ε≈4,8\varepsilon\approx4,8 in the seed-0 comparison. We will revise our utility claims to reflect this result.

Q1. Bias Order at λ=1.25\lambda=1.25

The fourth-order cancellation result applies to λ=1\lambda=1 with probes matched to the fresh Gaussian noise scale at the matrix-map input. Writing Q=Qκ,qQ=Q_{\kappa,q}, the general-strength expansion is

E[Otλ∣Gt]−Q(Ht)=1−λ2ρt2ΔQ(Ht)+O(ρt4).\mathbb E[O_t^\lambda\mid\mathcal G_t]-Q(H_t)=\frac{1-\lambda}{2}\rho_t^2\Delta Q(H_t)+O(\rho_t^4).

Consequently, at λ=1.25\lambda=1.25,

E[Ot1.25∣Gt]−Q(Ht)=−ρt28ΔQ(Ht)+O(ρt4).\mathbb E[O_t^{1.25}\mid\mathcal G_t]-Q(H_t)=-\frac{\rho_t^2}{8}\Delta Q(H_t)+O(\rho_t^4).

The residual is therefore generically second order. Appendix B.7 already states this distinction. We will emphasize it in the main text and describe λ=1.25\lambda=1.25 as a validation-selected extrapolation strength. The fourth-order guarantee concerns conditional output bias from the current perturbation, not improved final test NLL.

W2/Q3. Scope of the Evaluation

We agree that one model and dataset cannot establish broad empirical applicability. Our evidence concerns GPT-2 fine-tuning on E2E. We will keep the empirical conclusions specific to this setting and identify evaluation across additional tasks and models as an outstanding limitation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.