- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes a DP variant of the so called muon algorithm, that updates weight matrices of models (mainly applied to LLMs) with updates that are orthogonal products , where and are orthogonal matrices.
The first algorithm, Alg. 1 is a plain / naive version of DP-muon, i.e. do DP-SGD gradient computation, then do the orthogonalization as post-processing (Netwon-Schultz iteration).
Algorithm 2 has certain post-processing on top of Alg. 1, where additional noise averaging is carried to remove the bias generated by the plain DP muon (Alg. 1).
Critical strengths
- timely and interesting topic, to consider a DP variant of this popular algorithm
Critical weaknesses
-
The claim that "we prive the privacy guarantees" even given in the abstract is oversell and a very small contribution/novelty since the DP-SGD guarantees hold trivially since all the orthgonalization operations are just post-processing
-
there are no theoretical "DP utility results" since they are stated in terms of the noise parameter , not any privacy guarantee like DP . The DP guarantees can go arbitrarily large in those bounds.
-
There is only one model (GPT-2) and one dataset. I think a study of "DP muon" should include an extensive experimental comparison.
-
Looking at the metrics given in the appendix, the proposed muon method loses to DP-Adam in several metrics (ROUGE metrics e.g.).
-
In the Table 1, the main experimental result of the main text, does not have averaging over several seeds nor has any error bars. It is hard to judge the results (especially with that narrow margins).
Questions for authors
minor comments:
Alg. 1: kappa is defined in the appendix. moreover, it should be integer, right? since it is the order of the polynomial. The way it is written now makes the impression it is scalar.
line 10: equation not aligned with the for loop
line 10: isn't that squared Frobenius norm the same without that mathcal(T)? I.e., it is redundant
Alg. 2:
What is q, what is kappa?
Line 12: O_t^{(2})} is not inside the for loop (not aligned)
Line 259: that apparently refers to the sigma algebra, however is not defined (and clashes with DP sigma notation)
2 comments
Thank you for the rebuttal. These address my questions though the limitation of experimental comparisons remains. That answer to W2, expressing utility results in terms of instead of and , is satisfactory.
We thank the reviewer for the direct feedback on the theoretical claims, empirical evidence, and presentation.
W1. Contribution of the Privacy Result
We agree that privacy follows from standard accounting and post-processing. Theorem 2.1 specifies the joint mechanism, including global clipping over all trainable parameters and a single Gaussian release shared by matrix and auxiliary updates. We will describe this result as inheritance of the underlying mechanism's privacy guarantee. The main theoretical contributions are the nonlinear conditional output-bias analysis, its correction, and the stationarity bounds, rather than a new privacy-accounting principle.
W2. Fixed Privacy Budgets and Stationarity
We agree that fixed-noise-multiplier asymptotics do not establish convergence at a fixed total privacy budget. This restriction is stated after Corollary 4.4, and we will emphasize it more clearly.
For fixed horizon T, sampling probability γ, and target (ε⋆,δ), choose σ⋆ such that a valid accountant for the actual mechanism certifies εacc(T,γ,σ⋆,δ)≤ε⋆. The variance term in Theorem 4.1 then becomes
vC2(ε⋆,δ)=BC2+B2dC2σ⋆2.
Substitution gives a finite-horizon stationarity bound at the certified privacy budget, with all trajectory-dependent quantities evaluated for that run. The same substitution applies to Theorem 4.2 when bλ<1−εqbc. These statements retain the theorems' smooth-map assumptions. We do not claim a vanishing or optimal fixed-total-privacy rate, or that these results directly cover the practical Jordan-map implementation.
W3. Empirical Scope
We agree that GPT-2 on E2E provides a limited empirical evaluation. The current results support conclusions about this setting, while broader applicability requires additional evidence. We will make the limited scope explicit.
W4. Generation Metrics
We agree that lower NLL does not imply uniformly better generation metrics. Table 4 shows that DP-AdamBC outperforms both Muon variants in ROUGE-L at ε≈4,8. Table 3 also shows that DP-MuonBC trails DP-Muon in BLEU at ε≈1,2,8. All configurations were selected using validation NLL, and generation metrics were additional evaluations. We will limit any empirical advantage claim to the metrics and settings where it is observed and acknowledge the mixed generation-metric rankings.
W5. Three-Seed Comparison and Error Bars
We evaluated all five methods at ε≈1,4 using seeds 0,1,2. Test NLL (mean ± sample SD) is:
DP-MuonBC achieves the lowest mean test NLL at ε≈1, while DP-SGD performs best at ε≈4 after extending its learning-rate search. We will update the main comparison and revise our empirical claims accordingly. These results remain limited to one model/task and three seeds.
Minor Comments. Algorithm Notation and Presentation