- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper studies example-level differentially private optimization (DP-SGD / DP-Adam) with per-layer (group-wise) gradient clipping, where model parameters are partitioned into disjoint layer groups. Unlike flat (global) clipping (Abadi et al., 2016), per-layer clipping computes with and , allowing gradients to be clipped, aggregated, and noised layer-by-layer during backpropagation so intermediate activations can be freed immediately (He et al., 2023; Bu et al., 2023a). However, per-layer clipping faces two coupled challenges:
-
Because is determined independently per layer, heterogeneous coordinate and layer scales cause large dispersion in , rotating the concatenated gradient away from .
-
Gradient norms evolve across training steps and layers , requiring adaptive per-layer clipping scales without spending additional privacy budget on norm-quantile queries.
To address these challenges, the authors propose the following mechanisms:
-
Preconditioning (Section 3.1, Algorithm 1): Before clipping at step , each example's layer gradient is linearly transformed via , where combines a diagonal base preconditioner D_\{l,t\} = \{\rm\{diag\}\}((\sqrt\{\hat\{v\}_\{l,t-1\}\} + \epsilon_\{\text{num\}\})^\{-1\}) built from Adam's historical (already privatized) second-moment buffer with a scalar layer scale . After clipping and adding Gaussian noise in the preconditioned space, the noisy sum is mapped back via to form the original-space privatized gradient supplied to Adam.
-
Slack Encoding (Section 3.2, Algorithm 1): Keeping fixed, is updated multiplicatively via using SlaClip's -dimensional norm-slack encoding of (Zou et al., 2026). Because (Lemma D.2), jointly releasing preserves global sensitivity and incurs zero additional privacy loss (Theorem 4.1).
Critical strengths
The scale-then-privatize-then-invert formulation and its Rényi/Gaussian DP proof (Theorem 4.1, Appendix D) are clean and exact.
Empirically, SCP-Clip consistently improves over Fixed, Auto, and Adaptive per-layer clipping across RoBERTa-base/large (GLUE), ViT-small/large (CIFAR-10/100), and GPT-2/medium (E2E) with throughput and memory overhead over standard per-layer Book-Keeping (BK).
Critical weaknesses
First, the main paper and Algorithm 1 specify full coordinate-wise diagonal preconditioning , which breaks the implicit inner-product norm trick required by Ghost Clipping / Book-Keeping; buried in Appendix G, all experiments actually replace with a rank-1 Kronecker approximation in log space without mentioning or ablating this approximation in the main text.
Second, in the adaptive target rule , the authors divide the normalized slack coordinate by the fixed constant rather than the effective clipping threshold (unlike SlaClip, which divides by the adaptive threshold ); as a consequence, for small preconditioned norms (), the expected feedback has a positive derivative () and vanishes on a half-line when in the low-noise limit, violating the global stability condition (Condition 4) assumed by Theorem 4.2.
Third, Theorem 4.2 leaves on the right-hand side without analyzing the autoregressive noise-variance feedback loop in Adam's un-debiased second moment .
Questions for authors
-
Kronecker approximation () vs. full diagonal preconditioning (): Can you explicitly present the Kronecker-factored preconditioner from Appendix G in Section 3 so readers understand how SCP-Clip is actually implemented with Book-Keeping? On a model/setting where explicit per-example gradients fit in memory, how closely does match exact diagonal in cosine similarity (Figure 1) and test accuracy?
-
Fixing and Condition (4):
- In Algorithm 1 (line 16), should be divided by the effective clipping scale rather than the fixed ?
- How can Condition (4) hold on when is constant and is strictly decreasing for , causing to increase with (and vanish on when and )?
-
Bounding and noise amplification in : Can you provide conditions on \frac\{\sigma^2 C^2\}\{B^2 \rho_\{l,t\}^2} under which the noisy second-moment recurrence in Adam keeps bounded, or evaluate whether subtracting the known noise variance from when constructing improves performance at small privacy budgets ( on ViT-small / CIFAR-100)?
-
Hyperparameter details and statistical significance: What values of were used in Tables 2, 3, and 7, and does tuning the learning rate for Auto/Adaptive per-layer on CIFAR-100 change the rankings? Please also include 3-seed standard deviations in Tables 1, 2, 3, and 7.
-
Minor corrections:
- Page 4, line 185: "is the equal of the global clipping threshold" "is the equivalent global clipping threshold".
- Page 5, Algorithm 1, Line 1: Replace in with (or ).
- Page 5, Line 235 & Page 21, Eq. (12): Standardize the expression for between and .
- Page 8, lines 408-409: Qualify the statement "SCP-Clip maintains its advantage over Flat clipping" to reflect that Flat clipping outperforms SCP-Clip on CIFAR-100 at (ViT-small/large) and (ViT-small).
6 comments
Response to Reviewer SY3H (Part 2 / 4)
- [W2+Q2] Adaptive target and supplementary stability analysis
We sincerely thank the reviewer for this valuable comment. We provide clarification and supplementary analysis below.
The roles of and in the target design. Thank you for raising this question. First, we would like to clarify that SCP-Clip performs gradient clipping and slack encoding in the transformed space, where the clipping threshold remains fixed at . The threshold only represents the equivalent threshold obtained by mapping the same clipping region back to the base-preconditioned space; it is not the threshold actually used for slack encoding. Moreover, already enters the slack feedback through the preconditioned gradient norm. Replacing the denominator with would therefore introduce an additional explicit dependence on , changing the original feedback rule rather than simply correcting the normalization.
We provide further analysis of Condition (4) below and will include the details in the revised manuscript.
(1) Sufficient conditions for Condition (4). For fixed , Eq. (17) gives
\lim_{u\to-\infty}h_{l,t}(u) &=1-\mathbb E_t\\!\left[\Pi_{[0,1]}\\!\left( \frac{1+(1+\xi_{l,K})/C_l}{2}\right)\right]\>0,\\\\ \lim_{u\to+\infty}h_{l,t}(u)&=-\frac12. \end{aligned}$$ These limits follow from $A_{l,t},Q_{l,t}\to1$ on the left and $A_{l,t},Q_{l,t}\to0$ on the right. The Gaussian lower tail prevents complete target saturation on the left; symmetry gives an expected target of $1/2$ on the right. If the continuous feedback has a unique root $u^\star_{l,t}$ with $h'_{l,t}(u^\star_{l,t})<0$, the tail signs imply positive feedback to its left and negative feedback to its right. Consider $$\frac{-(u-u^\star_{l,t})h_{l,t}(u)} {\min\\{(u-u^\star_{l,t})^2,|u-u^\star_{l,t}|\\}}, \qquad u\ne u^\star_{l,t}.$$ This quotient tends to $-h'_{l,t}(u^\star_{l,t})\>0$ at the root and to positive limits at both tails. On the remaining compact intervals, continuity gives a positive minimum. Its positive infimum therefore establishes Eq. (4). **This permits $h'_{l,t}\>0$ in the small-norm region, provided that the region lies to the left of the equilibrium and the feedback remains positive. Global monotonic decrease is unnecessary.** **(2) No clipping occurs in the degenerate region.** For $K\ge2$, if $\sigma=0$, $A_{l,t}(u)=1$ and $Q_{l,t}(u)\ge C_l$ almost surely under conditional batch sampling, then $\gamma_{l,t}=1$ and $h_{l,t}(u)=0$. For $C_l<1$ and bounded norm support, this region includes a sufficiently negative half-line, so the global condition indeed cannot hold there. However, $A_{l,t}=1$ means that the first slack coordinate of every example equals one, and hence all gradients in that layer are unclipped. Thus, **in the zero-feedback interval identified by the reviewer, the clipping coefficient is one and there is no clipping bias**. **(3) Supplementary analysis of Condition (4).** The analysis in Appendix E of the original manuscript shows that Condition (4) is used only at the algorithm's current iterate $u_{l,t}$. It therefore suffices to have a common deterministic constant $\mu\>0$ such that, for all layers and iterations almost surely, $$(u_{l,t}-u^\star_{l,t})h_{l,t}(u_{l,t}) \le-\mu\min\\{(u_{l,t}-u^\star_{l,t})^2, |u_{l,t}-u^\star_{l,t}|\\}.$$ The condition is then required only at the actual iterates. If the current iterate lies in the unclipped region described above, choose $u^\star_{l,t}=u_{l,t}$. Since $h_{l,t}(u_{l,t})=0$, the iterate itself is an equilibrium, and $\alpha^\star_{l,t}=1$. Outside this region, choose an attracting equilibrium in the sense of the original analysis. To verify this extension, we use $e_{l,t}=u_{l,t}-u^\star_{l,t}$, the potential $\Psi$ and the feedback second-moment bound $M_l$ from Appendix E. The condition imposed only at the actual iterate already gives $\Psi'(e_{l,t})h_{l,t}(u_{l,t})\le-\mu\min\\{e_{l,t}^2,1\\}$. The subsequent potential-function analysis, scale-tracking error control and clipping-error bound can therefore follow the original proof, yielding the utility upper bound in Theorem 4.2. We will include the sufficient conditions above and the extension in point (3) in Appendix E.Response to Reviewer SY3H (Part 3 / 4)
- [W3+Q3] Noise feedback and bounded inverse preconditioning
We appreciate you raising this important concern. Noise amplification in the second-moment estimates can indeed occur. To address this formally, we analyze how noise enters the second moments and under what conditions this feedback can be controlled. We have added the complete analysis to the appendix of the revised manuscript.
Let denote the clipped batch gradient mapped back to the original space. Conditional on past private outputs, the current is fixed, and the newly added Gaussian noise is independent of the current batch. Therefore,
\tilde{g}_{l,t} &=c_{l,t}+\frac{\sigma C}{B}P_{l,t}^{-1}z_{l,t}, \qquad z_{l,t}\sim\mathcal{N}(0,I),\\\\ \mathbb{E}_t\lVert\tilde{g}_{l,t}\rVert_2^2 &=\mathbb{E}_t\lVert c_{l,t}\rVert_2^2 +\frac{\sigma^2C^2}{B^2}\lVert P_{l,t}^{-1}\rVert_F^2. \end{aligned}$$ The cross term in the second equation vanishes because the noise has zero conditional mean. Since $0\leq\alpha_{l,t}^{(i)}\leq1$, Jensen's inequality and the manuscript's gradient-moment assumption give $\mathbb{E}_t\lVert c_{l,t}\rVert_2^2\leq G_l^2$. Substituting this into $v_{l,t}=\beta_2v_{l,t-1}+(1-\beta_2)\tilde{g}_{l,t}^{\odot2}$ and using $\hat{v}_{l,t}=v_{l,t}/(1-\beta_2^t)$ yields, for $t\geq2$, $$\mathbb{E}_t\lVert\hat{v}_{l,t}\rVert_1 -\lVert\hat{v}_{l,t-1}\rVert_1 \leq \frac{1-\beta_2}{1-\beta_2^t} \left[ G_l^2-\lVert\hat{v}_{l,t-1}\rVert_1 +\frac{\sigma^2C^2}{B^2}\lVert P_{l,t}^{-1}\rVert_F^2 \right].$$ This recursion separates the two components of the feedback. The previous second moment decays, while the inverse-transformed noise increases the second moment again. For the exact diagonal preconditioner, the latter term is precisely $$\frac{\sigma^2C^2}{B^2}\lVert P_{l,t}^{-1}\rVert_F^2 =\left(\frac{\sigma C}{B\rho_{l,t}}\right)^2 \sum_j\left(\sqrt{\hat{v}_{l,t-1,j}}+\epsilon_{\mathrm{num}}\right)^2.$$ To isolate the strength of this feedback, temporarily set $\epsilon_{\mathrm{num}}=0$. The coefficient of the previous second moment in the recursion is then $1-\frac{1-\beta_2}{1-\beta_2^t}\left[1-\left(\frac{\sigma C}{B\rho_{l,t}}\right)^2\right]$. Thus, the factor $(\sigma C/(B\rho_{l,t}))^2$ identified by the reviewer determines whether noise feedback offsets the decay. **A sufficient condition for stability is that this feedback factor remains below 1 by a fixed margin**. Under this condition, decay dominates noise feedback as the second moments grow. Together with the manuscript's gradient-moment assumption and finite initialization, this ensures uniformly bounded expected second moments. The same condition also controls amplification under the inverse mapping, **keeping $\mathbb{E}\lVert P_{l,t}^{-1}\rVert_F^2$ uniformly bounded**. The positive numerical stabilizer $\epsilon_{\mathrm{num}}\>0$ used in practice introduces additional terms, but these grow more slowly and can be absorbed by the fixed margin. It therefore only requires a looser bound and does not change this sufficient condition. We have added the complete analysis to the revised manuscript. **Noise-variance correction.** Following your suggestion, we evaluate the effect of noise-variance correction on ViT-small / CIFAR-100. The only change is to replace the exponential moving average (EMA) of squared noisy gradients with an EMA computed after subtracting the known noise variance; all other settings remain unchanged. The additional results are shown below. > **R3-Table 3.** Effect of noise-variance correction on ViT-small / CIFAR-100. | Method | $\epsilon=2$ | $\epsilon=4$ | | --- | --- | --- | | Flat clipping | 81.88 ± 0.63 | 83.01 ± 0.33 | | SCP-Clip | 78.56 ± 0.29 | 80.31 ± 0.44 | | SCP-Clip + variance correction | 77.85 ± 0.38 | 79.97 ± 0.51 | As shown in the table, correcting for the known noise in the second-moment estimates does not improve utility and instead decreases accuracy. **This noise-variance correction is similar in spirit to the second-moment debiasing approach in DP-AdamBC [2]. Ganesh et al. [3], however, argue that obtaining an unbiased estimate of the second moment of noise-free gradients is not the right objective for designing private adaptive optimizers.** The optimizer operates on noisy gradients, and using a bias-corrected preconditioner to recover the geometry of noise-free gradients does not resolve its mismatch with the noisy gradients. Moreover, bias correction can make $\hat{v}_{l,t}$ negative, compromising optimization stability. These considerations may partially explain why noise-variance correction performs poorly in our additional experiments.
Response to Reviewer SY3H (Part 1 / 4)
We thank our reviewer for the time and effort dedicated to evaluating our work, as well as for the constructive comments and suggestions. Our detailed responses are provided below. We have also revised the paper to further improve clarity.
Thank you for your valuable comments. First, we'd like to clarify that the results in Tables 1–4 of the original manuscript use the Kronecker approximation DK, whereas Figures 1–2 use the exact diagonal preconditioner D. In response to the issues you raised, we describe the corresponding revisions and present additional experimental results and discussion below.
Explanation of the approximation to Dl,t in Section 3. We have added a paragraph entitled “Kronecker approximation of Dl,t” at the end of Section 3.2, after the paragraph “Effective clipping scale adaptation” and before Section 4. This paragraph explains the following.
The full diagonal preconditioner Dl,t scales each weight independently and generally cannot be represented by separate scalings of activations and output derivatives. For matrix parameters, we therefore reshape its diagonal scaling coefficients into a positive matrix S and approximate S by qr⊤ using least-squares fitting in log space. Under column-wise vectorization, the resulting preconditioner is
=\operatorname{diag}(r)\otimes\operatorname{diag}(q), \qquad P_{l,t}^{\mathrm K}=\rho_{l,t}D_{l,t}^{\mathrm K}.$$ For a linear layer with per-example weight gradient $G^{(i)}=(E^{(i)})^\top A^{(i)}$, where $A^{(i)}$ contains activations and $E^{(i)}$ contains output derivatives, we rescale these quantities as $$A'^{(i)}=A^{(i)}\operatorname{diag}(r), \qquad E'^{(i)}=\rho_{l,t}E^{(i)}\operatorname{diag}(q).$$ The transformed gradient retains the form $G'^{(i)}=(E'^{(i)})^\top A'^{(i)}$, so its squared norm can be computed as $$\lVert G'^{(i)}\rVert_F^2 =\left\langle A'^{(i)}(A'^{(i)})^\top, E'^{(i)}(E'^{(i)})^\top \right\rangle_F.$$ Thus, SCP-Clip does not require explicit construction of per-example weight gradients. The approximation occurs in $D_{l,t}\to D_{l,t}^{K}$; norm computation and clipping themselves are exact for the given factorized transform. Appendix G further discusses the construction of this approximation and its error relative to exact $D_{l,t}$. **Comparison of exact $D$ and $D^{\mathrm{K}}$.** We have added an end-to-end comparison of the two implementations on RoBERTa-base / SST-2 at $\epsilon=8$. The exact implementation streams explicit per-example gradients through small physical microbatches. > **R3-Table 1.** Accuracy comparison between $D$ and $D^{\mathrm{K}}$. | Implementation | Seed 42 | Seed 43 | Seed 44 | Mean ± standard deviation | | --- | --- | --- | --- | --- | | SCP-Clip, exact $D$ | 92.56 | 92.86 | 93.10 | 92.84 ± 0.27 | | SCP-Clip, Kronecker $D^{\mathrm{K}}$ | 92.43 | 92.80 | 93.11 | 92.78 ± 0.34 | The mean accuracies with exact $D$ and $D^{\mathrm{K}}$ are **92.84% and 92.78%**, respectively, differing by **0.06 percentage points**. Thus, the structural approximation has little effect on final task utility in this setting. We also compare the two implementations using all four metrics in Figures 1–2 of the original manuscript. Following Appendix F, we run these experiments under exactly the same settings as those used for Figures 1–2, replacing only exact $D$ with $D^{\mathrm{K}}$. > **R3-Table 2.** Comparison of the metrics in Figures 1–2 using $D$ and $D^{\mathrm{K}}$. | Diagnostic metric | Standard per-layer clipping | Exact $D$ | Kronecker $D^{\mathrm{K}}$ | | ---------------------------------------------- | --------------------------- | --------- | -------------------------- | | Cosine similarity to the unclipped gradient ↑ | 0.742 | 0.831 | 0.829 | | Relative clipping error ↓ | 0.784 | 0.576 | 0.581 | | Mean norm-to-threshold ratio | 1.48 | 1.23 | 1.26 | | Final loss from separate noise-free training ↓ | 0.195 | 0.140 | 0.142 | As shown in the table, using the Kronecker approximation $D^{\mathrm{K}}$ causes only small changes in diagnostic metrics such as cosine similarity, while retaining the main improvements in gradient direction and reductions in clipping error. We will include these additional results in the revised manuscript.