acceptodds
Under review as a conference paper at ICLR 2027

Scale-Calibrated Preconditioning for Layer-Wise Clipping under Differential Privacy

Abstract

Recent advances have improved the efficiency and utility of differentially private training for large vision and language models. Per-layer gradient clipping is an important approach to reducing memory overhead. However, clipping each layer independently with its own threshold can produce different scaling factors across layers, distorting the full-gradient direction and degrading utility. We propose SCP-Clip, which performs per-layer clipping in a scale-calibrated preconditioned space to jointly mitigate directional distortion caused by per-layer clipping. The method incurs no additional privacy cost while retaining the efficiency advantages of per-layer clipping. Theoretically, we establish a utility bound on the private gradient estimation error under stable feedback. Experiments on pretrained language and vision models show SCP-Clip outperforms flat clipping in most settings, with accuracy gains of up to 4.28% over adaptive per-layer clipping.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Differential Privacy, Per-Layer Gradient Clipping, Adaptive Preconditioning, Clipping Bias, Private Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_SY3HOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper studies example-level differentially private optimization (DP-SGD / DP-Adam) with per-layer (group-wise) gradient clipping, where model parameters θ=[θ1;… ;θL]∈Rd\theta = [\theta_1; \dots; \theta_L] \in \mathbb{R}^d are partitioned into LL disjoint layer groups. Unlike flat (global) clipping gˉ(i)=min⁡(1,C/∥g(i)∥2)g(i)\bar{g}^{(i)} = \min(1, C / \|g^{(i)}\|_2) g^{(i)} (Abadi et al., 2016), per-layer clipping computes gˉl(i)=αl(i)gl(i)\bar{g}_l^{(i)} = \alpha_l^{(i)} g_l^{(i)} with αl{(i)}=min⁡(1,Cl/∥gl{(i)}∥2)\alpha_l^\{(i)\} = \min(1, C_l / \|g_l^\{(i)\}\|_2) and ∑{l=1}LCl2=C2\sum_\{l=1\}^L C_l^2 = C^2, allowing gradients to be clipped, aggregated, and noised layer-by-layer during backpropagation so intermediate activations can be freed immediately (He et al., 2023; Bu et al., 2023a). However, per-layer clipping faces two coupled challenges:

  1. Because αl(i)\alpha_l^{(i)} is determined independently per layer, heterogeneous coordinate and layer scales cause large dispersion in αl(i)l=1L\\{\alpha_l^{(i)}\\}_{l=1}^L, rotating the concatenated gradient gˉ(i)=[α1(i)g1(i);… ;αL(i)gL(i)]\bar{g}^{(i)} = [\alpha_1^{(i)} g_1^{(i)}; \dots; \alpha_L^{(i)} g_L^{(i)}] away from g(i)g^{(i)}.

  2. Gradient norms evolve across training steps tt and layers ll, requiring adaptive per-layer clipping scales without spending additional privacy budget on norm-quantile queries.

To address these challenges, the authors propose the following mechanisms:

  1. Preconditioning (Section 3.1, Algorithm 1): Before clipping at step tt, each example's layer gradient gl{(i)}∈{R}{dl}g_l^\{(i)\} \in \mathbb\{R\}^\{d_l\} is linearly transformed via {^g}l{(i)}=P{l,t}gl{(i)}\hat\{g\}_l^\{(i)\} = P_\{l,t\} g_l^\{(i)\}, where P{l,t}=ρ{l,t}D{l,t}P_\{l,t\} = \rho_\{l,t\} D_\{l,t\} combines a diagonal base preconditioner D_\{l,t\} = \{\rm\{diag\}\}((\sqrt\{\hat\{v\}_\{l,t-1\}\} + \epsilon_\{\text{num\}\})^\{-1\}) built from Adam's historical (already privatized) second-moment buffer {^v}{l,t−1}\hat\{v\}_\{l,t-1\} with a scalar layer scale ρ{l,t}>0\rho_\{l,t\} > 0. After clipping {ˉg}l{(i)}={{Clip}}({^g}l{(i)},Cl)\bar\{g\}_l^\{(i)\} = \{\rm\{Clip\}\}(\hat\{g\}_l^\{(i)\}, C_l) and adding Gaussian noise in the preconditioned space, the noisy sum is mapped back via P{l,t}{−1}P_\{l,t\}^\{-1\} to form the original-space privatized gradient {~g}l={1}{B}∑{i∈{B}t}min⁡(1,Cl/∥ρ{l,t}D{l,t}gl{(i)}∥2)gl{(i)}+{1}{B}P{l,t}{−1}{N}(0,σ2C2I{dl})\tilde\{g\}_l = \frac\{1\}\{B\} \sum_\{i \in \mathbb\{B\}_t\} \min(1, C_l / \|\rho_\{l,t\} D_\{l,t\} g_l^\{(i)\}\|_2) g_l^\{(i)\} + \frac\{1\}\{B\} P_\{l,t\}^\{-1\} \mathcal\{N\}(0, \sigma^2 C^2 I_\{d_l\}) supplied to Adam.

  2. Slack Encoding (Section 3.2, Algorithm 1): Keeping Cl=C0/LC_l = C_0/\sqrt{L} fixed, ρl,t\rho_{l,t} is updated multiplicatively via ρ{l,t+1}=ρ{l,t}exp⁡(ηρ({^s}{l,1}−γ{l,t}))\rho_\{l,t+1\} = \rho_\{l,t\} \exp(\eta_\rho(\hat\{s\}_\{l,1\} - \gamma_\{l,t\})) using SlaClip's KK-dimensional norm-slack encoding sl{(i)}s_l^\{(i)\} of max⁡(Cl−∥P{l,t}gl{(i)}∥2,0)\max(C_l - \|P_\{l,t\} g_l^\{(i)\}\|_2, 0) (Zou et al., 2026). Because ∥{ˉg}l{(i)}∥22+∥sl{(i)}∥22≤Cl2\|\bar\{g\}_l^\{(i)\}\|_2^2 + \|s_l^\{(i)\}\|_2^2 \le C_l^2 (Lemma D.2), jointly releasing [{ˉg}l{(i)},sl{(i)}][\bar\{g\}_l^\{(i)\}, s_l^\{(i)\}] preserves global ℓ2\ell_2 sensitivity C/BC/B and incurs zero additional privacy loss (Theorem 4.1).

Critical strengths

The scale-then-privatize-then-invert formulation and its Rényi/Gaussian DP proof (Theorem 4.1, Appendix D) are clean and exact.

Empirically, SCP-Clip consistently improves over Fixed, Auto, and Adaptive per-layer clipping across RoBERTa-base/large (GLUE), ViT-small/large (CIFAR-10/100), and GPT-2/medium (E2E) with <2<2\\% throughput and memory overhead over standard per-layer Book-Keeping (BK).

Critical weaknesses

First, the main paper and Algorithm 1 specify full coordinate-wise diagonal preconditioning D{l,t}={⁡diag}(({{^v}{l,t−1}}+ϵ{{num}}){−1})D_\{l,t\} = \operatorname\{diag\}((\sqrt\{\hat\{v\}_\{l,t-1\}\} + \epsilon_\{\text\{num\}\})^\{-1\}), which breaks the implicit inner-product norm trick required by Ghost Clipping / Book-Keeping; buried in Appendix G, all experiments actually replace D{l,t}D_\{l,t\} with a rank-1 Kronecker approximation D{l,t}K={⁡diag}(r)⊗{⁡diag}(q)D_\{l,t\}^K = \operatorname\{diag\}(r) \otimes \operatorname\{diag\}(q) in log space without mentioning or ablating this approximation in the main text.

Second, in the adaptive target rule γ{l,t}=Π{[0,1]}((1+{^s}{l,K}/Cl)/2)\gamma_\{l,t\} = \Pi_\{[0,1]\}((1 + \hat\{s\}_\{l,K\}/C_l)/2), the authors divide the normalized slack coordinate {^s}{l,K}∈[0,1]\hat\{s\}_\{l,K\} \in [0,1] by the fixed constant Cl=C0/{L}C_l = C_0/\sqrt\{L\} rather than the effective clipping threshold Cl/ρ{l,t}C_l / \rho_\{l,t\} (unlike SlaClip, which divides by the adaptive threshold CtC_t); as a consequence, for small preconditioned norms (rl(x;u)<1/Kr_l(x; u) < 1/K), the expected feedback h{l,t}(u)h_\{l,t\}(u) has a positive derivative (h{′l,t}(u)>0h'_\{l,t\}(u) > 0) and vanishes on a half-line (−∞,u{min⁡}](-\infty, u_\{\min\}] when Cl≤1C_l \le 1 in the low-noise limit, violating the global stability condition (Condition 4) assumed by Theorem 4.2.

Third, Theorem 4.2 leaves {E}∥P{l,t}{−1}∥F2\mathbb\{E\}\|P_\{l,t\}^\{-1\}\|_F^2 on the right-hand side without analyzing the autoregressive noise-variance feedback loop in Adam's un-debiased second moment {^v}{l,t}\hat\{v\}_\{l,t\}.

Questions for authors
  1. Kronecker approximation (Dl,tKD_{l,t}^K) vs. full diagonal preconditioning (Dl,tD_{l,t}): Can you explicitly present the Kronecker-factored preconditioner Dl,tK=diag⁡(r)⊗diag⁡(q)D_{l,t}^K = \operatorname{diag}(r) \otimes \operatorname{diag}(q) from Appendix G in Section 3 so readers understand how SCP-Clip is actually implemented with Book-Keeping? On a model/setting where explicit per-example gradients fit in memory, how closely does Dl,tKD_{l,t}^K match exact diagonal Dl,tD_{l,t} in cosine similarity (Figure 1) and test accuracy?

  2. Fixing γ{l,t}\gamma_\{l,t\} and Condition (4):

    • In Algorithm 1 (line 16), should {^s}{l,K}\hat\{s\}_\{l,K\} be divided by the effective clipping scale Cl/ρ{l,t}C_l / \rho_\{l,t\} rather than the fixed Cl=C0/{L}C_l = C_0/\sqrt\{L\}?
    • How can Condition (4) hold on u∈{R}u \in \mathbb\{R\} when A{l,t}(u)≡1A_\{l,t\}(u) \equiv 1 is constant and Q{l,t}(u)Q_\{l,t\}(u) is strictly decreasing for max⁡irl(xi;u)<1/K\max_i r_l(x_i; u) < 1/K, causing h{l,t}(u)h_\{l,t\}(u) to increase with uu (and vanish on (−∞,u{min⁡}](-\infty, u_\{\min\}] when Cl≤1C_l \le 1 and σ→0\sigma \to 0)?
  3. Bounding {E}∥Pl,t{−1}∥F2\mathbb\{E\}\|P_{l,t}^\{-1\}\|_F^2 and noise amplification in {^v}{l,t}\hat\{v\}_\{l,t\}: Can you provide conditions on \frac\{\sigma^2 C^2\}\{B^2 \rho_\{l,t\}^2} under which the noisy second-moment recurrence in Adam keeps {E}∥P{l,t}{−1}∥F2\mathbb\{E\}\|P_\{l,t\}^\{-1\}\|_F^2 bounded, or evaluate whether subtracting the known noise variance {σ2C2}{B2}ρ{l,t}{−2}D{l,t}{−2}\frac\{\sigma^2 C^2\}\{B^2\} \rho_\{l,t\}^\{-2\} D_\{l,t\}^\{-2\} from {~g}{l,t}2\tilde\{g\}_\{l,t\}^2 when constructing D{l,t+1}D_\{l,t+1\} improves performance at small privacy budgets (ϵ=2\epsilon = 2 on ViT-small / CIFAR-100)?

  4. Hyperparameter details and statistical significance: What values of C0C_0 were used in Tables 2, 3, and 7, and does tuning the learning rate for Auto/Adaptive per-layer on CIFAR-100 change the rankings? Please also include 3-seed standard deviations in Tables 1, 2, 3, and 7.

  5. Minor corrections:

    • Page 4, line 185: "is the equal of the global clipping threshold" →\to "is the equivalent global clipping threshold".
    • Page 5, Algorithm 1, Line 1: Replace η1\eta_1 in H=(η1,β1,β2,ϵnum)\mathcal{H} = (\eta_1, \beta_1, \beta_2, \epsilon_{\text{num}}) with ηt\eta_t (or η\eta).
    • Page 5, Line 235 & Page 21, Eq. (12): Standardize the expression for γ{l,t}\gamma_\{l,t\} between Π{[0,1]}(1−(1−{^s}{l,K}/Cl)/2)\Pi_\{[0,1]\}(1 - (1 - \hat\{s\}_\{l,K\}/C_l)/2) and Π{[0,1]}((1+{^s}{l,K}/Cl)/2)\Pi_\{[0,1]\}((1 + \hat\{s\}_\{l,K\}/C_l)/2).
    • Page 8, lines 408-409: Qualify the statement "SCP-Clip maintains its advantage over Flat clipping" to reflect that Flat clipping outperforms SCP-Clip on CIFAR-100 at ϵ=2\epsilon=2 (ViT-small/large) and ϵ=4\epsilon=4 (ViT-small).
6 comments
AuthorsOpenReview ↗

Response to Reviewer SY3H (Part 1 / 4)

We thank our reviewer for the time and effort dedicated to evaluating our work, as well as for the constructive comments and suggestions. Our detailed responses are provided below. We have also revised the paper to further improve clarity.


  1. [W1+Q1] Kronecker-factored preconditioning and its Book-Keeping implementation

Thank you for your valuable comments. First, we'd like to clarify that the results in Tables 1–4 of the original manuscript use the Kronecker approximation DKD^{\mathrm{K}}, whereas Figures 1–2 use the exact diagonal preconditioner DD. In response to the issues you raised, we describe the corresponding revisions and present additional experimental results and discussion below.

Explanation of the approximation to Dl,tD_{l,t} in Section 3. We have added a paragraph entitled “Kronecker approximation of Dl,tD_{l,t}” at the end of Section 3.2, after the paragraph “Effective clipping scale adaptation” and before Section 4. This paragraph explains the following.

The full diagonal preconditioner Dl,tD_{l,t} scales each weight independently and generally cannot be represented by separate scalings of activations and output derivatives. For matrix parameters, we therefore reshape its diagonal scaling coefficients into a positive matrix SS and approximate SS by qr⊤qr^\top using least-squares fitting in log space. Under column-wise vectorization, the resulting preconditioner is

=\operatorname{diag}(r)\otimes\operatorname{diag}(q), \qquad P_{l,t}^{\mathrm K}=\rho_{l,t}D_{l,t}^{\mathrm K}.$$ For a linear layer with per-example weight gradient $G^{(i)}=(E^{(i)})^\top A^{(i)}$, where $A^{(i)}$ contains activations and $E^{(i)}$ contains output derivatives, we rescale these quantities as $$A'^{(i)}=A^{(i)}\operatorname{diag}(r), \qquad E'^{(i)}=\rho_{l,t}E^{(i)}\operatorname{diag}(q).$$ The transformed gradient retains the form $G'^{(i)}=(E'^{(i)})^\top A'^{(i)}$, so its squared norm can be computed as $$\lVert G'^{(i)}\rVert_F^2 =\left\langle A'^{(i)}(A'^{(i)})^\top, E'^{(i)}(E'^{(i)})^\top \right\rangle_F.$$ Thus, SCP-Clip does not require explicit construction of per-example weight gradients. The approximation occurs in $D_{l,t}\to D_{l,t}^{K}$; norm computation and clipping themselves are exact for the given factorized transform. Appendix G further discusses the construction of this approximation and its error relative to exact $D_{l,t}$. **Comparison of exact $D$ and $D^{\mathrm{K}}$.** We have added an end-to-end comparison of the two implementations on RoBERTa-base / SST-2 at $\epsilon=8$. The exact implementation streams explicit per-example gradients through small physical microbatches. > **R3-Table 1.** Accuracy comparison between $D$ and $D^{\mathrm{K}}$. | Implementation | Seed 42 | Seed 43 | Seed 44 | Mean ± standard deviation | | --- | --- | --- | --- | --- | | SCP-Clip, exact $D$ | 92.56 | 92.86 | 93.10 | 92.84 ± 0.27 | | SCP-Clip, Kronecker $D^{\mathrm{K}}$ | 92.43 | 92.80 | 93.11 | 92.78 ± 0.34 | The mean accuracies with exact $D$ and $D^{\mathrm{K}}$ are **92.84% and 92.78%**, respectively, differing by **0.06 percentage points**. Thus, the structural approximation has little effect on final task utility in this setting. We also compare the two implementations using all four metrics in Figures 1–2 of the original manuscript. Following Appendix F, we run these experiments under exactly the same settings as those used for Figures 1–2, replacing only exact $D$ with $D^{\mathrm{K}}$. > **R3-Table 2.** Comparison of the metrics in Figures 1–2 using $D$ and $D^{\mathrm{K}}$. | Diagnostic metric | Standard per-layer clipping | Exact $D$ | Kronecker $D^{\mathrm{K}}$ | | ---------------------------------------------- | --------------------------- | --------- | -------------------------- | | Cosine similarity to the unclipped gradient ↑ | 0.742 | 0.831 | 0.829 | | Relative clipping error ↓ | 0.784 | 0.576 | 0.581 | | Mean norm-to-threshold ratio | 1.48 | 1.23 | 1.26 | | Final loss from separate noise-free training ↓ | 0.195 | 0.140 | 0.142 | As shown in the table, using the Kronecker approximation $D^{\mathrm{K}}$ causes only small changes in diagnostic metrics such as cosine similarity, while retaining the main improvements in gradient direction and reductions in clipping error. We will include these additional results in the revised manuscript.
AuthorsOpenReview ↗

Response to Reviewer SY3H (Part 2 / 4)

  1. [W2+Q2] Adaptive target and supplementary stability analysis

We sincerely thank the reviewer for this valuable comment. We provide clarification and supplementary analysis below.

The roles of ClC_l and Cl/ρl,tC_l/\rho_{l,t} in the target design. Thank you for raising this question. First, we would like to clarify that SCP-Clip performs gradient clipping and slack encoding in the transformed space, where the clipping threshold remains fixed at ClC_l. The threshold Cl/ρl,tC_l/\rho_{l,t} only represents the equivalent threshold obtained by mapping the same clipping region back to the base-preconditioned space; it is not the threshold actually used for slack encoding. Moreover, ρl,t\rho_{l,t} already enters the slack feedback through the preconditioned gradient norm. Replacing the denominator with Cl/ρl,tC_l/\rho_{l,t} would therefore introduce an additional explicit dependence on ρl,t\rho_{l,t}, changing the original feedback rule rather than simply correcting the normalization.

We provide further analysis of Condition (4) below and will include the details in the revised manuscript.

(1) Sufficient conditions for Condition (4). For fixed σ 0\sigma\>0, Eq. (17) gives

\lim_{u\to-\infty}h_{l,t}(u) &=1-\mathbb E_t\\!\left[\Pi_{[0,1]}\\!\left( \frac{1+(1+\xi_{l,K})/C_l}{2}\right)\right]\>0,\\\\ \lim_{u\to+\infty}h_{l,t}(u)&=-\frac12. \end{aligned}$$ These limits follow from $A_{l,t},Q_{l,t}\to1$ on the left and $A_{l,t},Q_{l,t}\to0$ on the right. The Gaussian lower tail prevents complete target saturation on the left; symmetry gives an expected target of $1/2$ on the right. If the continuous feedback has a unique root $u^\star_{l,t}$ with $h'_{l,t}(u^\star_{l,t})<0$, the tail signs imply positive feedback to its left and negative feedback to its right. Consider $$\frac{-(u-u^\star_{l,t})h_{l,t}(u)} {\min\\{(u-u^\star_{l,t})^2,|u-u^\star_{l,t}|\\}}, \qquad u\ne u^\star_{l,t}.$$ This quotient tends to $-h'_{l,t}(u^\star_{l,t})\>0$ at the root and to positive limits at both tails. On the remaining compact intervals, continuity gives a positive minimum. Its positive infimum therefore establishes Eq. (4). **This permits $h'_{l,t}\>0$ in the small-norm region, provided that the region lies to the left of the equilibrium and the feedback remains positive. Global monotonic decrease is unnecessary.** **(2) No clipping occurs in the degenerate region.** For $K\ge2$, if $\sigma=0$, $A_{l,t}(u)=1$ and $Q_{l,t}(u)\ge C_l$ almost surely under conditional batch sampling, then $\gamma_{l,t}=1$ and $h_{l,t}(u)=0$. For $C_l<1$ and bounded norm support, this region includes a sufficiently negative half-line, so the global condition indeed cannot hold there. However, $A_{l,t}=1$ means that the first slack coordinate of every example equals one, and hence all gradients in that layer are unclipped. Thus, **in the zero-feedback interval identified by the reviewer, the clipping coefficient is one and there is no clipping bias**. **(3) Supplementary analysis of Condition (4).** The analysis in Appendix E of the original manuscript shows that Condition (4) is used only at the algorithm's current iterate $u_{l,t}$. It therefore suffices to have a common deterministic constant $\mu\>0$ such that, for all layers and iterations almost surely, $$(u_{l,t}-u^\star_{l,t})h_{l,t}(u_{l,t}) \le-\mu\min\\{(u_{l,t}-u^\star_{l,t})^2, |u_{l,t}-u^\star_{l,t}|\\}.$$ The condition is then required only at the actual iterates. If the current iterate lies in the unclipped region described above, choose $u^\star_{l,t}=u_{l,t}$. Since $h_{l,t}(u_{l,t})=0$, the iterate itself is an equilibrium, and $\alpha^\star_{l,t}=1$. Outside this region, choose an attracting equilibrium in the sense of the original analysis. To verify this extension, we use $e_{l,t}=u_{l,t}-u^\star_{l,t}$, the potential $\Psi$ and the feedback second-moment bound $M_l$ from Appendix E. The condition imposed only at the actual iterate already gives $\Psi'(e_{l,t})h_{l,t}(u_{l,t})\le-\mu\min\\{e_{l,t}^2,1\\}$. The subsequent potential-function analysis, scale-tracking error control and clipping-error bound can therefore follow the original proof, yielding the utility upper bound in Theorem 4.2. We will include the sufficient conditions above and the extension in point (3) in Appendix E.
AuthorsOpenReview ↗

Response to Reviewer SY3H (Part 3 / 4)

  1. [W3+Q3] Noise feedback and bounded inverse preconditioning

We appreciate you raising this important concern. Noise amplification in the second-moment estimates can indeed occur. To address this formally, we analyze how noise enters the second moments and under what conditions this feedback can be controlled. We have added the complete analysis to the appendix of the revised manuscript.

Let cl,t=B−1∑i∈Btαl,t(i)gl,t(i)c_{l,t}=B^{-1}\sum_{i\in B_t}\alpha_{l,t}^{(i)}g_{l,t}^{(i)} denote the clipped batch gradient mapped back to the original space. Conditional on past private outputs, the current Pl,tP_{l,t} is fixed, and the newly added Gaussian noise is independent of the current batch. Therefore,

\tilde{g}_{l,t} &=c_{l,t}+\frac{\sigma C}{B}P_{l,t}^{-1}z_{l,t}, \qquad z_{l,t}\sim\mathcal{N}(0,I),\\\\ \mathbb{E}_t\lVert\tilde{g}_{l,t}\rVert_2^2 &=\mathbb{E}_t\lVert c_{l,t}\rVert_2^2 +\frac{\sigma^2C^2}{B^2}\lVert P_{l,t}^{-1}\rVert_F^2. \end{aligned}$$ The cross term in the second equation vanishes because the noise has zero conditional mean. Since $0\leq\alpha_{l,t}^{(i)}\leq1$, Jensen's inequality and the manuscript's gradient-moment assumption give $\mathbb{E}_t\lVert c_{l,t}\rVert_2^2\leq G_l^2$. Substituting this into $v_{l,t}=\beta_2v_{l,t-1}+(1-\beta_2)\tilde{g}_{l,t}^{\odot2}$ and using $\hat{v}_{l,t}=v_{l,t}/(1-\beta_2^t)$ yields, for $t\geq2$, $$\mathbb{E}_t\lVert\hat{v}_{l,t}\rVert_1 -\lVert\hat{v}_{l,t-1}\rVert_1 \leq \frac{1-\beta_2}{1-\beta_2^t} \left[ G_l^2-\lVert\hat{v}_{l,t-1}\rVert_1 +\frac{\sigma^2C^2}{B^2}\lVert P_{l,t}^{-1}\rVert_F^2 \right].$$ This recursion separates the two components of the feedback. The previous second moment decays, while the inverse-transformed noise increases the second moment again. For the exact diagonal preconditioner, the latter term is precisely $$\frac{\sigma^2C^2}{B^2}\lVert P_{l,t}^{-1}\rVert_F^2 =\left(\frac{\sigma C}{B\rho_{l,t}}\right)^2 \sum_j\left(\sqrt{\hat{v}_{l,t-1,j}}+\epsilon_{\mathrm{num}}\right)^2.$$ To isolate the strength of this feedback, temporarily set $\epsilon_{\mathrm{num}}=0$. The coefficient of the previous second moment in the recursion is then $1-\frac{1-\beta_2}{1-\beta_2^t}\left[1-\left(\frac{\sigma C}{B\rho_{l,t}}\right)^2\right]$. Thus, the factor $(\sigma C/(B\rho_{l,t}))^2$ identified by the reviewer determines whether noise feedback offsets the decay. **A sufficient condition for stability is that this feedback factor remains below 1 by a fixed margin**. Under this condition, decay dominates noise feedback as the second moments grow. Together with the manuscript's gradient-moment assumption and finite initialization, this ensures uniformly bounded expected second moments. The same condition also controls amplification under the inverse mapping, **keeping $\mathbb{E}\lVert P_{l,t}^{-1}\rVert_F^2$ uniformly bounded**. The positive numerical stabilizer $\epsilon_{\mathrm{num}}\>0$ used in practice introduces additional terms, but these grow more slowly and can be absorbed by the fixed margin. It therefore only requires a looser bound and does not change this sufficient condition. We have added the complete analysis to the revised manuscript. **Noise-variance correction.** Following your suggestion, we evaluate the effect of noise-variance correction on ViT-small / CIFAR-100. The only change is to replace the exponential moving average (EMA) of squared noisy gradients with an EMA computed after subtracting the known noise variance; all other settings remain unchanged. The additional results are shown below. > **R3-Table 3.** Effect of noise-variance correction on ViT-small / CIFAR-100. | Method | $\epsilon=2$ | $\epsilon=4$ | | --- | --- | --- | | Flat clipping | 81.88 ± 0.63 | 83.01 ± 0.33 | | SCP-Clip | 78.56 ± 0.29 | 80.31 ± 0.44 | | SCP-Clip + variance correction | 77.85 ± 0.38 | 79.97 ± 0.51 | As shown in the table, correcting for the known noise in the second-moment estimates does not improve utility and instead decreases accuracy. **This noise-variance correction is similar in spirit to the second-moment debiasing approach in DP-AdamBC [2]. Ganesh et al. [3], however, argue that obtaining an unbiased estimate of the second moment of noise-free gradients is not the right objective for designing private adaptive optimizers.** The optimizer operates on noisy gradients, and using a bias-corrected preconditioner to recover the geometry of noise-free gradients does not resolve its mismatch with the noisy gradients. Moreover, bias correction can make $\hat{v}_{l,t}$ negative, compromising optimization stability. These considerations may partially explain why noise-variance correction performs poorly in our additional experiments.
Reviewer_pdQUOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a clipping technique, SCP-Clip, for differentially private (DP) training of deep learning models. The primary motivation is to reduce the clipping bias introduced while performing per-layer clipping prior to the noise addition step in DP, which degrades model utility. The authors demonstrate that transforming the gradients into a scale-calibrated preconditioned space can effectively mitigate clipping bias without spending additional privacy budget. Additionally, they introduce layer-wise scale calibration of the preconditioner, which allows SCP-Clip to adaptively adjust layer-level clipping within a fixed total threshold. The paper establishes a theoretical bound on the utility guarantee of the proposed method in terms of private gradient estimation error, and presents experimental results comparing the performance of SCP-Clip to several existing fixed and adaptive clipping algorithms across a range of tasks.

Critical strengths
  1. The paper is well-organized and easy to follow with sufficient preliminary discussion, technical details and experimental justification.
  2. The key idea to incorporate preconditioning into per-layer clipping is very interesting and addresses a challenging problem in differential privacy for deep learning.
  3. The proofs seem correct, and the theoretical claims are supported by the performance improvement over existing benchmarks.
  4. Notably, the paper shows that SCP-Clip can be implemented using existing memory efficient techniques, namely, Book-Keeping (BK), with negligible computational overhead.
Critical weaknesses
  1. A possible caveat of the proposed method is the newly introduced bias when performing Adam updates using privatized estimation of the gradient second moment (DP-AdamBC, Tang et al. AAAI 2024). Would the bias correction formula from Tang et al. be applicable to the preconditioned gradients of SCP-Clip?

  2. SCP-Clip is compared against flat and various per-layer clipping techniques which all use Adam or AdamW updates. However, Theorem 4.1 of the paper states that SCP-Clip admits the same per-step privacy guarantee as standard DP-SGD. A clarification from the authors on whether these bounds hold regardless of the update rule used would strengthen their claim.

Questions for authors

Minor comments:

  • It appears to me that the terms "layer" and "module" are used interchangeably in the paper. Please clarify in the Preliminaries (Section 2).
  • GeoClip (Gilani et al.) presents a technique to approximate the covariance matrix using low-rank PCA, which may help with implementing the algorithm in Appendix I.
4 comments
AuthorsOpenReview ↗

Response to Reviewer pdQU (Part 1 / 2)

We deeply appreciate Reviewer pdQU for the positive and insightful feedback. We are equally encouraged by your recognition of the novelty of the proposed “incorporate preconditioning into per-layer clipping”. To fully address your concerns, we provide a more detailed discussion below.


  1. [W1] Noise bias in second-moment estimates and DP-AdamBC

Thank you for your suggestion. Ganesh et al. [1] argue that obtaining an unbiased estimate of the second moment of noise-free gradients is not the right objective for designing private adaptive optimizers. The optimizer operates on noisy gradients, and recovering the geometry of noise-free gradients does not resolve its mismatch with the noisy gradients. The noise-variance correction principle in DP-AdamBC [2] can be adapted to SCP-Clip by accounting for the changing coordinate-wise noise variance after inverse preconditioning. Corrected estimates can become negative and require numerical safeguards, as implemented in DP-AdamBC.

We also evaluate the effect of noise-variance correction on ViT-small / CIFAR-100. The only change is to replace the exponential moving average (EMA) of squared noisy gradients used to construct the clipping preconditioner with an EMA computed after subtracting the known noise variance; all other settings, including the AdamW parameter update, remain unchanged. The additional results are shown below.

R2-Table 1. Effect of noise-variance correction on ViT-small / CIFAR-100.

Methodϵ=2\epsilon=2ϵ=4\epsilon=4
Flat clipping81.88 ± 0.6383.01 ± 0.33
SCP-Clip78.56 ± 0.2980.31 ± 0.44
SCP-Clip + variance correction77.85 ± 0.3879.97 ± 0.51

As shown in the table, at ϵ=2\epsilon=2, mean accuracy decreases from 78.56% to 77.85%; at ϵ=4\epsilon=4, it decreases from 80.31% to 79.97%. These are decreases of 0.71 and 0.34 percentage points, respectively. In the application settings considered, correcting for the known noise in the second-moment estimates does not improve utility and instead decreases mean accuracy.


  1. [W2] Privacy guarantees under different update rules

Thank you for requesting this clarification. The privacy guarantee in Theorem 4.1 applies to SGD, Adam, AdamW, RMSProp, and commonly used Adam variants. The derived privacy bound is independent of the update rule, provided that the updates access the training data only through privatized gradients. As shown in the proof in Section 4.1, SCP-Clip's joint release satisfies the same norm bound and Gaussian noise calibration as standard DP-SGD, so the same privacy accountant yields the same per-step privacy upper bound. This derivation does not depend on the SGD or Adam parameter-update formula. Since Pl,tP_{l,t} is determined by historical private outputs, the inverse mapping g~l,t=Pl,t−1h~l,t\tilde{g}_{l,t}=P_{l,t}^{-1}\tilde{h}_{l,t} is post-processing. Subsequently, SGD parameter updates, momentum accumulation, and Adam's first and second-moment updates are computed from privatized gradients and historical state. AdamW's decoupled weight decay likewise depends only on the current parameters and public coefficients. All of these operations are post-processing and incur no additional privacy loss. The comparison with standard DP-SGD in Theorem 4.1 therefore concerns the privacy upper bound for the joint release, which also applies to the Adam and AdamW updates used in our experiments and to other update rules satisfying the condition above.

We have added “This upper bound remains valid for any subsequent update rule that accesses the training data only through the mechanism’s privatized outputs.” to the final paragraph of the proof sketch for Theorem 4.1 in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer pdQU (Part 2 / 2)

  1. [Q1] Clarification of "layer" and "module"

Thank you for pointing out this ambiguity. In the original manuscript, we used "layer" and "module" interchangeably to refer to the same granularity, namely a single trainable submodule. The term "module" emphasizes its role as a model component, whereas "layer" emphasizes its role as a clipping unit. To avoid ambiguity and ensure consistent terminology, we have replaced both occurrences of "module" with "layer" in the revised manuscript.


  1. [Q2] Low-rank PCA approximation for GeoClip

Thank you for highlighting GeoClip’s low-rank covariance approximation. We discuss this approach and its computational requirements in Appendix I (page 32, lines 1708–1714). For clarity, we reproduce the relevant sentence below.

“The paper’s streaming rank-kk alternative avoids dense storage, but still maintains a d×kd \times k basis and updates it by a singular value decomposition.”

We attempted to use this low-rank PCA approximation to implement a per-layer clipping version of GeoClip with Book-Keeping (BK). However, even with this approximation, the computational overhead remained too high for the application settings considered in our work.

[1] On Design Principles for Private Adaptive Optimizers. Arun Ganesh et al. arXiv, 2025.

[2] DP-AdamBC: Your DP-Adam Is Actually DP-SGD (Unless You Apply Bias Correction). Qiaoyue Tang et al. AAAI, 2024.

I'd like to thank the authors for their thorough response and additional experiments. I appreciate the added sentence in the proof sketch of Thm 4.1.

The noise-variance correction experiment is also informative and I thank the authors for providing the insight from Ganesh et al. that recovering the second moment of noise-free gradients may not be the right objective for DP optimizers. I encourage them to include these findings in the revised version, as it directly motivates their design choice.

I will keep my current score for the paper.

AuthorsOpenReview ↗

Thank You for Your Comments

Thank you for your thoughtful follow-up and for taking the time to review our responses and additional experiments. We also sincerely appreciate your positive assessment of our work. We are glad that the clarification of Theorem 4.1 and the noise-variance correction experiment were helpful.

We have included the experimental results and the related discussion of Ganesh et al. in Appendix J (Table 15, p. 34) of the revised manuscript to clarify the rationale behind our preconditioner design. We sincerely appreciate your careful review and constructive suggestions, which have helped us improve the paper.

Reviewer_6ApYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Per-layer clipping in DP training saves memory, but it gives each layer its own clipping coefficient, which distorts the direction of the per-example gradient. SCP-Clip clips each layer in a diagonal preconditioned space built from the optimizer's second moments of earlier privatized gradients (Sec. 3.1). It rescales that space per layer with a scalar ρ, updated from private slack feedback borrowed from SlaClip (Sec. 3.2, Alg. 1). Thm 4.1 shows the privacy cost is the same as standard DP-SGD with C = (ΣC_l²)^½. Thm 4.2 bounds the error of the private gradient under a "stable feedback" assumption. The method is tested on RoBERTa/GLUE, ViT/CIFAR and GPT-2/E2E. It beats the existing per-layer baselines, by up to about 4–6 points on CIFAR-100, and roughly matches flat clipping.

Critical strengths
  1. The method offers a practical improvement with a sound privacy argument. SCP-Clip keeps the memory and pipelining benefits of per-layer clipping: 12–32% less peak memory than flat BK, and 0.3–1.6% time overhead over per-layer BK (Table 6). It closes most of the utility gap to flat clipping, and sometimes beats it slightly (Tables 1–3). The privacy guarantee costs nothing extra and is proved carefully, by reduction to the norm-bounded Gaussian sum mechanism (As shown in Lemmas D.1–D.4). This matters because memory-efficient DP fine-tuning of large models is an important practical setting, and the method is a drop-in change to existing BK code.

  2. The results show large, consistent gains over existing per-layer baselines where per-layer clipping hurts most. On ViT/CIFAR-100, SCP-Clip improves on Adaptive per-layer by 3.4–4.3 points at ε = 2–4 (Table 2). The ablation (Table 4) shows that base preconditioning and scale calibration each help on their own.

Critical weaknesses
  1. My main concern is the limited novelty, and the key controls are missing. SCP-Clip combines preconditioning from historical private second moments (close to DP2, Li et al. 2023, and to AdaCliP) with SlaClip's slack controller. The paper says its gains come from fixing per-layer distortion, but it never tests preconditioned flat clipping. If preconditioning helps flat clipping just as much, the gain is generic adaptive-geometry clipping and not specific to per-layer clipping. For the comparison, DP2, DP-MacAdam and the Ganesh et al. scale-then-privatize approach are not compared at all. App. I compares against layer-wise AdaCliP and SlaClip only on ViT-small/CIFAR-100. There, SCP-Clip's margin over AdaCliP-L + slack is only 0.7–1.1 points, which is about the size of the seed standard deviations in Table 4.

  2. The theory does not compare methods, and the statistical evidence is weak. Thm 4.2 is an upper bound under an assumed equilibrium u* (Eq. 4). It does not show that the clipping-error term is smaller than for ordinary per-layer clipping. The noise term grows as E‖P⁻¹‖_F² and is not controlled. In the coordinates where v̂ is large, the noise in the original space is amplified by √v̂/ρ. For the experimental results, Tables 1–3 report no standard deviations. Many gaps over Flat or Adaptive are 0.1–0.5 points, while Table 4 shows seed standard deviations of 0.3–0.7. Against flat clipping, results are mixed. SCP-Clip is lower on QNLI-base ε=3, SST-2-large ε=3, ViT-large CIFAR-100 ε=2, ViT-large CIFAR-10 ε=8, and GPT-2-medium BLEU ε=3 (63.14 vs 63.85). ViT and GPT-2 runs use a single learning rate shared by all methods. The reported results come from the Kronecker approximation (App. G), not from the exact diagonal preconditioner that the paper analyzes and diagnoses (Sec. 3, App. F).

Questions for authors
  1. About preconditioned flat clipping. How does flat clipping with the same D (and with ρ) perform? This is the most direct test of whether the gain is specific to per-layer clipping.
  2. Were Tables 1–4 produced with the Kronecker approximation D^K from App. G? Please compare exact diagonal D against D^K for accuracy on at least one setting, and explain how Fig. 1–2, which use exact D, carry over.
  3. The injected noise is mapped back through P⁻¹ ∝ √v̂, and v̂ is then updated from this noisy gradient. Is the loop stable when σC/(ρB) is not small (smaller batches or stronger privacy)? Please report typical values of ρ_{l,t} and ‖P⁻¹‖_F over training, and say whether SCP-Clip works with optimizers that do not keep second moments, such as DP-SGD with momentum.
  4. Please add standard deviations, or significance tests, to Tables 1–3. Were learning rate and C₀ tuned per method or shared? Was the SST-2 hyperparameter selection run with SCP-Clip?
  5. What is the rationale for γ_{l,t} = Π_{[0,1]}(1 − (1 − ŝ_{l,K}/C_l)/2), especially the division by C_l? How sensitive are the results to η_ρ and K?
  6. Is there any setting where Eq. 4 can be verified? Can you show, even empirically, that the calibrated clipping-error term in Eq. 5 is smaller than for per-layer clipping without preconditioning?
  7. A few suggestions: It would be helpful to include comparisons with DP2 and DP-MacAdam, extend the comparison in Appendix I to another model or task, and clarify the wording in Sec. 3.1, which seems to conflate flat clipping with uniform-threshold clipping.
11 comments
AuthorsOpenReview ↗

Response to Reviewer 6ApY (Part 1 / 9)

We really appreciate the reviewer for the detailed and constructive feedback. We are pleased to see that you found our proposed integration of gradient preconditioning into per-layer clipping to be a meaningful and practically impactful direction.

We believe our response has addressed most of your concerns and we respectfully ask if you're comfortable reconsidering the rating. Below, we provide detailed responses to all the comments:


  1. [W1+Q1] Preconditioned flat clipping

Thank you for suggesting this important experiment. Our initial focus was on reducing the additional directional distortion introduced by differences in clipping coefficients across layers, while retaining the execution efficiency of per-layer clipping. Accordingly, the original manuscript used flat clipping as a performance reference and restricted the ablations of preconditioning and scale calibration to the per-layer clipping framework.

Following the suggestion, we further evaluate preconditioning DD, global scale calibration ρglobal\rho_{\mathrm{global}}, and their combination under flat clipping. R1-Table 1 presents the results, allowing us to compare the gains from these components across the two clipping frameworks.

R1-Table 1. Effects of preconditioning and scale calibration on flat clipping.

MethodPreconditioningScale calibrationRoBERTa-base / SST-2, ϵ=3\epsilon=3ViT-small / CIFAR-10, ϵ=4\epsilon=4
FlatNoNone92.20 ± 0.3196.95 ± 0.36
Flat + DDYesNone92.34 ± 0.2996.98 ± 0.25
Flat + ρglobal\rho_{\mathrm{global}}NoGlobal92.38 ± 0.2797.02 ± 0.42
Flat + DD + ρglobal\rho_{\mathrm{global}}YesGlobal92.45 ± 0.3497.04 ± 0.30

Preconditioning and global scale calibration also improve the mean performance of flat clipping. On RoBERTa-base / SST-2 at ϵ=3\epsilon=3 and ViT-small / CIFAR-10 at ϵ=4\epsilon=4, combining DD and ρglobal\rho_{\mathrm{global}} yields gains of 0.25 and 0.09 percentage points, respectively. These gains are much smaller than those under per-layer clipping, which are 2.48 and 0.77 percentage points, respectively (Section 5.2, Table 4). This indicates that the gains from preconditioning and scale calibration arise predominantly from improvements to per-layer clipping, consistent with our motivation.

AuthorsOpenReview ↗

Response to Reviewer 6ApY (Part 2 / 9)

  1. [W1+Q7] Comparisons with related methods

We sincerely appreciate your valuable comments. We provide additional results and detailed discussion regarding the specific experiments mentioned.

We thank the reviewer for the question. Our work focuses on full-parameter fine-tuning of pretrained Transformers for vision and language tasks. By comparison, the original DP² [1] experiments include logistic regression on IMDB and tag classification on StackOverflow, with models ranging from approximately 10,000 to 5 million parameters. DP-MacAdam [2] trains two-layer fully connected networks and five-layer convolutional neural networks (CNNs) from scratch on MNIST and CIFAR-10, each with fewer than one million parameters. Their original evaluations therefore differ from our target setting in architecture, model scale, and training regime.

Nevertheless, both are relevant adaptive private optimization methods, and including them provides a more comprehensive evaluation of SCP-Clip. We extend our comparisons to DP²-L and DP-MacAdam-L to assess their performance under per-layer clipping.

Regarding the limited margin over AdaCliP-L + slack. We now report standard deviations across the three seeds from the original experiments. Following the reviewer's suggestion, we also extend the comparison to RoBERTa-base / SST-2. The newly added DP²-L and DP-MacAdam-L results follow the same reporting protocol.

R1-Table 2. Additional comparisons between SCP-Clip and related methods. Newly added experimental results are shown in bold.

MethodViT-small / CIFAR-100, ϵ=2\epsilon=2ViT-small / CIFAR-100, ϵ=4\epsilon=4ViT-small / CIFAR-100, ϵ=8\epsilon=8RoBERTa-base / SST-2, ϵ=3\epsilon=3RoBERTa-base / SST-2, ϵ=8\epsilon=8
AdaCliP-L76.18 ± 0.3478.36 ± 0.2783.24 ± 0.3591.59 ± 0.4191.88 ± 0.23
AdaCliP-L + slack77.42 ± 0.2279.24 ± 0.3683.90 ± 0.5692.17 ± 0.3092.53 ± 0.28
SlaClip-L77.21 ± 0.2778.94 ± 0.4283.57 ± 0.3192.11 ± 0.4192.45 ± 0.35
DP²-L75.09 ± 0.3277.80 ± 0.2181.42 ± 0.3191.02 ± 0.2891.19 ± 0.19
DP-MacAdam-L72.13 ± 0.4473.47 ± 0.3175.82 ± 0.2690.58 ± 0.3490.91 ± 0.31
GeoClip-L—————
GeoClip-L + slack—————
SCP-Clip78.56 ± 0.2980.31 ± 0.4484.61 ± 0.2692.65 ± 0.5992.78 ± 0.34

Discussion of the results. Our method maintains an advantage across the evaluated settings, whereas DP² and DP-MacAdam do not yield marked improvements when applied to per-layer clipping. These adaptive optimization methods were developed and evaluated with flat clipping, without accounting for the particular structure of per-layer clipping. Directly transferring them to per-layer clipping may therefore be inappropriate. Similar considerations apply to methods such as DP-KFC [3] and DP-AdamBC [4]. A possible direction for future work is to retain the utility gains of adaptive optimization while exploiting the efficiency of per-layer clipping to scale to larger settings.

AuthorsOpenReview ↗

Response to Reviewer 6ApY (Part 3 / 9)

  1. [W2+Q6] Empirical verification of the feedback condition and comparison of clipping errors

We thank the reviewer for this valuable suggestion. We have added two numerical experiments addressing the stable-feedback condition in Eq. (4) and the calibrated clipping error in Eq. (5). We run the complete SCP-Clip method on RoBERTa-base / SST-2 at ϵ=3\epsilon=3, using batch size B=1024B=1024, C0=1C_0=1, and random seed 42. All remaining training hyperparameters follow Appendix F. At iterations 200, 500, 800, and 1100, we freeze the model parameters and base preconditioner, then independently sample two sets of 8192 examples each. One set estimates the feedback and equilibrium scales; the other evaluates clipping error.

Verification of Eq. (4). We estimate hl,t(u)h_{l,t}(u) using the actual feedback expression in Appendix E, retaining both the Gaussian noise and the nonlinear projection in the target. We then numerically solve for the equilibrium u^l,t⋆\widehat{u}_{l,t}^{\star}. For all 102 clipping groups, we scan offsets δ=u−u^l,t⋆∈[−2,2]\delta=u-\widehat{u}_{l,t}^{\star}\in[-2,2] in increments of 0.05 and directly compare the two sides of Eq. (4). R1-Table 3 shows representative offsets. At each offset, we report the maximum left-hand-side value across all clipping groups and all four checkpoints to assess whether the condition holds in the worst case.

R1-Table 3. Numerical comparison of the two sides of Eq. (4).

Offset from equilibrium δ\deltaMaximum left-hand side of Eq. (4)Right-hand side of Eq. (4), μ=0.10\mu=0.10
−2.0−0.368−0.200
−1.0−0.142−0.100
−0.5−0.042−0.025
+0.5−0.039−0.025
+1.0−0.136−0.100
+2.0−0.416−0.200

As R1-Table 3 shows, even the worst-case left-hand-side value across all tested groups and checkpoints does not exceed the right-hand side. The estimated feedback also satisfies the inequality with μ=0.10\mu=0.10 at every tested grid point in the full scan. This provides direct numerical support for Eq. (4) near equilibrium at the training states examined.

Comparison of the clipping-error term in Eq. (5). On the second, independent evaluation set, we apply standard per-layer clipping and SCP-Clip to identical per-example gradients. Both methods share the model parameters, clipping thresholds, and dropout realization. SCP-Clip uses the equilibrium scales estimated in the preceding experiment. For each method mm, we directly estimate the full clipping-error term in Eq. (5), in the original parameter space and before adding the current step's gradient noise: 2Et![∑l=1L(1−αl,tm)2∥gl,t∥22].2\mathbb{E}_t\\!\left[\sum_{l=1}^{L}(1-\alpha_{l,t}^{m})^2\lVert g_{l,t}\rVert_2^2\right].

R1-Table 4. Comparison of the clipping-error term in Eq. (5).

Training iterationStandard per-layer clippingSCP-ClipError reduction
2001.84201.206034.53%
5001.37600.842038.81%
8001.12400.651042.08%
11000.96800.542044.01%
Average over four checkpoints1.327500.8102538.96%

SCP-Clip achieves a lower full clipping error at all four checkpoints, with reductions ranging from 34.53% to 44.01%. The average error across the four checkpoints decreases from 1.32750 to 0.81025, a reduction of 38.96%. Both clipping methods operate on identical per-example gradients. Thus, in this experimental setting, SCP-Clip at its equilibrium scales directly yields a lower full calibrated clipping error than standard per-layer clipping without preconditioning.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We sincerely thank the Area Chair for overseeing our submission and all reviewers for their thoughtful and constructive feedback. We are encouraged by the reviewers’ recognition of the key contributions and strengths of our work.

We have carefully responded to the reviewers’ comments through additional experiments and clarifications. In our revised manuscript, we have incorporated the suggested experiments, additional discussions, and reporting updates to further strengthen our work. Below, we briefly summarize the reviewer discussions.


  • Reviewer 6ApY. After we provided our rebuttal, the concerns (mainly novelty and empirical support) raised by Reviewer 6ApY were addressed. The reviewer acknowledged our response and raised their overall rating from 2 to 3 (Weak acceptance).

Thanks for the authors' response, which addressed most of my concerns. I will raise my overall rating to 3.

  • Reviewer pdQU. The reviewer provided a positive recommendation from the beginning with an overall rating of 3 (Weak acceptance). The reviewer acknowledged the clarification of Theorem 4.1, described the noise-variance correction experiment as informative, and maintained their positive assessment and rating.

I'd like to thank the authors for their thorough response and additional experiments. I appreciate the added sentence in the proof sketch of Thm 4.1.

I will keep my current score for the paper.

  • Reviewer SY3H. The reviewer provided a positive recommendation from the beginning with an overall rating of 3 (Weak acceptance). We clarified the Kronecker implementation and added comparisons and feedback-stability analysis. We are encouraged that the reviewer raised their Soundness score from 2 to 3.

Thank you for the response and the revised PDF. I have raised my Soundness score from 2 to 3 and maintain my favorable impression.


We are grateful for the constructive discussions, which have helped us strengthen the paper, and thank the Area Chair and reviewers again for their consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.