acceptodds
Withdrawn from ICLR 2027

Structural‑Prior Geometric Clipping for Differentially Private SGD

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Differentially private SGD (DP-SGD) and its adaptive variants treat the param- eter space of deep networks as a homogeneous Euclidean manifold, where a single global coordinate system suffices for gradient clipping and noise calibra- tion. We challenge this assumption through structural priors from mean-field theory: network parameters are organized into layers with ill-conditioned spec- tra and pronounced inter-layer heterogeneity. We prove that in the oracle setting (exact geometry, full rank), global clipping and clipping under the structural prior are exactly equivalent—their divergence is entirely privacy-induced, through two mechanisms: (i) geometric statistics estimated from released noisy gradients suf- fer an anisotropic, batch-size-invariant bias exhibiting the structure of the optimal metric, which we invert in closed form—the first treatment beyond the diago- nal case—and characterize the resulting spectral noise floor; (ii) under the rank budgets of privacy-scale models, pooled estimation is structurally blind to direc- tions discarded by top-k truncation, which per-layer estimation retains, enabling closed-form optimal transformations and decoupled convergence. We propose SP-GeoClip, which instantiates the structural prior via independent per-layer ge- ometry estimation with streaming low-rank PCA; all statistics are post-processing of released noisy gradients at no additional privacy cost, and the end-to-end (ε, δ) guarantee is identical to standard DP-SGD. Across nine benchmarks spanning five layer families, SP-GeoClip is uniformly best (paired-seed test, p < 0.05), with gains up to 2.4 percentage points.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Differential privacy, DP‑SGD, Geometry‑aware optimization, Spectral debiasing

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_kKrKOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

SP-GeoClip adapts geometric gradient clipping to individual network layers. Each layer estimates its geometry from previously privatized gradients using streaming low-rank PCA; subsequent per-example gradients are transformed, clipped, noised, and mapped back. The paper proposes a correction for noise-contaminated spectral estimates and argues that layer-wise estimation preserves directions missed by pooled PCA. It reports accuracy gains across nine dataset–model combinations, using 20 paired seeds, alongside spectral and debiasing ablations.

Critical strengths
  1. Practical structural idea. Estimating geometry separately per layer can allocate limited covariance storage across heterogeneous parts of a network, rather than allowing a pooled rank budget to concentrate on a few layers.
  2. Reuse of privatized information. Updating geometry solely from earlier DP releases avoids an additional private-statistics query. The reported multi-task evaluation and debiasing ablations attempt to connect this design to measurable utility gains.
Critical weaknesses
  1. The central guarantees are not established. Lemma A.1 is false: for A = diag(4, 1), k = 2, v₁ = (1, 1)/√2 and v₂ = (1, −1)/√2, the left side is 2√2.5 ≈ 3.162, exceeding the claimed bound 3. Its concavity inequality is reversed, and Proposition 4.2's proof explicitly relies on it. Separately, Appendix A.5 starts from a global unit-radius clip with noise inside it, whereas Algorithm 1 clips each layer at C/√L and adds noise afterward. A block-diagonal transform does not make these operations equivalent. The proof also uses per-layer threshold 1 instead of 1/√L. Theorem 4.4 therefore requires a derivation for the actual algorithm.
  2. The released demo omits core components and has inconsistent privacy calibration. In sp_geoclip_GPU.py, compute_transform (lines 52–60) only eigendecomposes and clamps eigenvalues: the proposed spectral debiasing is absent. Dense square matrices (lines 35–38) replace streaming rank-k PCA; there is no rank parameter or truncation. Each parameter tensor is clipped at C, not C/√L (lines 91–98), and the accountant's output is passed directly as per-example noise SD without converting it to the aggregate noise multiplier. Moreover, rdp.py:32 uses q²α/(2σ²) as a general RDP bound, although it can underestimate sampled-Gaussian RDP. demo.py:37 calibrates on 20,000 examples before the 80/20 split, while training uses 16,000 and a different sampling schedule. These issues prevent validation of the claimed privacy budget; they do not alone prove that the default demo violates it. The release contains only a single-seed synthetic-data demonstration, without the benchmark pipelines or records needed to reproduce the reported results. Because spectral debiasing is a central contribution, the missing DP-AdamBC comparison is particularly consequential: it already corrects DP-induced second-moment bias, so the benefit of extending correction beyond diagonal statistics needs to be isolated. DiSK filters privatized gradients rather than debiasing covariance, but is an important comparator for the claimed utility gains from noise correction. DP² and AdaDPS are also absent; the latter requires explicit qualification of its side information. The cited DP-KFC already compares pre-privatization methods with DP-AdamBC and DiSK and reports a DP-KFC + DP-AdamBC combination on MNIST. This demonstrates the relevance of evaluating both families and their complementarity; the pre-/post-privatization distinction alone does not justify omitting them here.
Questions for authors
  1. Theory: Can you correct Lemma A.1, identify which coverage conclusions survive, and prove convergence for Algorithm 1's actual clipping/noise mechanism?
  2. Implementation and privacy: Can you provide the implementation used for the tables, including spectral correction, streaming PCA, and a valid accountant calibrated to the actual sampling, clipping, batch sizes, and dataset size?
  3. Experimental evidence: Can you provide the 20 paired-seed results and test definition, plus controls separating layer-wise clipping, geometry estimation, and debiasing, including a rank sweep? Please also reconcile Section 6.5's whitening penalties with Table 7: MNIST is described as 3.0–3.6 points, but the table shows 0.6–1.1. Can you isolate the utility benefit of spectral debiasing against DP-AdamBC and DiSK, and compare with DP² and appropriately qualified AdaDPS under matched privacy and tuning budgets, or justify their exclusion?
Reviewer_UZDTOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a layer-wise geometry-informed optimization scheme for differentially private SGD. The paper reuses the methodology of Gilani et al, 2025; for which it provides an algorithmic improvement to enable the estimation of covariances from privatized data and provides an extension to take into account layer-wise geometries instead of only global parameter space correlations. The authors then show the method outperforms DP-SGD and GeoClip along with adaptive clipping approaches. The experiments are conducted on multiple datasets and evaluate multiple privacy budgets, with multiple seed mean and standard deviation evaluations. The improvements seem consistent.

Critical strengths
  • The equal configuration comparisons are run on multiple datasets, for multiple privacy budgets, and multiple model architectures; all of these experiments disclose mean and standard deviations for multiple seeds.

  • The idea to break down GeoClip into layer-wise covariance estimations is intuitive and results in strong performance improvements.

Critical weaknesses

I found three main issues with this paper.

  • The use of Karamata's inequality in Lemma A.1 seems incorrect, the function is concave therefore the result should be that ∑idi≤∑iλi\sum_i \sqrt{d_i} \leq \sum_{i} \sqrt{\lambda_i}. This directly impacts parts of the proof of Proposition 4.2 of the main paper. I did not have time to check the integrality of the other proofs for potential errors, this is the only problem I have found. In practice, this does not implicate the soundness of the private learning algorithm, as Proposition 4.2 seems to be a rather retrofitted justification of the methods interest using heavy formalism.found in this limited time.

  • The method does not compare itself to DP-SGD using equal runtime or equal memory consumption settings. Although additional epochs are not necessarily beneficial under a fixed privacy budget, runtime matched comparison remain informative. Similarly, reducing the value of kk could allow memory matched comparisons with optimizers such as DP-AdamBC (Tan et al. 2024), DP-KFC or even DP-SGD with a larger batch size, the authors only report the memory overhead wrt to GeoClip, but the memory overhead of the method relatively to DP-SGD seems to grow substantially for bigger neural networks.

  • Moreover, the paper is very difficult to follow and generally badly written. For example, Sections 4.1, 4.2, 4.3, until 4.4 only contain Propositions or Theorems without any sentences justifying why such tooling is introduced.

Questions for authors
  • Could the authors comment on the use of Karamata's inequality which seems incorrectly applied.

  • Given that the proposed method is ∼2.5×\sim 2.5 \times slower than DP-SGD. Could the authors provide a table or plot summarizing the runtime-utility performance of DP-SGD, GeoClip and SP-GeoClip using end of epoch (privacy, performance) logs ? A small dataset like MNIST would suffice.

Reviewer_QC5dOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SP-GeoClip, a layer-wise geometry-aware clipping method for DP-SGD. It analyzes anisotropic spectral pollution in covariance estimates formed from released noisy gradients, derives a closed-form debiasing expression, and argues that per-layer low-rank estimation can preserve directions that pooled global estimation may discard under a rank budget.

Critical strengths

(1) Theoretical explanation of privacy-induced geometric estimation errors. (2) Broad empirical coverage with repeated-seed evaluation. Nine dataset–model combinations and 20 paired seeds provide a stronger validation basis than a single-task proof of concept, and the paper attempts to test specific predictions from its theory.

Critical weaknesses

(1) The implementation and proof descriptions are not fully consistent. The treatment of the orthogonal complement differs between the main text and Appendix A.2. In addition, the exact quadratic-root debiasing formula in the main text differs from the plug-in subtraction expression in the appendix. These differences matter because the spectral estimate and its noise correction are central to the proposed algorithm. (2) The relationship between optimal layer allocation and the implemented uniform clipping rule is unclear.

Questions for authors

(1) Which orthogonal-complement treatment is implemented? (2) Which spectral debiasing expression is used in the released implementation: the exact quadratic-root formula in the main theorem or the plug-in subtraction formula in Appendix A.4? (3) The theory derives an optimal cross-layer allocation γl∝Sl\gamma_l\propto S_l, whereas the algorithm uses Cl=C/LC_l=C/\sqrt L. Are these the same allocation variables? (4) Sensitivity and noise-calibration derivation through per-layer transformation, clipping, concatenation, Gaussian release, and inverse transformation

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.