Structural‑Prior Geometric Clipping for Differentially Private SGD

Differentially private SGD (DP-SGD) and its adaptive variants treat the param- eter space of deep networks as a homogeneous Euclidean manifold, where a single global coordinate system suffices for gradient clipping and noise calibra- tion. We challenge this assumption through structural priors from mean-field theory: network parameters are organized into layers with ill-conditioned spec- tra and pronounced inter-layer heterogeneity. We prove that in the oracle setting (exact geometry, full rank), global clipping and clipping under the structural prior are exactly equivalent—their divergence is entirely privacy-induced, through two mechanisms: (i) geometric statistics estimated from released noisy gradients suf- fer an anisotropic, batch-size-invariant bias exhibiting the structure of the optimal metric, which we invert in closed form—the first treatment beyond the diago- nal case—and characterize the resulting spectral noise floor; (ii) under the rank budgets of privacy-scale models, pooled estimation is structurally blind to direc- tions discarded by top-k truncation, which per-layer estimation retains, enabling closed-form optimal transformations and decoupled convergence. We propose SP-GeoClip, which instantiates the structural prior via independent per-layer ge- ometry estimation with streaming low-rank PCA; all statistics are post-processing of released noisy gradients at no additional privacy cost, and the end-to-end (ε, δ) guarantee is identical to standard DP-SGD. Across nine benchmarks spanning five layer families, SP-GeoClip is uniformly best (paired-seed test, p < 0.05), with gains up to 2.4 percentage points.