- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
SP-GeoClip adapts geometric gradient clipping to individual network layers. Each layer estimates its geometry from previously privatized gradients using streaming low-rank PCA; subsequent per-example gradients are transformed, clipped, noised, and mapped back. The paper proposes a correction for noise-contaminated spectral estimates and argues that layer-wise estimation preserves directions missed by pooled PCA. It reports accuracy gains across nine dataset–model combinations, using 20 paired seeds, alongside spectral and debiasing ablations.
Critical strengths
- Practical structural idea. Estimating geometry separately per layer can allocate limited covariance storage across heterogeneous parts of a network, rather than allowing a pooled rank budget to concentrate on a few layers.
- Reuse of privatized information. Updating geometry solely from earlier DP releases avoids an additional private-statistics query. The reported multi-task evaluation and debiasing ablations attempt to connect this design to measurable utility gains.
Critical weaknesses
- The central guarantees are not established. Lemma A.1 is false: for A = diag(4, 1), k = 2, v₁ = (1, 1)/√2 and v₂ = (1, −1)/√2, the left side is 2√2.5 ≈ 3.162, exceeding the claimed bound 3. Its concavity inequality is reversed, and Proposition 4.2's proof explicitly relies on it. Separately, Appendix A.5 starts from a global unit-radius clip with noise inside it, whereas Algorithm 1 clips each layer at C/√L and adds noise afterward. A block-diagonal transform does not make these operations equivalent. The proof also uses per-layer threshold 1 instead of 1/√L. Theorem 4.4 therefore requires a derivation for the actual algorithm.
- The released demo omits core components and has inconsistent privacy calibration. In
sp_geoclip_GPU.py,compute_transform(lines 52–60) only eigendecomposes and clamps eigenvalues: the proposed spectral debiasing is absent. Dense square matrices (lines 35–38) replace streaming rank-k PCA; there is no rank parameter or truncation. Each parameter tensor is clipped at C, not C/√L (lines 91–98), and the accountant's output is passed directly as per-example noise SD without converting it to the aggregate noise multiplier. Moreover,rdp.py:32uses q²α/(2σ²) as a general RDP bound, although it can underestimate sampled-Gaussian RDP.demo.py:37calibrates on 20,000 examples before the 80/20 split, while training uses 16,000 and a different sampling schedule. These issues prevent validation of the claimed privacy budget; they do not alone prove that the default demo violates it. The release contains only a single-seed synthetic-data demonstration, without the benchmark pipelines or records needed to reproduce the reported results. Because spectral debiasing is a central contribution, the missing DP-AdamBC comparison is particularly consequential: it already corrects DP-induced second-moment bias, so the benefit of extending correction beyond diagonal statistics needs to be isolated. DiSK filters privatized gradients rather than debiasing covariance, but is an important comparator for the claimed utility gains from noise correction. DP² and AdaDPS are also absent; the latter requires explicit qualification of its side information. The cited DP-KFC already compares pre-privatization methods with DP-AdamBC and DiSK and reports a DP-KFC + DP-AdamBC combination on MNIST. This demonstrates the relevance of evaluating both families and their complementarity; the pre-/post-privatization distinction alone does not justify omitting them here.
Questions for authors
- Theory: Can you correct Lemma A.1, identify which coverage conclusions survive, and prove convergence for Algorithm 1's actual clipping/noise mechanism?
- Implementation and privacy: Can you provide the implementation used for the tables, including spectral correction, streaming PCA, and a valid accountant calibrated to the actual sampling, clipping, batch sizes, and dataset size?
- Experimental evidence: Can you provide the 20 paired-seed results and test definition, plus controls separating layer-wise clipping, geometry estimation, and debiasing, including a rank sweep? Please also reconcile Section 6.5's whitening penalties with Table 7: MNIST is described as 3.0–3.6 points, but the table shows 0.6–1.1. Can you isolate the utility benefit of spectral debiasing against DP-AdamBC and DiSK, and compare with DP² and appropriately qualified AdaDPS under matched privacy and tuning budgets, or justify their exclusion?