- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies differentially private optimization for finite-sum distributionally robust learning with nonconvex losses. For general -divergences, it uses a dual reformulation involving the model parameters and a scalar auxiliary variable, and proposes DP Double-SPIDER with separate private variance-reduced estimators for the two variable blocks. The authors claim an expected gradient-norm bound of $ \mathcal{O}\left(n^{-1/2}+\left(\frac{\sqrt{d\log(1/\delta)}}{n\varepsilon}\right)^{2/3}\right). $ For KL divergence, the paper introduces a compositional formulation and DP Recursive-SPIDER, claiming to remove the term and match the cited private nonconvex ERM rate. Experiments on CIFAR10-ST, MNIST-ST, Fashion-MNIST, and CelebA compare predictive performance, gradient norms, membership-inference attacks, and hyperparameter sensitivity.
The intended contribution is a privacy–utility analysis tailored to nonconvex DRO. My principal concerns are whether the stated assumptions and formulations support the stationarity guarantees, and whether the privacy analysis applies to the algorithms and training pipeline actually used.
Critical strengths
-
Private learning under distributional uncertainty is a worthwhile problem. Exploiting scalar dual or compositional reformulations avoids explicitly optimizing an adversarial weight vector with one coordinate per training example. Adapting variance reduction and privacy-noise calibration to the different quantities in these formulations is a sensible direction. A rigorous improvement in private stationarity guarantees for this setting would be valuable.
-
The evaluation covers four datasets, several privacy budgets, and complementary optimization, prediction, and privacy metrics. In particular, Appendix E.4 investigates the practical tuning burden through tied, pairwise, and independently sampled clipping thresholds, together with phase-size and learning-rate sensitivity. The sensitivity experiments report three seeds, and the membership-inference experiments report five runs. These investigations address relevant practical questions, although the interpretation of some results requires clarification.
Critical weaknesses
- Assumption 3 requires the conjugate to have bounded domain (p. 4, lines 201–202). However, the divergence definition on p. 3 requires , , and for . These conditions imply
Questions for authors
-
Could the authors clarify what the precise intended replacement for Assumption 3 is? Please identify a nonempty class of divergences satisfying the corrected assumptions and explain how they establish Proposition 1 along the actual Gaussian-perturbed iterates. This is necessary to determine whether the general-divergence result applies to the examples considered.
-
Does the KL method solve the fixed-penalty problem in Equation (2), or a KL-ball-constrained problem corresponding to Equation (5)? Please state the exact objective controlled by Theorem 4 and establish the claimed stationarity implication. Also clarify how Algorithm 3 enforces and positivity of the noisy scalar estimate used in and reciprocal terms. As written, the initialization and unconstrained updates do not supply these conditions, while Lemma 7 uses a logarithm inequality requiring positive arguments bounded away from zero.
-
Does Algorithm 2 clip individual contributions before averaging, or the batch average as written? Please provide the neighboring-dataset sensitivity and composition calculation for the implemented mechanism. Where does the utility proof control clipping bias, or establish that clipping is inactive for all four estimators? A correction to the clipping operation should be accompanied by the corresponding estimator-error argument.
-
Was the pretraining described in Appendix E.4 performed on public data, independently privatized, or included in the reported privacy accounting? Please specify the accounting for pretraining, warm starts, and any data-dependent model selection. For Table 2, how are members and nonmembers sampled, including their class distributions? Please reconcile the reported AUCs at with the stated privacy parameters and explain whether the attack represents a matched neighboring-dataset membership test.
-
What exact quantity is plotted in Figure 1, at which privacy budget, and is the legend correct? Under the displayed legend, the Diff baseline has the lowest terminal gradient norm in all four panels, while the proposed methods’ norms increase in several cases. This conflicts with the interpretation in lines 422–431 and 452–454. Please clarify whether all curves measure stationarity of the same objective under comparable initialization and computational settings, and reconcile the figure with the claimed convergence advantage.
I think the paper addresses an important problem, and I appreciate the attempt to develop differentially private algorithms specifically for nonconvex distributionally robust optimization. The proposed combination of variance reduction and privacy-preserving gradient estimation is interesting, and the authors have included a reasonable range of experiments. However, I have significant concerns about the theoretical justification of the main results. In particular, the assumptions underlying the convergence analysis, the relationship between the original DRO objective and the KL-based reformulation, and the consistency between the clipping mechanisms and privacy proofs need further clarification. I also find that some experimental conclusions are difficult to reconcile with the reported gradient-norm curves and membership-inference results. These issues are important because they directly affect the validity of the paper's main claims, rather than being merely presentation concerns. Overall, I see potential in the research direction, but I do not think the current manuscript provides sufficiently convincing support for its theoretical and empirical conclusions.