- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes DP-BiSGD, which randomly alternates between frequent small-noise updates and occasional large-noise updates using a data-independent coin. Small-noise steps update momentum normally, while large-noise steps bypass the momentum buffer and receive additional scaling. The intended benefit is to preserve useful gradient information during most iterations without allowing occasional large perturbations to contaminate future updates. A Gaussian-mixture privacy-loss accountant is used to determine the allowable training duration. Experiments report accuracy improvements over DP-SGD, particularly on CIFAR-10, but those comparisons depend critically on whether the proposed accountant correctly bounds privacy loss.
Critical strengths
- Combining two noise scales with a momentum bypass is an algorithmic idea worth investigating. The ablations help examine how its components affect utility, although comparisons at matched privacy budgets depend on correcting the accountant.
Critical weaknesses
- The one-dimensional privacy reduction in Section 2.4 is not justified by isotropy. Because all coordinates share the same randomly selected noise scale, orthogonal coordinates reveal information about that scale. Projecting them away can reduce distinguishability and cannot simply replace the full-dimensional privacy analysis.
- The claimed conservative discretization is incorrect. Mapping privacy losses above back to decreases them, contradicting Appendix A.1.2’s assertion that the mapped loss is always larger. Consequently, the stated upper bound on privacy failure probability is not established.
- The convergence proof analyzes an update without the algorithm’s momentum mechanism and assumes an unbiased gradient. Clipping generally introduces bias, which bounded variance alone does not address. The theorem therefore does not establish convergence for the implemented method.
Questions for authors
Can authors respond the the weakness part? I have used AI tool to do the review since I do not have enough time, but it seems the claims are not supported by correct proofs. I will revise my points if the incorrectness claims turns out to be invalid.