- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes DP-RoLL for learning low-rank multivariate response models from contaminated data under DP. It combines a low-rank coefficient matrix with row-sparse contamination adjustments, using Gaussian-perturbed coefficient updates and unperturbed row-wise thresholding for the internal contamination variables. The paper presents a privacy guarantee and conditional error bounds separating optimization, statistical, and privacy effects. Evaluation includes synthetic classification and regression, and real data applications. The reported results suggest improved utility under contamination compared with the selected private baselines.
Critical strengths
-
A privacy design that accounts for the internal contamination updates.
-
Broad empirical evidence supporting the proposed combination.
Critical weaknesses
- The statistical guarantees are not sufficiently connected to the actual algorithm and experimental models.
Theorem 3 assumes regularity on a sparsity-restricted set, but the proof does not establish that the hard-thresholding iterates remain in that set before applying the condition in Eq. (B.22). The Eq. (13) also has applicability problems for logistic loss and, together with the step-size requirement for Eq. (15), for the standardized squared-loss setting. Although the manuscript notes that the condition need only hold along the trajectory, it does not establish the corresponding trajectory-level guarantee. Moreover, the statistical analysis assumes inactive clipping, leaving the actively clipped regression implementation outside the established analysis. These issues materially weaken the theoretical contribution, although they do not by themselves invalidate the privacy guarantee.
- The experimental privacy design and complete release interface remain insufficiently specified.
The relationship between the public pilot data, private training records, and final test records is not sufficiently clear to assess the intended privacy boundary and independence of the evaluation. In addition, the experiments release a privatized intercept, whereas the main mechanism and privacy theorem are formulated for the coefficient matrix; the corresponding sensitivity and accounting details are not fully described.
Questions for authors
Except weaknesses,
-
For the standardized squared-loss design in Appendix D.1, what range of simultaneously satisfies Eq. (13) and the step-size requirement preceding Eq. (15)?
-
What ensures at every iteration where the regularity condition is applied in Eq. (B.22)?
-
For logistic loss, on what local parameter region does Eq. (13) hold, and with what probability do the iterates remain in that region?
-
Which records, if any, are shared between the public pilot splits and the final training or test splits, and which records are treated as private?
-
How is the intercept privatized, and what joint sensitivity bound or budget allocation covers its release together with ?
8 comments
Response to W1
W1
Theorem 3 assumes regularity on a sparsity-restricted set, but the proof does not establish that the hard-thresholding iterates remain in that set before applying the condition in Eq. (B.22). The Eq. (13) also has applicability problems for logistic loss and, together with the step-size requirement for Eq. (15), for the standardized squared-loss setting. Although the manuscript notes that the condition need only hold along the trajectory, it does not establish the corresponding trajectory-level guarantee. Moreover, the statistical analysis assumes inactive clipping, leaving the actively clipped regression implementation outside the established analysis. These issues materially weaken the theoretical contribution, although they do not by themselves invalidate the privacy guarantee.
Response:
We thank the reviewer for the detailed comments. We first present an extension of Theorem 3 to account for active clipping, and then clarify the applicability of the regularity conditions to our experimental models.
Extension of Theorem 3 to active clipping
The original proof of Theorem 3 in Appendix B.3 considers the regime where clipping is inactive. We have extended the statistical analysis to account for active clipping under an additional bounded-gradient-moment assumption.
Specifically, assume that the empirical second moment of the unclipped sample gradients is uniformly bounded along the iterates:
Similar bounded-moment assumptions have been used in analyses of gradient clipping and differentially private optimization [1, 2].
By the clipping rule in Eq. (6), the clipping residual satisfies
Extension of Theorem 3. Under the assumptions of Theorem 3 and the additional condition above, the following bound holds with probability at least : Here, is defined by Eq. (B.19) with .
Proof sketch. Following Eqs. (B.15)–(B.16), clipping introduces an additional inner product between the clipping residual and . By Young's inequality and the clipping residual bound above, this term is bounded by . The Gaussian noise term contributes another when . Applying Eq. (13) and iterating as in Eq. (B.23) yields the stated bound.
The additional clipping-dependent term bounds the contribution of active clipping to the statistical error guarantee. Moreover, since the Gaussian noise variance scales with , the extended bound reveals a clipping–privacy trade-off: increasing reduces the clipping-error bound but increases the privacy-error bound. We have incorporated the extended theorem and its detailed proof into Appendix B.7 of the revised manuscript.
Empirical verification of the regularity conditions
As noted following Theorem 3, the regularity condition in Eq. (13) only needs to hold along the algorithmic trajectory. We therefore examine its applicability to our experimental models through numerical diagnostics along the iterates.
Specifically, we investigate (i) the compatibility between Eq. (13) and the step-size requirement for standardized squared-loss regression (Q1), (ii) the sparsity of the hard-thresholding iterates (Q2), and (iii) the validity of Eq. (13) and trajectory containment under logistic loss (Q3).
Our numerical observations indicate that the relevant conditions are satisfied along the observed trajectories in the evaluated settings. Detailed numerical results are provided in our responses to Q1–Q3 and have been incorporated into Appendix B.4.1 of the revised manuscript. Complementing these numerical findings, Appendix B.4 discusses sufficient conditions for Eq. (13) in terms of curvature and penalty-tolerance requirements, with squared loss serving as an illustrative example.
Turning to empirical performance, the results in Section 5 and Appendices C and D demonstrate the effectiveness of DP-RoLL in multi-label classification and multivariate regression, with strong estimation accuracy and predictive performance under data contamination and differential privacy.
References
[1] Cutkosky, A., and Mehta, H. (2021). High-Probability Bounds for Non-Convex Stochastic Optimization with Heavy Tails. NeurIPS 2021.
[2] Das, R., Kale, S., Xu, Z., Zhang, T., and Sanghavi, S. (2023). Beyond Uniform Lipschitz Condition in Differentially Private Optimization. ICML 2023.
Response to W2
W2
The relationship between the public pilot data, private training records, and final test records is not sufficiently clear to assess the intended privacy boundary and independence of the evaluation. In addition, the experiments release a privatized intercept, whereas the main mechanism and privacy theorem are formulated for the coefficient matrix; the corresponding sensitivity and accounting details are not fully described.
Response:
We thank the reviewer for these comments. We clarify the data partitioning and privacy boundary in our response to Q4, and the joint privacy mechanism and accounting for the coefficient matrix and intercept in our response to Q5.
Author Response
We sincerely thank you for the careful reading of our manuscript and the constructive comments and suggestions. We have carefully considered the feedback and revised the manuscript accordingly. Detailed point-by-point responses are provided below, with the corresponding revisions highlighted in red in the manuscript.
For clarity, our detailed responses to each weakness and question are provided in separate comments below.