acceptodds
Under review as a conference paper at ICLR 2027

DP-RoLL: Fortifying Private Low-Rank Learning against Data Contamination

Abstract

Multivariate response learning often involves sensitive individual-level data and corrupted observations. Low-rank coefficient structure can improve statistical efficiency under differential privacy by exploiting shared response factors, but its benefits may be compromised by outliers. In this work, we propose a differentially private and robust low-rank learning framework for a broad range of loss functions. Theoretically, we establish a non-asymptotic error bound that decomposes the error into optimization, statistical, and privacy-induced components and reveals an optimization-privacy trade-off in the number of private iterations. Extensive experiments demonstrate the effectiveness of our proposed framework in protecting sensitive data while maintaining strong performance in the presence of data anomalies.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
robust learning, differential privacy, low-rank learning, non-asymptotic analysis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_rHSkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DP-RoLL for learning low-rank multivariate response models from contaminated data under DP. It combines a low-rank coefficient matrix with row-sparse contamination adjustments, using Gaussian-perturbed coefficient updates and unperturbed row-wise thresholding for the internal contamination variables. The paper presents a privacy guarantee and conditional error bounds separating optimization, statistical, and privacy effects. Evaluation includes synthetic classification and regression, and real data applications. The reported results suggest improved utility under contamination compared with the selected private baselines.

Critical strengths
  1. A privacy design that accounts for the internal contamination updates.

  2. Broad empirical evidence supporting the proposed combination.

Critical weaknesses
  1. The statistical guarantees are not sufficiently connected to the actual algorithm and experimental models.

Theorem 3 assumes regularity on a sparsity-restricted set, but the proof does not establish that the hard-thresholding iterates remain in that set before applying the condition in Eq. (B.22). The Eq. (13) also has applicability problems for logistic loss and, together with the step-size requirement for Eq. (15), for the standardized squared-loss setting. Although the manuscript notes that the condition need only hold along the trajectory, it does not establish the corresponding trajectory-level guarantee. Moreover, the statistical analysis assumes inactive clipping, leaving the actively clipped regression implementation outside the established analysis. These issues materially weaken the theoretical contribution, although they do not by themselves invalidate the privacy guarantee.

  1. The experimental privacy design and complete release interface remain insufficiently specified.

The relationship between the public pilot data, private training records, and final test records is not sufficiently clear to assess the intended privacy boundary and independence of the evaluation. In addition, the experiments release a privatized intercept, whereas the main mechanism and privacy theorem are formulated for the coefficient matrix; the corresponding sensitivity and accounting details are not fully described.

Questions for authors

Except weaknesses,

  1. For the standardized squared-loss design in Appendix D.1, what range of ρ\rho simultaneously satisfies Eq. (13) and the step-size requirement preceding Eq. (15)?

  2. What ensures ∣∣C(t)∣∣2,0≤CsJ||C^{(t)}||_{2,0}\le C_sJ at every iteration where the regularity condition is applied in Eq. (B.22)?

  3. For logistic loss, on what local parameter region does Eq. (13) hold, and with what probability do the iterates remain in that region?

  4. Which records, if any, are shared between the public pilot splits and the final training or test splits, and which records are treated as private?

  5. How is the intercept privatized, and what joint sensitivity bound or budget allocation covers its release together with BB?

8 comments
AuthorsOpenReview ↗

Author Response

We sincerely thank you for the careful reading of our manuscript and the constructive comments and suggestions. We have carefully considered the feedback and revised the manuscript accordingly. Detailed point-by-point responses are provided below, with the corresponding revisions highlighted in red in the manuscript.

For clarity, our detailed responses to each weakness and question are provided in separate comments below.

AuthorsOpenReview ↗

Response to W1

W1

Theorem 3 assumes regularity on a sparsity-restricted set, but the proof does not establish that the hard-thresholding iterates remain in that set before applying the condition in Eq. (B.22). The Eq. (13) also has applicability problems for logistic loss and, together with the step-size requirement for Eq. (15), for the standardized squared-loss setting. Although the manuscript notes that the condition need only hold along the trajectory, it does not establish the corresponding trajectory-level guarantee. Moreover, the statistical analysis assumes inactive clipping, leaving the actively clipped regression implementation outside the established analysis. These issues materially weaken the theoretical contribution, although they do not by themselves invalidate the privacy guarantee.

Response:

We thank the reviewer for the detailed comments. We first present an extension of Theorem 3 to account for active clipping, and then clarify the applicability of the regularity conditions to our experimental models.

Extension of Theorem 3 to active clipping

The original proof of Theorem 3 in Appendix B.3 considers the regime where clipping is inactive. We have extended the statistical analysis to account for active clipping under an additional bounded-gradient-moment assumption.

Specifically, assume that the empirical second moment of the unclipped sample gradients is uniformly bounded along the iterates:

max⁡0≤t<T1n∑i=1n∥Ui(t)∥F2≤MU.\max_{0\le t<T}\frac{1}{n}\sum_{i=1}^n \lVert U_i^{(t)}\rVert_F^2 \le M_U. Similar bounded-moment assumptions have been used in analyses of gradient clipping and differentially private optimization [1, 2].

By the clipping rule in Eq. (6), the clipping residual satisfies

∥∑i=1n(U~i(t)−Ui(t))∥F≤nMU4CB.\lVert \sum_{i=1}^n(\tilde U_i^{(t)}-U_i^{(t)}) \rVert_F \le \frac{nM_U}{4C_B}. Extension of Theorem 3. Under the assumptions of Theorem 3 and the additional condition above, the following bound holds with probability at least 1−O(n−c0)1-O(n^{-c_0}): Δψ(Bˉ∗,Bˉ(T))≤(1+τ)−TΔψ(Bˉ∗,Bˉ(0))+1−(1+τ)−Tτ[e(T)+n2MU216K2CB2]−1−(1+τ)−Tτmin⁡1≤t≤TΔψ(Bˉ(t),Bˉ(t−1)).\Delta_\psi(\bar B^\ast,\bar B^{(T)}) \le (1+\tau)^{-T}\Delta_\psi(\bar B^\ast,\bar B^{(0)}) + \frac{1-(1+\tau)^{-T}}{\tau}\left[e(T)+\frac{n^2M_U^2}{16K_2C_B^2}\right] - \frac{1-(1+\tau)^{-T}}{\tau}\min_{1\le t\le T}\Delta_\psi(\bar B^{(t)},\bar B^{(t-1)}). Here, e(T)e(T) is defined by Eq. (B.19) with aDP=2/K2a_{\mathrm{DP}}=2/K_2.

Proof sketch. Following Eqs. (B.15)–(B.16), clipping introduces an additional inner product between the clipping residual and B∗−B(t+1)B^\ast-B^{(t+1)}. By Young's inequality and the clipping residual bound above, this term is bounded by (K2/2)D2(B(t+1),B∗)+n2MU2/(16K2CB2)(K_2/2)D_2(B^{(t+1)},B^\ast)+n^2M_U^2/(16K_2C_B^2). The Gaussian noise term contributes another (K2/2)D2(B(t+1),B∗)(K_2/2)D_2(B^{(t+1)},B^\ast) when aDP=2/K2a_{\mathrm{DP}}=2/K_2. Applying Eq. (13) and iterating as in Eq. (B.23) yields the stated bound.

The additional clipping-dependent term bounds the contribution of active clipping to the statistical error guarantee. Moreover, since the Gaussian noise variance scales with CB2C_B^2, the extended bound reveals a clipping–privacy trade-off: increasing CBC_B reduces the clipping-error bound but increases the privacy-error bound. We have incorporated the extended theorem and its detailed proof into Appendix B.7 of the revised manuscript.

Empirical verification of the regularity conditions

As noted following Theorem 3, the regularity condition in Eq. (13) only needs to hold along the algorithmic trajectory. We therefore examine its applicability to our experimental models through numerical diagnostics along the iterates.

Specifically, we investigate (i) the compatibility between Eq. (13) and the step-size requirement for standardized squared-loss regression (Q1), (ii) the sparsity of the hard-thresholding iterates C(t)\boldsymbol{C}^{(t)} (Q2), and (iii) the validity of Eq. (13) and trajectory containment under logistic loss (Q3).

Our numerical observations indicate that the relevant conditions are satisfied along the observed trajectories in the evaluated settings. Detailed numerical results are provided in our responses to Q1–Q3 and have been incorporated into Appendix B.4.1 of the revised manuscript. Complementing these numerical findings, Appendix B.4 discusses sufficient conditions for Eq. (13) in terms of curvature and penalty-tolerance requirements, with squared loss serving as an illustrative example.

Turning to empirical performance, the results in Section 5 and Appendices C and D demonstrate the effectiveness of DP-RoLL in multi-label classification and multivariate regression, with strong estimation accuracy and predictive performance under data contamination and differential privacy.

References

[1] Cutkosky, A., and Mehta, H. (2021). High-Probability Bounds for Non-Convex Stochastic Optimization with Heavy Tails. NeurIPS 2021.

[2] Das, R., Kale, S., Xu, Z., Zhang, T., and Sanghavi, S. (2023). Beyond Uniform Lipschitz Condition in Differentially Private Optimization. ICML 2023.

AuthorsOpenReview ↗

Response to W2

W2

The relationship between the public pilot data, private training records, and final test records is not sufficiently clear to assess the intended privacy boundary and independence of the evaluation. In addition, the experiments release a privatized intercept, whereas the main mechanism and privacy theorem are formulated for the coefficient matrix; the corresponding sensitivity and accounting details are not fully described.

Response:

We thank the reviewer for these comments. We clarify the data partitioning and privacy boundary in our response to Q4, and the joint privacy mechanism and accounting for the coefficient matrix and intercept in our response to Q5.

Reviewer_1zJ2OpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3(+1)
Soundness
3
Read review
Paper summary

This paper considers the problem of privately training a low-rank multivariate response model. In this problem, we are given a dataset comprised of features X and labels Y, and want to fit a linear model B to minimize some loss l(XB, Y). The key difference from GLMs is that the labels are now vectors (i.e., multiple outcomes are being simultaneously fit for every sample), and so instead of fitting a general model B, we limit our attention to low rank models B (this is both a form of regularization that utilizes the fact that we are fitting multiple prediction variables at once and in some sense asking the training to extract a useful low dimensional embedding of the features).

The paper introduces a new differentially private algorithm for this problem. As with many private algorithms, the key challenge is dealing with outliers, which might otherwise have a significant effect on the model, causing information leakage in the form of membership inference on these outliers.

To counteract this, the authors introduce an approach based on allowing the training loss to "give up" on some samples at a fixed cost per sample. This is done by adding the C parameter which allows the model to fix responses to some of the training samples (thus making their loss contribution effectively 0), and adding an L_0-esque penalty for C.

The authors show that this additional penalty can be used to force the training to be robust, and then privatize it using a Gaussian mechanism. Finally, this new method is evaluated experimentally vs previous private algorithms for this setting.

Overall, I really like this paper. It is well written and presents new ideas (at least new to me) that improve upon the previous state of the art in an interesting problem.

I think the biggest room for improvement would be to separate out the different contributions in a clearer way. Thinking of DP-SGD for standard GLMs / full rank B as a more standard baseline, to my view the paper considers an algorithm three steps removed from this baseline:

  1. Outliers are dealt with by allowing for an L_0 regularized C term in the model.
  2. Privacy is obtained by proving a bound on how each training sample affects the model instead of the more standard approaches of standard composition or advanced composition commonly used for DP-SGD.
  3. The focus is shifted to a less standard problem of low-degree fitting for multivariate data.

To my understanding, these 3 components are somewhat independent in the sense that any subset of them could work (for instance, the robustness to outliers could be used to obtain privacy by showing that any single step of the optimization is robust and then adding noise at each step to make it private and then using the standard arguments of advanced composition for Gaussian mechanism used in typical DP-SGD algorithms would allow us to use step 1 without 2; I think -- though I am not sure -- that the same multi-step robustness argument could have been applied to more standard robust SGD methods like gradient clipping even without the C term approach; and it seems like all of these methods should be applicable to more standard problems like GLMs / full rank models for multivariate data).

A clearer explanation of the significance of each component of the algorithm (whether it is experimentally through ablation, through theory, intuition or referencing any relevant literature) would help me (and hopefully other readers) better understand the papers contributions beyond this very specific setting.

Critical strengths

The paper improves upon the state of the art for an interesting problem by using an interesting and seemingly very generalizable method for dealing with outliers in training.

The experimental evaluations seems sound and I think the writing was mostly good (up to fixable issues listed below).

Critical weaknesses

I found it hard to understand how this papers contributions correspond to the existing literature of DP: is the approach of adding an L0 regularized component allowing the model to miss some samples a key contribution / how novel is this? does the approach of making an iterative optimization private by viewing the whole process as robust and adding noise in the end beat the more standard approach of adding noise at each step and using advanced composition to ensure the privacy of the whole algorithm? which (if any) of these contributions would we expect to still work and be novel for other settings such as fitting a GLM?

I found most of the paper to be very well written, but pages 4-6 were very notation / math heavy. Some degree of notation / math is unavoidable in a paper like this, but I think that the writing here could be improved (e.g., I did not follow the details in Theorem 3 at all and I think that a more high level / informal theorem with details in an appendix might be more useful, and the equation on line 172 takes up a lot of attention but in my opinion it does not add a lot compared to the clearer equation (8)).

Questions for authors

See above.

5 comments
AuthorsOpenReview ↗

Author Response

We thank the reviewer for the encouraging assessment of our work and the constructive suggestions on clarifying its contributions and improving the presentation. We have carefully addressed the comments and revised the manuscript accordingly, with all revisions highlighted in red. Our point-by-point responses are provided below.

For clarity, our detailed responses to each weakness are provided in separate comments below.

AuthorsOpenReview ↗

Response to W1

W1

I found it hard to understand how this papers contributions correspond to the existing literature of DP: is the approach of adding an L0L_0 regularized component allowing the model to miss some samples a key contribution / how novel is this? does the approach of making an iterative optimization private by viewing the whole process as robust and adding noise in the end beat the more standard approach of adding noise at each step and using advanced composition to ensure the privacy of the whole algorithm? which (if any) of these contributions would we expect to still work and be novel for other settings such as fitting a GLM?

Response:

Thank you for the thoughtful questions. We highlight three key contributions of DP-RoLL.

Novelty and significance of the problem setting

While L0L_0-regularized mean-shift modeling is established in robust statistics, to the best of our knowledge, DP-RoLL is the first framework for robust low-rank multivariate response learning under differential privacy in the presence of outliers. As summarized in Table A.1, existing methods address only subsets of the requirements of this important yet underexplored problem, whereas DP-RoLL jointly addresses data contamination, low-rank structure, and differential privacy in multivariate response learning, with corresponding statistical and optimization theory.

Privacy-preserving optimization and sensitivity analysis

We introduce an auxiliary matrix C\boldsymbol{C} to capture outlying effects. By exploiting the row-wise separability of its updates, we establish bounded sensitivity for the B\boldsymbol{B}-gradient despite the data dependence of C\boldsymbol{C} (Eq. (9)). This enables Gaussian noise to be added only to the clipped B\boldsymbol{B}-gradient at each iteration, while C\boldsymbol{C} remains an internal variable without direct noise perturbation, thereby avoiding additional noise in the high-dimensional nuisance matrix and helping preserve statistical utility.

The overall privacy guarantee follows from the Gaussian mechanism and zCDP composition across iterations (Theorem 2), rather than noise addition only at the end of optimization. The key distinction from standard iterative DP methods lies in accommodating data-dependent robustification through sensitivity analysis. Our framework also supports alternative DP composition techniques, with the corresponding Gaussian noise variance adjusted accordingly while preserving the forms of the theoretical bounds under the same assumptions.

Theoretical analysis and broader applicability

We establish a non-asymptotic error bound that separates optimization, statistical, and privacy-induced errors, explicitly characterizing their dependence on model rank, contamination level, and privacy budget (Theorem 3). The bound also reveals an optimization–privacy trade-off: under a fixed privacy budget, increasing the number of iterations decreases the optimization-error term but increases the privacy-induced error term. Complementing the theoretical analysis, the ablation studies in Section 5.5 and Appendix C.6 illustrate the respective benefits of robustification and low-rank modeling.

Beyond squared-loss regression, DP-RoLL accommodates general differentiable sample-separable losses, including logistic loss used in our multi-label classification experiments. In particular, the sensitivity analysis for data-dependent robustification remains a key technical contribution applicable to GLMs, as it relies on sample-wise separability and the row-wise updates of C\boldsymbol{C} rather than a specific loss function. The non-asymptotic error and convergence analyses also extend to GLMs under their respective assumptions (Theorems 3 and 5).

Overall, DP-RoLL provides a principled solution to robust low-rank multivariate learning under differential privacy, with rigorous privacy guarantees and non-asymptotic error analysis.

To better highlight these contributions, we have revised the contribution summary in Section 1.

AuthorsOpenReview ↗

Response to W2

W2

I found most of the paper to be very well written, but pages 4-6 were very notation / math heavy. Some degree of notation / math is unavoidable in a paper like this, but I think that the writing here could be improved (e.g., I did not follow the details in Theorem 3 at all and I think that a more high level / informal theorem with details in an appendix might be more useful, and the equation on line 172 takes up a lot of attention but in my opinion it does not add a lot compared to the clearer equation (8)).

Response:

Thank you for the constructive suggestions on improving the presentation. We have revised Sections 3 and 4 to make the algorithmic formulation and theoretical results more accessible.

Improving the Algorithmic Presentation

The surrogate formulation provides the optimization basis for deriving the explicit updates in Eq. (8), and retaining it helps clarify the connection between the majorization–minimization framework and the resulting algorithm. To improve readability, we have revised Section 3 by presenting the surrogate formulation inline, adding a high-level explanation of the algorithmic design, and clarifying the roles of the low-rank projection and row-wise hard-thresholding steps in Eq. (8).

Improving the Presentation of the Theoretical Analysis

We have also revised Section 4 to make the theoretical analysis more accessible. Specifically, we have clarified the motivation for introducing generalized Bregman functions, provided a high-level explanation of Theorem 3 before its formal statement, and added intuition for the role of the regularity condition. We have also reformatted Eq. (15) to explicitly distinguish the optimization, statistical, and privacy-induced error components, making their dependence on the model rank, contamination level, and privacy noise more transparent. In addition, we have clarified the optimization–privacy trade-off in the convergence analysis. These revisions aim to make the main theoretical insights easier to follow, with detailed proofs and further discussions of the regularity conditions provided in Appendix B.

Reviewer_ASepOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the differential privacy mechanism of multivariate learning with low-rank coefficients and contaminated data points. Without differential privacy, the existing work (She and Chen (2017)) has already studied this problem without differential privacy. And this paper adds the gradient privacy mechanism to it.

For me, there are two interesting things:

  1. The authors show that, the additional mean-shift parameter matric is just for contaminated data points. Hence, there is no need to make it public. Therefore, the privacy mechanism is applied to the public parameter matrix B only, which reduces the privacy cost.
  2. The results on four real data sets look very nice.

The main concern is the novelty: there is no prior work about DP on reduced-rank robust regression, so it is the first one. When I read the paper, I used AI for better understanding.

Critical strengths
  1. This is the first work on differential private robust reduced-rank multivariate regression problem. The results on four real data sets strongly support the effectiveness of this algorithm.

  2. On the differential privacy mechanism, the authors show that, the additional mean-shift parameter matric is just for contaminated data points. Hence, there is no need to make it public. Therefore, the privacy mechanism is applied to the public parameter matrix B only, which reduces the privacy cost. It is very smart.

Critical weaknesses
  1. After reading this paper, I have not learned new knowledge about the privacy mechanism. The gradient descent algorithm is standard. On the regression-wise, the robust reduced-rank regression idea is very brilliant and useful, but it is not from this paper. Therefore, I hope the authors can clarify the core contribution of this paper. How it deepens our understanding about this problem?

  2. The choice of tuning parameters, especially λC\lambda_C, may also reveal data information. I wonder how the authors deal with this problem?

I would like to raise my score if these two questions can be well addressed.

Questions for authors

Some clarification questions. If not properly answered, I may decrease my rating.

  1. For real data, all the quantities are normalized to be in the interval [0, 1]. I wonder what is the absolute improvement?
  2. Is there an intercept in this problem? I read "intercept" in line 458, but I cannot find it in the model and theoretical analysis. If yes, authors should consider the privacy on the intercept and apply the composition theorem.
  3. The hyperparameter choices in the real datasets, and whether it affects the privacy.
  4. The proof of Theorem 5 ignores the clipping step in Appendix B.5.
9 comments
AuthorsOpenReview ↗

Author Response

We thank the reviewer for the positive assessment of our work and the insightful comments on its core contributions, privacy guarantees, and theoretical analysis. We have carefully addressed the comments and revised the manuscript accordingly, with all revisions highlighted in red. Our point-by-point responses are provided below.

For clarity, our detailed responses to each weakness and question are provided in separate comments below.

AuthorsOpenReview ↗

Response to W1

W1

After reading this paper, I have not learned new knowledge about the privacy mechanism. The gradient descent algorithm is standard. On the regression-wise, the robust reduced-rank regression idea is very brilliant and useful, but it is not from this paper. Therefore, I hope the authors can clarify the core contribution of this paper. How it deepens our understanding about this problem?

Response:

We thank the reviewer for the insightful comment. We clarify the core contributions of DP-RoLL and explain how they advance our understanding of robust low-rank learning under differential privacy.

Novelty and significance of the problem setting

While robust reduced-rank regression and L0L_0-regularized mean-shift modeling are established in robust statistics, to the best of our knowledge, DP-RoLL is the first framework for robust low-rank multivariate response learning under differential privacy in the presence of outliers. As summarized in Table A.1, existing methods address only subsets of the requirements of this important yet underexplored problem, whereas DP-RoLL jointly addresses data contamination, low-rank structure, and differential privacy in multivariate response learning, with corresponding statistical and optimization theory.

Privacy-preserving optimization and sensitivity analysis

We introduce an auxiliary matrix C\boldsymbol{C} to capture outlying effects. A key challenge is that the data-dependent updates of C\boldsymbol{C} influence subsequent gradients with respect to B\boldsymbol{B}, complicating the privacy analysis. By exploiting the row-wise separability of these updates, we establish bounded sensitivity for the B\boldsymbol{B}-gradient despite the dependence on C\boldsymbol{C} (Eq. (9)).

This enables Gaussian noise to be added only to the clipped B\boldsymbol{B}-gradient at each iteration, while C\boldsymbol{C} remains an internal variable without direct noise perturbation, thereby avoiding additional noise in the high-dimensional nuisance matrix and helping preserve statistical utility. The overall privacy guarantee follows from the Gaussian mechanism and zCDP composition (Theorem 2). Our key technical contribution is the sensitivity analysis that enables data-dependent robustification within iterative DP optimization. Our framework also accommodates alternative DP composition techniques through appropriate calibration of the Gaussian noise variance, while preserving the forms of the theoretical bounds under the same assumptions.

Theoretical insights into statistical accuracy

We establish a non-asymptotic error bound for robust low-rank multivariate learning under differential privacy, explicitly separating optimization, statistical, and privacy-induced errors under the stated regularity conditions (Theorem 3). The bound quantifies the effects of model rank and contamination on statistical accuracy, as well as the additional estimation error induced by differential privacy and its dependence on the privacy budget. Moreover, it reveals an optimization–privacy trade-off: under a fixed privacy budget, increasing the number of iterations decreases the optimization-error term but increases the privacy-induced error term. These results clarify how low-rank structure, data contamination, and privacy constraints jointly affect estimation accuracy, while providing a theoretical basis for balancing optimization progress against privacy-induced error.

Overall, DP-RoLL provides a principled framework for robust low-rank multivariate learning under differential privacy, with a tailored sensitivity analysis and non-asymptotic error guarantees that advance our understanding of the interplay between robustness, low-rank structure, and privacy.

To better highlight these contributions, we have revised the contribution summary in Section 1.

AuthorsOpenReview ↗

Response to W2

W2

The choice of tuning parameters, especially λC\lambda_C, may also reveal data information. I wonder how the authors deal with this problem?

Response:

We thank the reviewer for this insightful question. As described in Sections 5.1 and 5.3 and detailed in Appendix C.12, all tuned hyperparameters, including λC\lambda_C, are selected using public pilot data. For synthetic experiments, these data are generated independently of the private training data. For real-world experiments, calibration uses two prespecified pilot splits treated as public data. All selected hyperparameters are fixed before private training, so their selection incurs no additional privacy cost. The stated (ε,δ)(\varepsilon,\delta)-DP guarantee applies to the private training mechanism conditional on these fixed public hyperparameters.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Responses to Reviewers

We thank all three reviewers for their careful evaluations and constructive feedback. We have provided detailed point-by-point responses and revised the manuscript, with all revisions highlighted in red. The main responses and revisions are summarized below.

  • Core contributions and methodological novelty (Reviewer ASep, W1; Reviewer 1zJ2, W1). We clarified that, to the best of our knowledge, DP-RoLL is the first framework for robust low-rank multivariate response learning under differential privacy in the presence of outliers. We emphasized the key methodological novelty: a tailored sensitivity analysis that exploits the row-wise separability of the data-dependent contamination updates to bound the sensitivity of the coefficient gradient. This enables Gaussian noise to be added only to the clipped coefficient gradient at each iteration, without directly perturbing the high-dimensional contamination matrix, thereby helping preserve statistical utility. We also highlighted the non-asymptotic error bound separating optimization, statistical, and privacy-induced errors, the resulting optimization–privacy trade-off, and the applicability of the framework to GLMs under the stated assumptions. We revised the contribution summary in Section 1 accordingly.
  • Statistical and convergence guarantees under active clipping (Reviewer rHSk, W1; Reviewer ASep, Q4). We extended Theorems 3 and 5 to account for active clipping under an additional bounded-gradient-moment assumption. The resulting bounds characterize clipping-induced errors and the clipping–privacy trade-off, with detailed proofs added to Appendix B.7.
  • Joint privacy accounting for the coefficient matrix and intercept (Reviewer ASep, Q2; Reviewer rHSk, W2 and Q5). We clarified that the intercept, used only in real-world multi-label classification, is incorporated into an augmented coefficient matrix. Joint per-sample gradient clipping yields a replace-one sensitivity bound of 2CB2C_B, while the Gaussian mechanism and zCDP composition account for the joint release of the coefficient matrix and intercept without requiring an additional privacy budget. We revised Section 4.1 accordingly.
  • Public calibration and data-splitting protocol (Reviewer ASep, W2 and Q3; Reviewer rHSk, W2 and Q4). We clarified that hyperparameters are selected using public pilot data and fixed before private training, without additional privacy cost. Synthetic pilot data are generated independently of the final training and test data. For real-world experiments, two public pilot splits are used for hyperparameter selection, with the selected hyperparameters fixed across 20 evaluation splits drawn from the same public benchmarks. Within each evaluation split, the training data are treated as private, and the test data are held out from model training. Further details are provided in Appendix C.12.
  • Absolute improvements on real-world benchmarks (Reviewer ASep, Q1). We reported original-scale metric values and absolute improvements over the best-performing baseline for each dataset and metric, covering four representative datasets and five evaluation metrics. These results have been included in Appendix C.5.
  • Improvements to algorithmic and theoretical presentation (Reviewer 1zJ2, W2). We revised Sections 3 and 4 by presenting the surrogate formulation inline, explaining the main algorithmic updates, and adding intuition for Theorem 3 and its regularity conditions. We also reformatted Eq. (15) to distinguish optimization, statistical, and privacy-induced errors.
  • Regularity conditions and numerical diagnostics (Reviewer rHSk, W1 and Q1–Q3). We examined the compatibility of the regularity and step-size conditions for squared loss, the sparsity of the hard-thresholding iterates, and trajectory containment under logistic loss. We added numerical diagnostics in Appendix B.4.1 to assess these conditions along the observed trajectories.

We believe these responses and revisions have strengthened the manuscript and clarified the contributions of DP-RoLL.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.