- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes privacy-preconditioned representation learning, where public training explicitly prepares a representation for later differentially private adaptation. The authors introduce two regularization methods that penalize the average or maximum norm of projected features to reduce the effects of gradient clipping and DP noise. The paper provides theoretical excess-risk bounds for linear regression and evaluates the methods on synthetic and real-world regression tasks.
Critical strengths
- Designing public representations specifically for downstream DP training is a reasonable idea. The analysis of how feature norms affect gradient clipping and privacy noise provides useful insights.
- The authors include non-private comparisons, clipping/noise ablations, matched-threshold experiments, and independent repetitions. These help explain why the proposed regularization improves private utility in the synthetic setting.
Critical weaknesses
1. The experimental improvements are limited.
The strongest result is an 8.8% improvement on a specially constructed synthetic regression problem. However, the real-world results are much less convincing. On ILEA School, the regularized method initially achieves a 10–13% improvement, but a simple rank-one baseline matches or outperforms it. This suggests that the benefit mainly comes from reducing effective dimension rather than the proposed privacy-aware regularization. On CT Slice Localization, improvements are only 0.08–0.97%. The maximum-norm method also fails to demonstrate a statistically clear improvement.
Overall, I do not think the experiments establish a consistent practical advantage.
2. The theory is not well connected to the main experiments.
The theoretical guarantees rely on bounded linear models, projected task parameters, and, for the maximum-norm method, a strong public-data coverage assumption. However, the main positive synthetic experiment uses a rank-five representation for a six-dimensional predictive family and does not project private parameters as required by the risk bounds. The real-data experiments also use unprojected updates. Furthermore, the theoretical bounds appear too loose to be practically informative. For the maximum-norm method, using the theoretically sufficient clipping threshold results in substantially worse utility than the experimentally tuned threshold. It is therefore unclear how much the theoretical guarantees explain the observed improvements.
3. Limited novelty and missing comparisons.
The proposed regularizer is essentially a penalty on projected feature norms, motivated by their contribution to DP-SGD gradient norms. The paper should better distinguish this approach from existing public pretraining, feature normalization, and gradient-clipping methods. Comparisons with alternative public preprocessing strategies, direct gradient-norm regularization, or adaptive clipping would help establish the added value. The evaluation also focuses entirely on regression. A standard classification task such as CIFAR-10 would help demonstrate whether the approach generalizes to more practical DP learning settings.
Questions for authors
-
Can the authors evaluate the method on standard classification benchmarks? For example, CIFAR-10 with different model architectures and privacy budgets. How sensitive are the improvements to the representation dimension?
-
How does the method compare with simpler alternatives? In particular, public feature normalization, rank selection, direct gradient-norm regularization, or adaptive gradient clipping?
-
Can the authors better connect the theory with the experiments? Are the risk bounds meaningful for the actual experimental settings, especially without task-vector projection?
-
When is maximum-norm regularization useful? Can the authors provide a practical setting where Max clearly outperforms both AvgSq and the unregularized baseline?
3 comments
Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts in addressing my concerns.
However, I still have concerns about the limited practical improvements. While the new synthetic experiments show some benefits, simpler rank selection achieves similar results, and the improvements on real-world datasets remain small. I also remain concerned about the gap between the theoretical guarantees and the actual experiments, as well as the lack of classification experiments and comparisons with simpler alternatives.
Overall, I think the rebuttal partially addresses my concerns, but the main limitations remain. I will maintain my original rating and leave the final assessment to the AC.
Clarification on the Broader Contribution of Privacy Preconditioning
We thank the reviewer for their thoughtful follow-up and for acknowledging our efforts to address the concerns raised.
We would like to offer one final clarification regarding the broader contribution and motivation of our work, which we believe is important when assessing its significance.
Our primary goal is not simply to introduce a particular regularizer that consistently outperforms existing methods across all benchmarks. Rather, we aim to introduce and formally investigate privacy preconditioning as a general design principle: explicitly incorporating the requirements of downstream private learning into the objectives of an earlier, non-private learning stage.
The central distinction is that conventional public representation learning primarily seeks representations that transfer predictive information effectively, whereas privacy preconditioning asks how the public-learning objective itself should be designed to facilitate subsequent private optimization. This changes the objective of the upstream learning problem: downstream privacy constraints and their utility costs become explicit considerations before any private data are accessed. Our projected-feature regularizers provide concrete instantiations of this principle, while our theoretical analysis establishes formal connections between the public representation, downstream gradient geometry, and private prediction risk.
We fully recognize that the practical benefits of our particular instantiations are setting-dependent, as reflected in our experiments and discussion. However, the significance of the proposed framework extends beyond the performance of these specific regularizers on the particular regression benchmarks considered here. We believe that explicitly designing upstream learning objectives around downstream privacy requirements opens a broader direction for future research, potentially extending to other learning architectures, objectives, and private adaptation procedures.
We hope this perspective helps clarify the intended contribution and potential significance of the work. We appreciate the reviewer's careful consideration and constructive feedback throughout the discussion.
Thank you for the questions about alternatives, dimension, and the theory–experiment connection. Our revision addresses these through the following comparisons and clarifications.
1. Classification and sensitivity to representation dimension
Our evaluation concerns shared-representation regression, which allows exact synthetic population risks and a direct distinction between approximation and private adaptation error. Classification requires a separately validated training and calibration pipeline; we are not adding a classification claim in this revision.
For dimension sensitivity, Table 10 reports the full sweep over ranks one through six. At ε=2, AvgSq improves mean squared error by 0.0324, 0.0893, 0.1566, and 0.2286 at ranks three through six; ranks one and two do not improve. At ε=8, ranks four through six improve, rank three is inconclusive, and ranks one and two favor the baseline. Each method learns a representation for that rank and calibrates its private optimizer publicly. At rank six, all predictive directions are representable, so the benefit is not restricted to the original rank-five bottleneck.
2. What is different from normalization, rank selection, or clipping?
Normalization or whitening changes feature scale or covariance. Our public objective learns which predictive subspace to retain by balancing prediction loss against projected-feature norms. Row orthonormality prevents global shrinkage of U and compensating enlargement of the task vector; different subspaces can still have different projected norms.
Directly penalizing gradient norms would also penalize the task-dependent residual, since ∥∇vℓ∥2=2∣y−v⊤Ux∣∥Ux∥2. Our feature penalty is shared across tasks, while prediction loss retains the predictive objective. Adaptive clipping changes how the private optimizer chooses its threshold; we change its input representation before the private task is accessed. These are distinct, potentially complementary choices; we have not evaluated the three alternatives head-to-head.
Rank selection is evaluated explicitly. Both methods select rank two in this family's joint public search, and AvgSq provides no further improvement there (Table 9). At a fixed rank, however, changing the public objective still improves private prediction after each method independently tunes its threshold and learning rate: 10.5% lower MSE at rank five and ε=2. The experiments distinguish selecting a dimension from preparing a representation of that dimension.
3. Connecting the analysis to the observed gains
Equations (7)–(9) link predictive recovery to uniform transfer of projected moments. Theorem 1 quantifies the resulting approximation, clipping, estimation, and privacy-noise costs. Its clipping/noise balance improves with a smaller transferred moment, provided that saving outweighs the approximation cost; our reply to 69B9 gives the explicit equal-rank bound comparison.
The experiments test the corresponding trade-off rather than only a decrease in the penalty. In the primary synthetic comparison, AvgSq incurs about 0.029 more approximation error but 0.195 less private adaptation error at ε=2. Removing noise, and then clipping, shows how the comparison changes; matching thresholds and independently calibrating the optimizers preserve the gain. The improvement is in private adaptation rather than ordinary prediction.
Theorem 1 additionally constrains the private task vector to a norm ball. The main synthetic, School, and CT runs use the frozen features Ux but do not apply that task-vector projection. The explicit certificates are loose at the tested sample sizes (Appendix C.5); they explain the analytical trade-off rather than certify the measured MSE gaps.
4. What does Max add?
AvgSq penalizes an average; Max targets the largest projected norm across all public examples. Under the symmetric-hull coverage condition, that public maximum bounds every private training projected norm, simultaneously for all publicly learned representations. With bounded labels and task vectors, this gives the sufficient gradient threshold Ccert=2rU(Y+BvrU) and Theorem 2's expected private-risk bound, including the probability of coverage failure. Signed-anchor and directional task families in Appendix D.4 show concrete settings in which coverage holds.
The signed-coordinate experiment distinguishes these objectives: its population average-squared norm is constant across subspaces, while its maximum norm varies. Max reduces the latter, but the current private-MSE comparisons do not establish an advantage over both AvgSq and the baseline (Appendix D.5). Its theoretical contribution is the public-to-private radius guarantee. The sufficient threshold is not an empirical tuning prescription; its conservatism does not describe the separately tuned method.