acceptodds
Under review as a conference paper at ICLR 2027

Privacy-Preconditioned Representation Learning for Differentially Private Adaptation

Abstract

We introduce privacy preconditioning: designing public training to improve the accuracy of a later learning stage subject to differential privacy. Rather than prepare a model only for predictive transfer, the public stage anticipates the clipping and noise required during private adaptation. We instantiate this principle in shared-representation learning: public tasks learn a low-dimensional representation, which is frozen while each new private task learns its own predictor. The public objective combines prediction loss with a penalty on either the average squared norm or the maximum norm of projected features. These norms influence the per-example gradients that private optimization clips and perturbs. For shared linear regression, we relate public prediction risk to representation recovery and bound the expected squared projected-feature norm on new tasks. Our private excess-risk bounds quantify the contributions of representation approximation, gradient clipping, and privacy noise. For the maximum penalty, a public-data coverage condition yields a bound on every private projected norm and an excess-risk guarantee using a sufficient clipping threshold. In synthetic regression tasks where the relationship between a predictive feature direction and the target varies across tasks, average-squared-norm regularization improves private mean squared error by 8.8% at a fixed privacy budget despite slightly worse non-private prediction. Repeating the experiment with fresh random seeds and newly generated tasks gives a similar improvement. Privacy preconditioning makes downstream private utility an explicit objective of public representation design, before any private records are accessed.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Differential Privacy, Representation Learning, Private Adaptation, DP-SGD, Multi-Task Learning, Private Transfer Learning, Privacy-Preserving Machine Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_z3P4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes privacy-preconditioned representation learning, where public training explicitly prepares a representation for later differentially private adaptation. The authors introduce two regularization methods that penalize the average or maximum norm of projected features to reduce the effects of gradient clipping and DP noise. The paper provides theoretical excess-risk bounds for linear regression and evaluates the methods on synthetic and real-world regression tasks.

Critical strengths
  1. Designing public representations specifically for downstream DP training is a reasonable idea. The analysis of how feature norms affect gradient clipping and privacy noise provides useful insights.
  2. The authors include non-private comparisons, clipping/noise ablations, matched-threshold experiments, and independent repetitions. These help explain why the proposed regularization improves private utility in the synthetic setting.
Critical weaknesses

1. The experimental improvements are limited.

The strongest result is an 8.8% improvement on a specially constructed synthetic regression problem. However, the real-world results are much less convincing. On ILEA School, the regularized method initially achieves a 10–13% improvement, but a simple rank-one baseline matches or outperforms it. This suggests that the benefit mainly comes from reducing effective dimension rather than the proposed privacy-aware regularization. On CT Slice Localization, improvements are only 0.08–0.97%. The maximum-norm method also fails to demonstrate a statistically clear improvement.

Overall, I do not think the experiments establish a consistent practical advantage.

2. The theory is not well connected to the main experiments.

The theoretical guarantees rely on bounded linear models, projected task parameters, and, for the maximum-norm method, a strong public-data coverage assumption. However, the main positive synthetic experiment uses a rank-five representation for a six-dimensional predictive family and does not project private parameters as required by the risk bounds. The real-data experiments also use unprojected updates. Furthermore, the theoretical bounds appear too loose to be practically informative. For the maximum-norm method, using the theoretically sufficient clipping threshold results in substantially worse utility than the experimentally tuned threshold. It is therefore unclear how much the theoretical guarantees explain the observed improvements.

3. Limited novelty and missing comparisons.

The proposed regularizer is essentially a penalty on projected feature norms, motivated by their contribution to DP-SGD gradient norms. The paper should better distinguish this approach from existing public pretraining, feature normalization, and gradient-clipping methods. Comparisons with alternative public preprocessing strategies, direct gradient-norm regularization, or adaptive clipping would help establish the added value. The evaluation also focuses entirely on regression. A standard classification task such as CIFAR-10 would help demonstrate whether the approach generalizes to more practical DP learning settings.

Questions for authors
  1. Can the authors evaluate the method on standard classification benchmarks? For example, CIFAR-10 with different model architectures and privacy budgets. How sensitive are the improvements to the representation dimension?

  2. How does the method compare with simpler alternatives? In particular, public feature normalization, rank selection, direct gradient-norm regularization, or adaptive gradient clipping?

  3. Can the authors better connect the theory with the experiments? Are the risk bounds meaningful for the actual experimental settings, especially without task-vector projection?

  4. When is maximum-norm regularization useful? Can the authors provide a practical setting where Max clearly outperforms both AvgSq and the unregularized baseline?

3 comments

Thank you for the questions about alternatives, dimension, and the theory–experiment connection. Our revision addresses these through the following comparisons and clarifications.

1. Classification and sensitivity to representation dimension

Our evaluation concerns shared-representation regression, which allows exact synthetic population risks and a direct distinction between approximation and private adaptation error. Classification requires a separately validated training and calibration pipeline; we are not adding a classification claim in this revision.

For dimension sensitivity, Table 10 reports the full sweep over ranks one through six. At ε=2\varepsilon=2, AvgSq improves mean squared error by 0.0324, 0.0893, 0.1566, and 0.2286 at ranks three through six; ranks one and two do not improve. At ε=8\varepsilon=8, ranks four through six improve, rank three is inconclusive, and ranks one and two favor the baseline. Each method learns a representation for that rank and calibrates its private optimizer publicly. At rank six, all predictive directions are representable, so the benefit is not restricted to the original rank-five bottleneck.

2. What is different from normalization, rank selection, or clipping?

Normalization or whitening changes feature scale or covariance. Our public objective learns which predictive subspace to retain by balancing prediction loss against projected-feature norms. Row orthonormality prevents global shrinkage of UU and compensating enlargement of the task vector; different subspaces can still have different projected norms.

Directly penalizing gradient norms would also penalize the task-dependent residual, since ∥∇vℓ∥2=2∣y−v⊤Ux∣∥Ux∥2\|\nabla_v\ell\|_2=2|y-v^\top Ux|\|Ux\|_2. Our feature penalty is shared across tasks, while prediction loss retains the predictive objective. Adaptive clipping changes how the private optimizer chooses its threshold; we change its input representation before the private task is accessed. These are distinct, potentially complementary choices; we have not evaluated the three alternatives head-to-head.

Rank selection is evaluated explicitly. Both methods select rank two in this family's joint public search, and AvgSq provides no further improvement there (Table 9). At a fixed rank, however, changing the public objective still improves private prediction after each method independently tunes its threshold and learning rate: 10.5% lower MSE at rank five and ε=2\varepsilon=2. The experiments distinguish selecting a dimension from preparing a representation of that dimension.

3. Connecting the analysis to the observed gains

Equations (7)–(9) link predictive recovery to uniform transfer of projected moments. Theorem 1 quantifies the resulting approximation, clipping, estimation, and privacy-noise costs. Its clipping/noise balance improves with a smaller transferred moment, provided that saving outweighs the approximation cost; our reply to 69B9 gives the explicit equal-rank bound comparison.

The experiments test the corresponding trade-off rather than only a decrease in the penalty. In the primary synthetic comparison, AvgSq incurs about 0.029 more approximation error but 0.195 less private adaptation error at ε=2\varepsilon=2. Removing noise, and then clipping, shows how the comparison changes; matching thresholds and independently calibrating the optimizers preserve the gain. The improvement is in private adaptation rather than ordinary prediction.

Theorem 1 additionally constrains the private task vector to a norm ball. The main synthetic, School, and CT runs use the frozen features UxUx but do not apply that task-vector projection. The explicit certificates are loose at the tested sample sizes (Appendix C.5); they explain the analytical trade-off rather than certify the measured MSE gaps.

4. What does Max add?

AvgSq penalizes an average; Max targets the largest projected norm across all public examples. Under the symmetric-hull coverage condition, that public maximum bounds every private training projected norm, simultaneously for all publicly learned representations. With bounded labels and task vectors, this gives the sufficient gradient threshold Ccert=2rU(Y+BvrU)C_{\rm cert}=2r_U(Y+B_vr_U) and Theorem 2's expected private-risk bound, including the probability of coverage failure. Signed-anchor and directional task families in Appendix D.4 show concrete settings in which coverage holds.

The signed-coordinate experiment distinguishes these objectives: its population average-squared norm is constant across subspaces, while its maximum norm varies. Max reduces the latter, but the current private-MSE comparisons do not establish an advantage over both AvgSq and the baseline (Appendix D.5). Its theoretical contribution is the public-to-private radius guarantee. The sufficient threshold is not an empirical tuning prescription; its conservatism does not describe the separately tuned method.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts in addressing my concerns.

However, I still have concerns about the limited practical improvements. While the new synthetic experiments show some benefits, simpler rank selection achieves similar results, and the improvements on real-world datasets remain small. I also remain concerned about the gap between the theoretical guarantees and the actual experiments, as well as the lack of classification experiments and comparisons with simpler alternatives.

Overall, I think the rebuttal partially addresses my concerns, but the main limitations remain. I will maintain my original rating and leave the final assessment to the AC.

AuthorsOpenReview ↗

Clarification on the Broader Contribution of Privacy Preconditioning

We thank the reviewer for their thoughtful follow-up and for acknowledging our efforts to address the concerns raised.

We would like to offer one final clarification regarding the broader contribution and motivation of our work, which we believe is important when assessing its significance.

Our primary goal is not simply to introduce a particular regularizer that consistently outperforms existing methods across all benchmarks. Rather, we aim to introduce and formally investigate privacy preconditioning as a general design principle: explicitly incorporating the requirements of downstream private learning into the objectives of an earlier, non-private learning stage.

The central distinction is that conventional public representation learning primarily seeks representations that transfer predictive information effectively, whereas privacy preconditioning asks how the public-learning objective itself should be designed to facilitate subsequent private optimization. This changes the objective of the upstream learning problem: downstream privacy constraints and their utility costs become explicit considerations before any private data are accessed. Our projected-feature regularizers provide concrete instantiations of this principle, while our theoretical analysis establishes formal connections between the public representation, downstream gradient geometry, and private prediction risk.

We fully recognize that the practical benefits of our particular instantiations are setting-dependent, as reflected in our experiments and discussion. However, the significance of the proposed framework extends beyond the performance of these specific regularizers on the particular regression benchmarks considered here. We believe that explicitly designing upstream learning objectives around downstream privacy requirements opens a broader direction for future research, potentially extending to other learning architectures, objectives, and private adaptation procedures.

We hope this perspective helps clarify the intended contribution and potential significance of the work. We appreciate the reviewer's careful consideration and constructive feedback throughout the discussion.

Reviewer_69B9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies a multitask learning setting where a shared representation is learned from public data and then used for private adaptation on new tasks. The authors propose modifying the public representation-learning objective to penalize the norm of the projected features, motivated by the observation that smaller feature norms can reduce gradient clipping and noise costs during downstream DP-SGD. They derive excess-risk guarantees for shared linear regression that account for representation approximation error, clipping bias, and privacy noise. Experiments on synthetic and real datasets show some improvements in private prediction accuracy over unregularized representation learning.

Critical strengths

The paper considers a reasonable question: whether the best representation for non-private prediction is necessarily the best one for subsequent private adaptation, and whether regularaization can help in the private setting.

The synthetic experiments provide an example where sacrificing a small amount of non-private prediction accuracy improves private utility.

Critical weaknesses

1. Limited novelty and theoretical contribution.

Algorithm: The proposed method simply adds a regularizer penalizing projected feature norms to an otherwise standard public representation-learning objective. The motivation follows directly from the basic observation that the norm of the per-example squared-loss gradient is proportional to the projected feature norm. While sensible, the proposed modification seems straightforward and not very interesting.

Results: Crucially, the results do not establish that the proposed regularization actually improves private excess risk relative to standard representation learning. It is unclear what substantial new theoretical insight is gained.

Analysis: The theoretical analysis is also limited. It combines standard shared-representation recovery arguments with bounds on gradient clipping bias and Gaussian noise under restrictive linear model assumptions. For the maximum-norm penalty, the analysis also requires a strong public-data coverage condition.

2. Unconvincing empirical evidence.

The strongest positive result (an 8.8% improvement) comes from a specially constructed synthetic regression problem. The real-data results are considerably less convincing.

Moreover, the main positive synthetic experiment falls outside the realizable rank-k shared representation assumption underlying the theory.

Overall, the experiments do not convincingly establish that privacy preconditioning provides a meaningful benefit beyond existing public-representation methods and simpler approaches to selecting representations for private adaptation.

  1. The discussion of public-data-assisted private learning could also be more complete. In particular, Lowy et al., Optimal Differentially Private Model Training with Public Data (ICML 2024) was not cited. Although that work considers a different setting from the multitask representation transfer studied here, it provides relevant context for understanding when and how public information can improve private learning.
Questions for authors
  1. Can the authors identify a setting in which their theoretical bounds provably establish a meaningful advantage over ordinary public representation learning with the same representation dimension?

  2. Can the authors demonstrate meaningful improvements on real datasets that cannot be explained by simply reducing the effective representation dimension?

1 comment

Thank you for asking for a sharper account of the theoretical advantage at equal dimension and of the real-data evidence. The relevant comparison is between the upper bounds for two public representation choices. We make that comparison explicit below.

1. When does the private-risk upper bound improve at the same rank?

Let U0U_0 be the unregularized representation and UλU_\lambda the regularized one, both of rank kk. Under Theorem 1's assumptions and public recovery/transfer events, fix common n,b,T,η,σn,b,T,\eta,\sigma and distributional constants. Write ri=rpub(Ui)r_i=r_{\rm pub}(U_i) and Ai=βB⋆2Bsub,i/2A_i=\beta B_\star^2 B_{{\rm sub},i}/2 for each representation's approximation certificate. Define

K=βη2ST,c1=2aBr4X2(1+1/n),c2=16a(Bv2X2+Y2)/n,c3=νn,b+kσ2/b2.\begin{aligned}K&=\beta\eta^2S_T,& c_1&=2aB_r^4X^2(1+1/n),\\ c_2&=16a(B_v^2X^2+Y^2)/n,& c_3&=\nu_{n,b}+k\sigma^2/b^2.\end{aligned}

Equation (11) then has right-hand side

Bi(C)=Ai+4βBv2qT+K(c1ri/C2+c2ri+c3C2).\mathcal B_i(C)=A_i+4\beta B_v^2q^T+K\bigl(c_1r_i/C^2+c_2r_i+c_3C^2\bigr).

Balancing the clipping and noise terms gives the public choice Ci=(c1ri/c3)1/4C_i=(c_1r_i/c_3)^{1/4}, exactly the calculation in Appendix B.6. The minimized bound is

Bi∗=Ai+4βBv2qT+K(2c1c3ri+c2ri).\mathcal B_i^*=A_i+4\beta B_v^2q^T+K\bigl(2\sqrt{c_1c_3r_i}+c_2r_i\bigr).

Consequently the regularized representation has a strictly smaller bound whenever

Aλ−A0<K[2c1c3(r0−rλ)+c2(r0−rλ)].A_\lambda-A_0<K\bigl[2\sqrt{c_1c_3}(\sqrt{r_0}-\sqrt{r_\lambda})+c_2(r_0-r_\lambda)\bigr].

This identifies a concrete regime: a reduction in the transferred projected moment outweighs the extra approximation cost. For example, if rλ=r0/4r_\lambda=r_0/4, the balanced clipping/noise contribution is halved and the finite-sample contribution is quartered, at unchanged dimension and privacy budget. The gain remains in the total bound provided the approximation increase is smaller than these savings. The required moment separation is a statement about the transferred quantities, not merely a lower training penalty; when both transfer bounds saturate at X2X^2, this particular comparison gives no gain.

Thus the analysis does more than relate gradient norms to features: it quantifies how much predictive approximation can be traded for a better private-risk guarantee and how the clipping threshold should change. This is a conditional comparison of the representations produced by public training, not an assertion that every positive penalty satisfies the inequality. The numerical certificates in Appendix C.5 are loose at that experiment's sample sizes; we do not use them to certify the observed MSE differences.

The closest transfer result, Thaker et al. (2024), analyzes public subspace estimation and private regression with isotropic Gaussian features. In that model Tr⁡(UIdU⊤)=k\operatorname{Tr}(U I_d U^\top)=k for every row-orthonormal UU: the population AvgSq quantity cannot distinguish equal-rank subspaces. Our bounds retain the varying feature geometry and clipping trade-off in a bounded, potentially anisotropic model. The advantage above is representation-dependent; it is not a claim to improve the optimal dimension/sample-size rates in their different model.

2. Real data and the equal-dimension evidence

On ILEA School, the penalty selects a representation with lower effective dimension and improves normalized MSE by 10.9%–13.3% over the nominally rank-five baseline at ε=1,2,4,8\varepsilon=1,2,4,8. The submitted rank-one comparison reproduces that gain. CT gives smaller within-rank differences of 0.08%–0.97%, but does not isolate an advantage over effective-rank selection. We therefore use these benchmarks to characterize the benefit, not claim a large real-data gain beyond dimension choice.

The synthetic comparison directly examines the objective at equal dimension. After independent optimizer calibration, rank-five MSE improves from 1.8416 to 1.6477 at ε=2\varepsilon=2. At rank six, sufficient to represent the six-direction coefficient family, the MSE reductions are 0.2286 and 0.0500 at ε=2,8\varepsilon=2,8, with both 95% intervals excluding zero (Tables 9–10). At rank six and ε=2\varepsilon=2, approximation error rises from 0.1044 to 0.1381 while adaptation error falls from 0.8272 to 0.5649. This isolates the central trade-off much more clearly than attributing the improvement simply to a missing predictive direction. These runs use the learned features but not Theorem 1's additional projection of the private task vector into a norm ball.

Max adds a distinct theoretical route: coverage converts the public maximum into a uniform bound on private training gradients and an expected excess-risk guarantee without strong convexity. Appendix D.4 gives signed-anchor and directional distributions satisfying coverage. Section 2 also now discusses Lowy et al. (2024) and distinguishes public/private gradient weighting from learning a reusable shared representation for new private tasks.

Reviewer_g15nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper trains a shared linear representation on public data with an added penalty on projected feature norms, and then train on private dataset by freezing this representation and only tuning the linear head with DP-SGD. The penalty term is meant to reduce the cost of clipping and noise in that stage. The paper selects the penalty weight and clipping threshold on running private optimizations on held-out public datasets. The paper bounds subspace recovery and private excess risk for the shared linear model. On synthetic regression, the AvgSq variant of this method reduces private MSE by around 8% at ε=2\varepsilon=2 in a task-instability regime. On ILEA School, it shows the gain via a rank-one unregularized baseline. Overall, the paper presents a method to improve privacy-preserving algorithms given some assumptions.

Critical strengths
  1. A well-posed question supported by one clear demonstration. The paper investigates which public representation is most suitable for subsequent private adaptation, rather than simply optimizing non-private accuracy, and Section 6.1 provides a case where these objectives diverge. The regularized representation performs slightly worse without clipping or noise but better under differential privacy. This conclusion is further supported by the clipping/noise decomposition and matched-threshold comparison. 2. Careful and candid evaluation. All decisions are based on public data, the comparisons use paired evaluations, and every result is validated on an independent set of seeds. The paper also includes findings that temper its own conclusions.
Critical weaknesses

W1 (Limited contribution beyond standard public pretraining with private linear probing): The overall setup is standard: learn a representation on public data, freeze it, and privately train a linear head. Public hyperparameter tuning without privacy cost is also well established. The main new component is the regularizer, but its effect is not clearly separated from simpler factors. In the synthetic experiment, using the same private step size means that reducing projected variance also changes the effective optimization scale. On ILEA, a lower-rank unregularized representation achieves a similar gain, suggesting that dimension may explain much of the effect. The paper also misses several closely related works on using public information to shape private optimization.

W2 (The claimed insights do not match the results): The paper presents privacy preconditioning as a general design principle, but the evidence is much narrower. The main gain appears in a selected misspecified synthetic setting, while nearby settings show little benefit or even worse performance. Under the realizable model used in the theory, regularization hurts and public selection favors no regularization. On real data, the benefit of the penalty itself is also unclear. On ILEA, dimension seems to explain the gain, while CT shows only small improvements without paired uncertainty estimates. The theoretical bounds also fail to meaningfully separate the representations in these experiments. Overall, the results show that the penalty can help in some settings, but not that it is a reliable general principle.

Minor.

  • The public and private tasks are assumed to be exchangeable, so robustness to distribution shift is not tested. ILEA also violates the feature-excitation assumption.
  • One setting described as showing no benefit actually shows a clear loss. A stricter privacy setting where the method underperforms a simple baseline is also omitted from the main table.
  • CT privacy is defined per slice rather than per patient. The paper should also clarify the sensitivity used by the privacy accountant for ILEA.
  • There are a few minor citation and grammar errors.
Questions for authors
  1. If the private step size is tuned publicly for both λ=0\lambda=0 and AvgSq in the task-instability experiment, how much of the MSE gain at ε=2\varepsilon=2 remains?
  2. Does an unregularized representation with a publicly selected rank kk match AvgSq in the synthetic study, as the rank-one baseline does on ILEA?
3 comments

Thank you for recognizing the central phenomenon: a representation that is worse without clipping or noise can nevertheless support better private prediction. We address your optimizer-tuning and rank-selection questions through separate comparisons, keeping their different purposes explicit.

1. Does the gain survive publicly tuning the private optimizer?

It does. In the optimizer-calibration experiment, both methods retain rank five and independently select the clipping threshold CC and learning rate η\eta using the same public calibration tasks and candidate grids. AvgSq additionally selects its penalty weight, with λ=0\lambda=0 included. At ε=2\varepsilon=2, mean squared error falls from 1.8416 to 1.6477: a 10.5% reduction, favorable in all 24 independent task collections. The regularized-minus-unregularized difference is -0.1940, with 95% paired bootstrap interval [-0.2253, -0.1630]. At ε=8\varepsilon=8, the difference is -0.0357 [-0.0456, -0.0260] (Table 9).

This answers the optimization-scale concern directly: the improvement remains when the unregularized baseline gets its own publicly selected learning rate and threshold. Comparisons using a common (C,η)(C,\eta) also retain most of the gain. The ablations help interpret it: in that experiment at ε=2\varepsilon=2, introducing clipping increases MSE by 0.177 for AvgSq versus 0.262 for the baseline, and subsequently adding noise costs 0.245 versus 0.370. The representation approximation error is slightly larger, 0.146 versus 0.107. Thus the favorable private result is accompanied by lower adaptation costs, not better underlying predictive approximation. Appendix C.2.4 reports the full tuning procedure and its grid boundaries.

2. What happens when the unregularized method can select its rank?

In this synthetic family, public rank selection finds a rank-two unregularized representation that matches or improves on AvgSq. The distinction is between choosing the dimension and choosing the representation within a given dimension. We evaluated both questions explicitly.

Allowing only the baseline to select rank, while keeping AvgSq at rank five, gives MSE 1.3284 versus 1.6048 at ε=2\varepsilon=2. In the symmetric comparison, both methods select rank, threshold, and learning rate, and AvgSq also selects λ\lambda. Both choose rank two in all 24 collections; MSE is 1.3268 versus 1.3297 at ε=2\varepsilon=2 and 1.2037 versus 1.2057 at ε=8\varepsilon=8 (baseline versus AvgSq). The regularized search therefore provides no additional improvement under this selection procedure. Per-candidate calibration effort is matched; the regularized family searches more candidates (Table 9).

This result identifies an effective alternative when predictive signal is concentrated in two directions. The equal-rank experiment answers a different question: whether changing the objective improves the representation once its dimension is fixed. More importantly, Table 10 now shows the same phenomenon at rank six, where the representation is large enough to contain the entire predictive coefficient family: MSE differences favor AvgSq by 0.2286 at ε=2\varepsilon=2 and 0.0500 at ε=8\varepsilon=8. Hence the evidence is not limited to the original rank-five-versus-six mismatch. The two findings together tell us when changing the public objective helps at a fixed dimension and when selecting a smaller dimension is already sufficient.

3. Scope, benchmarks, and technical clarifications

Our design principle is to account for downstream private estimation during public learning; it does not require a positive penalty to be optimal for every distribution. The bounded-anisotropic experiment selects λ=0\lambda=0, and the fixed-coefficient comparison has a small MSE increase at ε=8\varepsilon=8, now stated explicitly (Table 6; Appendix C.5). Both inform the trade-off, rather than negate the demonstrated reversal between non-private and private prediction.

On ILEA School, the rank-one comparison already in the submission attributes the gain primarily to lower effective dimension. The unfavorable ε=0.5\varepsilon=0.5 outcome remains explicit in Section 6.2 and Table 12. For CT, Table 18 supplies paired intervals over public runs and patient-level summaries; the latter include zero at ε=5,8\varepsilon=5,8. The CT threshold comparison is retrospective, and the stated privacy guarantee covers each fixed-threshold fit and each training slice, not selection using private evaluation outcomes or patient-level privacy.

School's clipped average has replace-one sensitivity 2C/B2C/B and Gaussian noise standard deviation σC/B\sigma C/B; Appendix C.3 identifies the Birrell accountant and its input convention. Section 7 distinguishes the theoretical task-vector projection from the empirical updates, and Section 2 expands the relevant public-data comparisons. Our transfer analysis uses a common task distribution; distribution shift remains a separate extension.

Thank you for the thorough and candid response. The optimizer-calibration experiment resolves my step-size concern (Q1). However, the symmetric comparison shows that a publicly rank-selected unregularized baseline matches AvgSq, which, together with the ILEA rank-one result, confirms that dimension selection explains most of the gain and leaves the regularizer's contribution and the "general design principle" claim unsupported, so I maintain my score.

Thank you for confirming that the optimizer-calibration experiment resolves Q1. We would like to clarify the conclusion about the regularizer’s contribution. The symmetric rank-selection comparison establishes that selecting rank two publicly is sufficient to match the regularized search in this synthetic family; our revision reports that result. However, this does not negate the improvements obtained by changing the public objective at a fixed rank. Choosing the representation dimension and improving the representation learned at that dimension are distinct questions, and our experiments address both.

The equal-rank evidence is substantive. At rank five and privacy budget ε=2\varepsilon=2, independently selecting each method’s clipping threshold and learning rate on public calibration tasks reduces private MSE from 1.8416 to 1.6477, a 10.5% improvement, favorable in all 24 task collections (Table 9). The gain also extends beyond the originally chosen rank: Table 10 reports improvements at ranks three through six for ε=2\varepsilon=2 and ranks four through six for ε=8\varepsilon=8. At rank six, the MSE reductions are 0.2286 and 0.0500, with 95% intervals [0.1888, 0.2775] and [0.0364, 0.0637], respectively. Each comparison uses the same rank for both methods and independently calibrates their private optimizers. Rank six is sufficient in principle to represent all six predictive directions, so the phenomenon is not confined to the original rank-five bottleneck.

The rank-six decomposition also shows why this matters for representation learning. At ε=2\varepsilon=2, regularization increases representation approximation error from 0.1044 to 0.1381, but reduces private adaptation error from 0.8272 to 0.5649 (Appendix C.2.4). The reduction in adaptation error more than compensates for the loss of predictive fidelity, yielding the 0.2286 net improvement. The clipping/noise ablations in Section 6.1 and Appendix C.2.4 further show smaller accuracy costs from both operations. This is direct evidence for the motivation of our objective: the representation that preserves predictive information better can be less suitable for learning under privacy constraints.

The public rank-selection experiment addresses a complementary practical question: can selecting a smaller model already attain comparable utility on these tasks? Here, it can. The synthetic family’s predictive signal is strongly concentrated in two directions, and both methods select rank two. The ILEA comparison likewise shows that lower effective dimension accounts for that benchmark’s gain. These findings identify situations where rank reduction suffices. The equal-rank comparisons establish an additional finding: altering the public objective improves private prediction with the number of representation coordinates and private parameters held fixed. We describe those improvements as equal-rank gains, consistently with what the experiments establish.

Our design principle has a concrete implementation and analytical basis: retain public prediction loss while adding projected-feature penalties that anticipate the clipping and noise of subsequent private adaptation. Theorem 1 and Appendix B.6 quantify the trade-off at a fixed dimension: reducing the transferred projected-feature moment can improve the private-risk upper bound when the resulting savings outweigh any increase in representation approximation cost. The experiments illustrate the corresponding trade-off in achieved prediction accuracy. This makes downstream private utility an explicit criterion for public representation training, with the penalty strength selected using public data. A useful design principle need not prescribe a positive penalty in every setting.

We therefore believe that the evidence supports a more specific conclusion than the regularizer’s contribution being unsupported: rank selection suffices in the reported joint-selection and ILEA comparisons, while privacy-aware public representation training produces reproducible gains in the reported equal-rank comparisons. We respectfully ask that the contribution be assessed using both findings, together with the analysis explaining the underlying trade-off.

Comments on the paper

1 comment

Thank you for the thoughtful reviews. Using a public representation for private learning is not the same as learning it specifically for its later private use. We have revised the manuscript to explain this distinction and added comparisons addressing optimizer tuning and representation dimension. Changes are highlighted in blue.

What privacy preconditioning changes

Ordinary public pretraining asks which features best preserve predictive information. After those features are frozen, private adaptation must work with the gradient magnitudes and prediction sensitivity they induce. Our public objective anticipates that second stage: it learns a predictive subspace while also penalizing a property of the features that enters the accuracy cost of clipping and privacy noise. The aim is not merely a good representation followed by a private optimizer, but a representation chosen to serve that optimizer.

Concretely, we add either the average squared projected-feature norm (AvgSq) or the maximum projected norm (Max) to public prediction loss. For squared loss, the task-gradient norm is 2∣y−v⊤Ux∣∥Ux∥22|y-v^\top Ux|\|Ux\|_2. Penalizing the feature factor while retaining prediction loss creates an explicit trade-off between preserving signal and making that signal easier to learn privately. The constraint UU⊤=IkUU^\top=I_k prevents simply shrinking features and compensating with larger coefficients: the objective changes which subspace is learned. We then freeze UU and use the same clipped, noisy private adaptation procedure and accounting. All representation learning precedes access to the new private task.

This distinction matters even when both methods train exactly the same number of private parameters. A representation can be slightly less accurate without clipping or noise yet more accurate after privatization. Our synthetic experiments demonstrate precisely this reversal, rather than only the familiar benefit of replacing an ambient-dimensional private model by a smaller one.

How this differs from the closest public-data methods

Thaker et al. (2024) are closest in setup: public tasks estimate a shared predictive subspace, and a new task learns privately within it. Their analysis explains the benefit of transferring low-dimensional structure, including under concept shift. We ask a further question within that pipeline: should the public stage favor a different subspace because its eventual use involves clipping and noise? Our explicit penalties make that downstream consideration part of representation training, not just its evaluation.

Amid et al. (2022) instead use public loss to define the mirror map governing private optimization. Lowy et al. (2024) study optimal learning with public and private samples and algorithms that combine their gradients. Jiang et al. (2026) build a conditioning matrix from public features of the training records under label or semi-sensitive-feature privacy. Our construction differs in what is learned and when: a shared, rank-reducing representation is trained across public tasks with a predictive and privacy-motivated objective, then reused for new-task adaptation that protects both features and labels. It does not require the new task's features to be public or replace the private optimizer with a public-data-dependent mirror-descent procedure.

What the analysis and new evidence add

The theory follows the consequences of this public choice through recovery of predictive structure, transfer of projected-feature moments to new tasks, and private excess risk. It retains the approximation cost of changing the subspace alongside the reduction in clipping-related and noise-related costs. Max provides a complementary guarantee: public coverage makes the public maximum a bound on every private training projected norm, yielding a sufficient gradient threshold and an expected private-risk bound (Theorems 1–2).

The empirical question is whether the gain survives fair private-optimizer tuning and a larger representation. At rank five and privacy budget ε=2\varepsilon=2, independent public selection of each method's clipping threshold and learning rate reduces mean squared error from 1.8416 to 1.6477, a 10.5% improvement across 24 fresh task collections (Table 9). The original 8.8% result repeats on fresh seeds. At rank six, enough to represent all six predictive directions, AvgSq again improves private prediction (Table 10). The effect is therefore neither confined to one common untuned learning rate nor to the rank-five representational bottleneck. We also report public rank selection separately, where both methods choose rank two and the penalty provides no further gain in this family.

Privacy preconditioning makes downstream private utility a design criterion for public learning, rather than a consequence assessed only after the representation is frozen. The individual replies address the bound comparison, rank selection, and real-data evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.