Privacy-Preconditioned Representation Learning for Differentially Private Adaptation

We introduce privacy preconditioning: designing public training to improve the accuracy of a later learning stage subject to differential privacy. Rather than prepare a model only for predictive transfer, the public stage anticipates the clipping and noise required during private adaptation. We instantiate this principle in shared-representation learning: public tasks learn a low-dimensional representation, which is frozen while each new private task learns its own predictor. The public objective combines prediction loss with a penalty on either the average squared norm or the maximum norm of projected features. These norms influence the per-example gradients that private optimization clips and perturbs. For shared linear regression, we relate public prediction risk to representation recovery and bound the expected squared projected-feature norm on new tasks. Our private excess-risk bounds quantify the contributions of representation approximation, gradient clipping, and privacy noise. For the maximum penalty, a public-data coverage condition yields a bound on every private projected norm and an excess-risk guarantee using a sufficient clipping threshold. In synthetic regression tasks where the relationship between a predictive feature direction and the target varies across tasks, average-squared-norm regularization improves private mean squared error by 8.8% at a fixed privacy budget despite slightly worse non-private prediction. Repeating the experiment with fresh random seeds and newly generated tasks gives a similar improvement. Privacy preconditioning makes downstream private utility an explicit objective of public representation design, before any private records are accessed.