A Natively Differentially Private Foundation Model from Noisy Chebyshev Moments

Tabular foundation models such as TabPFN predict by in-context learning: the training rows of a table are placed in the model context, and one forward pass returns predictions for the test rows. The scheme is incompatible with differential privacy (DP), because every prediction is then a deterministic function of the sensitive rows. To address this gap, we present NoisyMomentPFN, a tabular foundation model designed for inputs that are already a differentially private object. The data holder publishes, once, the Chebyshev moments of the target along public random projections of the features, together with the Gram matrix of the columns, noised in a single Gaussian mechanism. A prior-fitted transformer is pretrained on millions of synthetic and public tables whose releases are noised on the fly, so that it learns a prior over noisy statistical moment releases; at test time it reads the DP moment release of a new table and returns a predictive distribution for any query. All queries are then post-processing, and incur no additional privacy budget. On a large suite of tabular datasets, NoisyMomentPFN beats the strongest one-shot baseline, sufficient statistics perturbation for ridge regression, at every privacy budget in aggregate, and at moderate budgets returns better predictive distributions, as measured by CRPS, than a neural network trained per table with DP-SGD, orders of magnitude faster.