acceptodds
Under review as a conference paper at ICLR 2027

A Natively Differentially Private Foundation Model from Noisy Chebyshev Moments

Abstract

Tabular foundation models such as TabPFN predict by in-context learning: the training rows of a table are placed in the model context, and one forward pass returns predictions for the test rows. The scheme is incompatible with differential privacy (DP), because every prediction is then a deterministic function of the sensitive rows. To address this gap, we present NoisyMomentPFN, a tabular foundation model designed for inputs that are already a differentially private object. The data holder publishes, once, the Chebyshev moments of the target along public random projections of the features, together with the Gram matrix of the columns, noised in a single Gaussian mechanism. A prior-fitted transformer is pretrained on millions of synthetic and public tables whose releases are noised on the fly, so that it learns a prior over noisy statistical moment releases; at test time it reads the DP moment release of a new table and returns a predictive distribution for any query. All queries are then post-processing, and incur no additional privacy budget. On a large suite of tabular datasets, NoisyMomentPFN beats the strongest one-shot baseline, sufficient statistics perturbation for ridge regression, at every privacy budget in aggregate, and at moderate budgets returns better predictive distributions, as measured by CRPS, than a neural network trained per table with DP-SGD, orders of magnitude faster.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Differential Privacy, Foundation Models, TabPFN, Prior Fitted Network, Chebyshev, Moments, High Dimensional Stats

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_FjTxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes NoisyMomentPFN, a tabular foundation model that allows for an unlimited number of queries with differentially private results at much smaller cost than ad-hoc trained DP-SGD models. The model is trained in two macro-phases, first on clean data (producing MomentPFN) and then on noised data. Instead of tabular rows, the model is trained and consumes a fixed-size private release. This is composed of the Chebyshev moments of the target along public random feature projections and the Gram matrix of the augmented columns. The model is evaluated against other differentially private baselines. It achieves competitive utility, worse in RMSE than a DP-SGD MLP but comparable predictive distributions, at much smaller compute cost, as it does not require re-training and instead only needs the single release computation.

Critical strengths
  • Effective and cheap: the model generalizes well and performs on par with ad-hoc DP-SGD. The compute cost, excluding the training of the backbone, is negligible.
  • Broad evaluation: 53 regression and 33 classification tables tested over multiple privacy budgets. Utility is reported with multiple metrics together with cost. The result analysis is also transparent in the limitations.
Critical weaknesses
  • Novelty: the paper is a combination of known techniques in DP. The main novel component, the two-phase (clean-then-noisy) pretraining, is not directly ablated. appendix G.6 reports that the training curriculum with 30% clean releases mixed in, as opposed to the sequential one used, performs better on large tables at a small cost on small ones. Consequently the central claim is not well isolated.
  • Clarity: the paper is quite hard to parse, and the notation is particularly hard to follow: P/p (distributions, parameters, prior), S/s/s_abc (dataset, scale, moment weight), and L (both the number of pairs and the loss) are all overloaded. Some interesting results and caveats are in the appendix, which makes reading the main body hard and obscures both the limitations and the paper's most informative analyses. Some typos and inconsistencies are present ("he release" -> "The release"; "filled back in.." double period). Section 4 says the numeric and mixed strata hold tables of at most 10k rows, but Table 1 shows much larger ones (e.g., sarcos, 48933 rows). Together, these issues make the paper's claims harder to verify from the main text alone.
Questions for authors
  • Can you provide a direct ablation of: MomentPFN on noisy releases; NoisyMomentPFN trained from scratch; the mixed-releases curriculum; and the checkpoint based on MomentPFN?
  • Can you demonstrate the model's robustness (or lack thereof) to a mismatch between the epsilon seen during training and during inference?
  • s=1 and w=5 are never ablated, and with these values the Gram term dominates the sensitivity while s=1 clips about a tenth of the projections. Can you show how much the results depend on them?
5 comments

Thank you so much for your time and for your detailed review! We address each of your concerns below. Our updates are also in the revised PDF (changes in blue).

W1: questioning novelty, also "two-phase pretraining is not directly ablated."

We appreciate this call for additional ablations, and for clarity on the novelty! We acknowledge that the individual ingredients (the Gaussian mechanism, Chebyshev moments, PFNs, etc.) are all well established, but our contribution is indeed the paradigm itself. To the best of our knowledge, ours is the first natively DP foundation model that reads DP statistics (computed just once from a table) as context, and generates all subsequent predictions privately via post-processing.

Moreover, beyond the paradigm itself, the choice of statistics is not merely an engineering trick. For example, standard DP statistics such as marginals have schema-dependent layouts that are not directly suitable for foundation models. Directly privatizing individual rows destroys much of the signal, as each row's sensitivity is the row itself, while generating DP synthetic data introduces an expensive per-table step. We instead designed a fixed-size release, sliced Chebyshev moments over pairs of random projections plus a Gram block, taking inspiration from prior work on unsliced Chebyshev moment matching (Musco et al. 2025). The benefits are threefold: 1. the size of the vector of statistics does not depend on the number of rows in the data, 2. its sensitivity grows only polylogarithmically in the degree, and 3. it can be privatized via a simple Gaussian mechanism. Importantly, beyond these desirable theoretical properties, our ablations demonstrate that this choice is also empirically effective: every alternative release we pretrained in Section G.4 (Gram alone, Chebyshev alone, monomial moments with or without the Gram block) is worse, and monomial moments are far worse.

Together, our paradigm and choice of statistics make the method extremely efficient: it requires neither per-table DP training nor DP synthesis, and its input size is independent of the number of rows. In terms of utility, our method also outperforms a strong DP ridge regression baseline (sufficient statistics perturbation with analytic Gaussian calibration and AdaSSP-style regularization, the strongest one-shot variant we are aware of), while remaining competitive with deep models trained specifically on each dataset. Thus, we emphasize that the main novelty of our work is not in any individual component or the two-phase pretraining, but in the overall paradigm and the carefully motivated design choices that make these components work together.

That said, we agree that the suggested ablation would be valuable. We had run part of it already (Figure 14 of the original submission) and have now run it more completely (see Appendix G, Table 8). Here is RMSE / std(y) over 39 of our datasets for different settings:

pretrainingclean releaseε = 1ε = 8
clean only (MomentPFN)0.662.031.28
noisy from scratch0.820.900.74
clean, then noisy (reported)0.740.890.73
mixed curriculum (30% clean)0.660.890.73

If a model has never seen noisy statistics during training, it clearly cannot properly parse them when presented some at inference time. Thus, noisy pretraining is an essential part of our paradigm, rather than a separate engineering trick, while the specific pretraining schedule matters much less. Specifically, training on noisy statistics from scratch is only about 0.01 RMSE worse than starting from a clean initialization, suggesting that clean initialization provides only a small benefit rather than being the key to downstream performance. Moreover, the mixed curriculum performs essentially identically to our reported model on noisy releases, with only a slight improvement on clean statistics.

(Our response continues in a second official comment below.)

This continues our response above.

Q2: robustness to a mismatch between the ε seen in training and at inference.

This is a natural question from a privacy perspective! We added the following ablation (see Appendix G, Table 9): we noise the statistics at one budget and tell the model another (i.e. set its budget tokens to be a different budget), for all privacy budget pairs in {0.5,1,2,4,8}\{0.5, 1, 2, 4, 8\}. In this table, the rows are the true budget, while columns are what we told the model (RMSE / std(y)):

true \ told0.51248
0.51.011.081.211.421.70
10.920.910.931.011.22
20.890.860.850.870.97
80.820.790.760.750.76

From these results, it appears that telling the model that the budget is smaller than it actually is, is harmless (perhaps it discounts the signal in the moments a bit more than needed), BUT telling it that the budget is larger hurts, roughly in proportion to the gap in privacy budget. We see this as evidence that the model has learned to "trust" the moments more, and reads into the extra noise as a signal unnecessarily. Thank you for pushing us to add this ablation, we think it's a really cool insight under the hood of the model.

Q3: s=1s = 1 and w=5w = 5 are never ablated.

You are correct, we did not ablate these settings, due to computational limits and that our results were already compelling. Here, we consider how to properly ablate these parameters, and include results from the new sweep (see Appendix G, Table 10).

Consider the Gram block's share of the squared sensitivity, given w2NG/(w2NG+LΣk)w^2 N_G / (w^2 N_G + L \Sigma_k): 0.62 (d=8d = 8) and 0.88 (d=20d = 20) at w=5w = 5, against 0.06 and 0.23 at w=1w = 1 and 0.96 and 0.99 at w=20w = 20. In order to ablate these, we need to do a full 60k-step pretraining, so we just checked tuning each, one at a time.

We found that neither setting mattered that much. At w=1w = 1 and w=20w = 20, the results were within 0.01 of what we reported in the main paper (0.90 and 0.90 vs 0.89 at ε = 1; 0.74 and 0.74 vs 0.73 at ε = 8). Squinting, you might say that w=20w = 20 pays a little on CRPS and w=1w = 1 pays a little when ε = 8 (0.72 vs 0.70).

W2: clarity.

Thanks for your care in reviewing our paper, and in pointing out areas where the presentation could have been clearer, especially with the notation. We revised the notation throughout the paper to resolve any place where it was overloaded (please see the revised PDF). Additionally, we added a notation table in Appendix B, which we hope will make the paper easier to follow (as we agree, there is a lot of notation to track). We have also fixed the typos and the row-count inconsistency you pointed out, and have added a "Limitations" paragraph to Section 4.2 where we discuss some of the caveats that you and other reviewers have brought up.

Thank you again, we hope our additional ablations and clarifications have addressed your concerns. We would be grateful if you would consider raising your score.

Thank you for your engagement throughout the review process! Let us know if this answers your question and if you would like any clarification.

Reviewer_eK49OpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors propose a DP-enabled PFN style tabular foundation model that is conditioned for prediction using moment-based features derived from the input data. The Chebyshev moment mechanism allows the model to handle datasets that were privatized for DP and still extract meaningful statistical information about the input data. The author focus on regression and test this method on 52 regression + 33 classification tabular datasets from previous benchmarks and show that this method outperforms training NNs with DP-SGD and or pretraining a TabPFN on synthetic private data.

Critical strengths

The paper is well motivated and addresses a problem not previously addressed. The authors propose a novel method for working with DP tabular data and empirically show strong performance on a large number of datasets. The authors also thoroughly evaluate their method and show that the moment mechanism indeed captures meaningful information from DP-fied datasets better than previous works.

Critical weaknesses

None

Questions for authors
  • How does MomentPFN compare to TabPFN? Figure 1 shows MomentPFN's performance without DP, but another comparison to the vanilla (2.5) TabPFN would be useful to understand the utility gain/loss of MomentPFN without DP. Section G.5 seems to touch on this, but only against GBDT.
2 comments

We really appreciate the time you took to review our paper, and your overall positive assessment! Below we answer your question, and we have updated the PDF as well to address all the reviewers' concerns (revisions in blue).

Q: How does MomentPFN compare to vanilla TabPFN 2.5 without DP? Section G.5 only compares against GBDT.

We think this is a good question, and a fair push to include the obvious comparison; it is a natural reference for reading Figure 1 and the results of Section G. We ran an additional experiment in which we fitted TabPFN 2.5 on the raw training rows of every regression table (so NO privacy) and scored it on the same test rows as everything else. In the revision this appears in Section G as Table 7:

no privacy, RMSE / std(y)numeric (17)mixed (22)large, 50k (13)all (52)
ridge0.6060.7530.6440.678
MomentPFN, clean release0.5420.7440.6270.649
GBDT0.4170.6430.4480.521
TabPFN 2.5, raw rows0.3680.6120.4010.479

We found that, perhaps unsurprisingly, the strong TabPFN 2.5 foundation model, given all the real rows, is well ahead of every non-private reference, GBDT included. We expect this for two reasons: 1. TabPFN 2.5 is a large, carefully engineered foundation model, and 2. it reads the complete table in context, so it has access to all of the information in the data. Our MomentPFN instead reads a lossy, fixed-size summary of the table, L[(k+1)3−1]+NGL[(k+1)^3 - 1] + N_G statistics. In exchange, this summary stays the same however many rows the table has, whereas TabPFN's inference cost grows linearly with the number of rows. As a result, even without noise, MomentPFN's performance is between non-private ridge and GBDT, with its largest gain over ridge on the numeric tables. We thus view this gap to GBDT and TabPFN as the price of a representation that can be privatized once and then reused for every subsequent query at no additional privacy budget, by the post-processing property of DP.

Of course, in the private regime (the main focus of this paper), we cannot place the rows directly in context, so TabPFN 2.5 can only validly be used via DP synthetic data, which Figure 1 shows is actually the weakest of the private options (even with a very strong DP synthetic data approach like AIM). We have also placed this "no-privacy" TabPFN reference next to the pretraining-schedule ablation requested by Reviewer FjTx (Appendix G, Table 8), for easy comparison.

Again, we thank you for your continued engagement with our paper, and for your positive feedback!

Thank you for your engagement throughout the review process! Let us know if this answers your question and if you would like any clarification.

Reviewer_pZfXOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a model called NoisyMomentPFN for privacy-preserving tabular prediction. This method uses Chebyshev moments and a Gram matrix to extract statistical information, and uses the Gaussian mechanism to achieve differential privacy. Then, a pretrained Transformer is used to make predictions. This method does not require retraining for new datasets. The authors conduct experiments on 52 regression datasets and 33 classification datasets. The results show that the proposed method has advantages in prediction efficiency and some probabilistic prediction metrics, but its point prediction accuracy is generally lower than that of DP-SGD MLP.

Critical strengths

1、The authors achieve differential privacy protection by adding noise to the statistical information of the training data. With the relevant parameters made public in advance, each dataset only needs to undergo privacy processing once. Subsequent predictions do not consume additional privacy budget, and the model does not need to be retrained.

Critical weaknesses

1、In Section D on page 18, the authors use the maximum and minimum values of the entire dataset (including the training and test sets) for data scaling, and treat these parameters as public information. However, the privacy proof requires these parameters to be public in advance. There may be a situation where these parameters need to be calculated from private data in practical applications. In this case, the original differential privacy proof cannot be directly applied.

2、Although the proposed method performs well on the CRPS metric, Figure 9 shows that on mixed-type datasets, the average actual coverage of the 90% prediction intervals is only 63%–66%. This raises concerns about the reliability of the predicted uncertainty on mixed-type datasets.

Questions for authors

1、The paper uses the maximum and minimum values of the entire dataset for data scaling in the experiments and treats these parameters as public information. However, if these public parameters cannot be obtained in advance in practical applications, how can the entire data processing procedure be guaranteed to satisfy differential privacy? If predetermined public bounds are used, will the experimental results change significantly?

2、In Appendix G.1, the authors state that the insufficient coverage of prediction intervals on mixed-type datasets may be related to the encoding of categorical features. Have they verified this reason through experiments? If other categorical feature encoding methods are used, can the coverage of prediction intervals be improved?

2 comments

We are very appreciative of the time that you took to review our paper. Below, we respond to each of your points (changes in the revised PDF are highlighted in blue).

W1/Q1: the scaling bounds come from the whole table and are treated as public; what if they are not available in advance, and do results change with predetermined bounds?

This is a very fair point, but one that we were up front about in the original submission (footnote 2 of Section 3, Appendix B.1, and Section D). We are sure to be even more explicit about it in the revision. We made this choice as a convention for our benchmark, as hand-curating a public range for each of 52 tables would have introduced arbitrary decisions of our own.

To that end, we'd like to point out two design choices in our benchmarking: our comparisons between methods do not depend on this convention, since DP ridge and the DP-SGD MLP are scaled with the same bounds. Additionally, NoisyMomentPFN is pretrained with bounds drawn from all sorts of regimes, including ranges much wider than the actual data, so that it does not learn to assume the data fills the range cleanly, even though that assumption would potentially help performance were we allowed to make it (see Appendix B).

So, we scored all three private methods under two alternative assumptions (see Appendix G, Table 11). We first considered very loose, assumed public bounds (the range widened by half its length on each side, as one might get from a coarse public description of a column). We found that every method lost some accuracy, but that the ordering among the methods did not change.

More interestingly, we also tried private bounds: in this setting, 10% of the ϵ\epsilon budget is spent on exponential-mechanism estimates of the 0.5% and 99.5% quantiles of every column on the training rows. We then clip the rows to those private quantiles, and the release spends the remaining 90% of the budget. The whole pipeline is then truly (ϵ,δ)(\epsilon, \delta)-DP in the way you requested, with no public assumption on the bounds.

RMSE / std(y), 39 tablesNoisyMomentPFNDP ridgeDP-SGD MLP
public bounds, ϵ=1\epsilon=10.9061.0700.803
private bounds, ϵ=1\epsilon=10.9030.9010.882
public bounds, ϵ=8\epsilon=80.7580.8060.717
private bounds, ϵ=8\epsilon=80.7230.7390.693

We actually found that this small amount of budget for private bounds did not hurt performance: the quantiles clip only a few percent of outliers, but also effectively tighten the range. This is a classic tradeoff, and it turned out to be worth about as much as the privacy budget it costs. Thus, with no assumed public bounds at all, NoisyMomentPFN is level with its public-bounds result at ϵ=1\epsilon = 1 and better at ϵ=8\epsilon = 8 (as is, we note, DP ridge). We believe this experiment strengthens the result and makes the method more practical, so thank you for the push to run the additional ablation!

W2/Q2: coverage of only 63% to 66% on the mixed stratum; was the categorical explanation verified?

This is a very reasonable critique of our results as stated. We'd first like to note that CRPS is a proper scoring rule, so the CRPS comparison on the mixed stratum already penalizes NoisyMomentPFN for having "too-narrow intervals." Despite that penalty, it is still ahead of the Gaussian-head MLP (see Figure 8). That said, we now state the coverage observation in a Limitations paragraph in the main body (Section 4.2).

We additionally tested the categorical explanation (see Table 4 in Appendix G), and found that it is not the cause. We evaluated the noise-free MomentPFN on clean releases of the mixed tables, and scored the private model on the 17 mixed tables with at least four numeric columns with and without their categorical columns, using the identical model.

mixed stratum, 90% interval coverageNoisyMomentPFN (ϵ=1\epsilon=1 / 88)MomentPFN (clean)
every column0.63 / 0.660.65
numeric columns only0.62 / 0.640.64

This shows that the shortfall is not caused by the privacy noise (the clean model shows the same coverage), and since dropping the categorical columns barely changes things, it is not an encoding issue either. Instead, we believe it is a property of the prior on these particular tables, whose targets are often skewed or few-valued (we now say this in Appendix G). As the shortfall does not depend on the privacy budget at all, it could be corrected by widening the intervals by a factor calibrated on tables from the synthetic prior, at no additional privacy cost. The more principled fix is of course an expanded prior with a richer treatment of such targets, which we view as a natural and promising avenue for improving this V1 NoisyMomentPFN model.

Thank you again for your time and attention; we believe we have addressed your concerns, and if so, we would be grateful if you would consider raising your score.

Thank you for your engagement throughout the review process! Let us know if this answers your question and if you would like any clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.