- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes NoisyMomentPFN, a tabular foundation model that allows for an unlimited number of queries with differentially private results at much smaller cost than ad-hoc trained DP-SGD models. The model is trained in two macro-phases, first on clean data (producing MomentPFN) and then on noised data. Instead of tabular rows, the model is trained and consumes a fixed-size private release. This is composed of the Chebyshev moments of the target along public random feature projections and the Gram matrix of the augmented columns. The model is evaluated against other differentially private baselines. It achieves competitive utility, worse in RMSE than a DP-SGD MLP but comparable predictive distributions, at much smaller compute cost, as it does not require re-training and instead only needs the single release computation.
Critical strengths
- Effective and cheap: the model generalizes well and performs on par with ad-hoc DP-SGD. The compute cost, excluding the training of the backbone, is negligible.
- Broad evaluation: 53 regression and 33 classification tables tested over multiple privacy budgets. Utility is reported with multiple metrics together with cost. The result analysis is also transparent in the limitations.
Critical weaknesses
- Novelty: the paper is a combination of known techniques in DP. The main novel component, the two-phase (clean-then-noisy) pretraining, is not directly ablated. appendix G.6 reports that the training curriculum with 30% clean releases mixed in, as opposed to the sequential one used, performs better on large tables at a small cost on small ones. Consequently the central claim is not well isolated.
- Clarity: the paper is quite hard to parse, and the notation is particularly hard to follow: P/p (distributions, parameters, prior), S/s/s_abc (dataset, scale, moment weight), and L (both the number of pairs and the loss) are all overloaded. Some interesting results and caveats are in the appendix, which makes reading the main body hard and obscures both the limitations and the paper's most informative analyses. Some typos and inconsistencies are present ("he release" -> "The release"; "filled back in.." double period). Section 4 says the numeric and mixed strata hold tables of at most 10k rows, but Table 1 shows much larger ones (e.g., sarcos, 48933 rows). Together, these issues make the paper's claims harder to verify from the main text alone.
Questions for authors
- Can you provide a direct ablation of: MomentPFN on noisy releases; NoisyMomentPFN trained from scratch; the mixed-releases curriculum; and the checkpoint based on MomentPFN?
- Can you demonstrate the model's robustness (or lack thereof) to a mismatch between the epsilon seen during training and during inference?
- s=1 and w=5 are never ablated, and with these values the Gram term dominates the sensitivity while s=1 clips about a tenth of the projections. Can you show how much the results depend on them?
5 comments
This continues our response above.
Q2: robustness to a mismatch between the ε seen in training and at inference.
This is a natural question from a privacy perspective! We added the following ablation (see Appendix G, Table 9): we noise the statistics at one budget and tell the model another (i.e. set its budget tokens to be a different budget), for all privacy budget pairs in . In this table, the rows are the true budget, while columns are what we told the model (RMSE / std(y)):
| true \ told | 0.5 | 1 | 2 | 4 | 8 |
|---|---|---|---|---|---|
| 0.5 | 1.01 | 1.08 | 1.21 | 1.42 | 1.70 |
| 1 | 0.92 | 0.91 | 0.93 | 1.01 | 1.22 |
| 2 | 0.89 | 0.86 | 0.85 | 0.87 | 0.97 |
| 8 | 0.82 | 0.79 | 0.76 | 0.75 | 0.76 |
From these results, it appears that telling the model that the budget is smaller than it actually is, is harmless (perhaps it discounts the signal in the moments a bit more than needed), BUT telling it that the budget is larger hurts, roughly in proportion to the gap in privacy budget. We see this as evidence that the model has learned to "trust" the moments more, and reads into the extra noise as a signal unnecessarily. Thank you for pushing us to add this ablation, we think it's a really cool insight under the hood of the model.
Q3: and are never ablated.
You are correct, we did not ablate these settings, due to computational limits and that our results were already compelling. Here, we consider how to properly ablate these parameters, and include results from the new sweep (see Appendix G, Table 10).
Consider the Gram block's share of the squared sensitivity, given : 0.62 () and 0.88 () at , against 0.06 and 0.23 at and 0.96 and 0.99 at . In order to ablate these, we need to do a full 60k-step pretraining, so we just checked tuning each, one at a time.
We found that neither setting mattered that much. At and , the results were within 0.01 of what we reported in the main paper (0.90 and 0.90 vs 0.89 at ε = 1; 0.74 and 0.74 vs 0.73 at ε = 8). Squinting, you might say that pays a little on CRPS and pays a little when ε = 8 (0.72 vs 0.70).
W2: clarity.
Thanks for your care in reviewing our paper, and in pointing out areas where the presentation could have been clearer, especially with the notation. We revised the notation throughout the paper to resolve any place where it was overloaded (please see the revised PDF). Additionally, we added a notation table in Appendix B, which we hope will make the paper easier to follow (as we agree, there is a lot of notation to track). We have also fixed the typos and the row-count inconsistency you pointed out, and have added a "Limitations" paragraph to Section 4.2 where we discuss some of the caveats that you and other reviewers have brought up.
Thank you again, we hope our additional ablations and clarifications have addressed your concerns. We would be grateful if you would consider raising your score.
Thank you for your engagement throughout the review process! Let us know if this answers your question and if you would like any clarification.
Thank you so much for your time and for your detailed review! We address each of your concerns below. Our updates are also in the revised PDF (changes in blue).
We appreciate this call for additional ablations, and for clarity on the novelty! We acknowledge that the individual ingredients (the Gaussian mechanism, Chebyshev moments, PFNs, etc.) are all well established, but our contribution is indeed the paradigm itself. To the best of our knowledge, ours is the first natively DP foundation model that reads DP statistics (computed just once from a table) as context, and generates all subsequent predictions privately via post-processing.
Moreover, beyond the paradigm itself, the choice of statistics is not merely an engineering trick. For example, standard DP statistics such as marginals have schema-dependent layouts that are not directly suitable for foundation models. Directly privatizing individual rows destroys much of the signal, as each row's sensitivity is the row itself, while generating DP synthetic data introduces an expensive per-table step. We instead designed a fixed-size release, sliced Chebyshev moments over pairs of random projections plus a Gram block, taking inspiration from prior work on unsliced Chebyshev moment matching (Musco et al. 2025). The benefits are threefold: 1. the size of the vector of statistics does not depend on the number of rows in the data, 2. its sensitivity grows only polylogarithmically in the degree, and 3. it can be privatized via a simple Gaussian mechanism. Importantly, beyond these desirable theoretical properties, our ablations demonstrate that this choice is also empirically effective: every alternative release we pretrained in Section G.4 (Gram alone, Chebyshev alone, monomial moments with or without the Gram block) is worse, and monomial moments are far worse.
Together, our paradigm and choice of statistics make the method extremely efficient: it requires neither per-table DP training nor DP synthesis, and its input size is independent of the number of rows. In terms of utility, our method also outperforms a strong DP ridge regression baseline (sufficient statistics perturbation with analytic Gaussian calibration and AdaSSP-style regularization, the strongest one-shot variant we are aware of), while remaining competitive with deep models trained specifically on each dataset. Thus, we emphasize that the main novelty of our work is not in any individual component or the two-phase pretraining, but in the overall paradigm and the carefully motivated design choices that make these components work together.
That said, we agree that the suggested ablation would be valuable. We had run part of it already (Figure 14 of the original submission) and have now run it more completely (see Appendix G, Table 8). Here is RMSE / std(y) over 39 of our datasets for different settings:
If a model has never seen noisy statistics during training, it clearly cannot properly parse them when presented some at inference time. Thus, noisy pretraining is an essential part of our paradigm, rather than a separate engineering trick, while the specific pretraining schedule matters much less. Specifically, training on noisy statistics from scratch is only about 0.01 RMSE worse than starting from a clean initialization, suggesting that clean initialization provides only a small benefit rather than being the key to downstream performance. Moreover, the mixed curriculum performs essentially identically to our reported model on noisy releases, with only a slight improvement on clean statistics.
(Our response continues in a second official comment below.)