- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies differentially private adaptation of tabular foundation models (TFMs). The authors first show that private records used as in-context examples in TFMs are vulnerable to membership inference attacks. To address this issue, they propose two approaches: (1) TabDPSGD, which uses DP-SGD to optimize a learnable context and a subset of the TFM parameters; and (2) TabPATE, which partitions private data among TFM teachers, privately aggregates their predictions on synthetic queries, and uses the resulting labeled examples as a student context.
Critical strengths
-
The paper studies privacy concerns in tabular foundation models, which is a valid problem.
-
The paper proposes two approaches for private TFM adaptation, building on two well-established DP techniques, DP-SGD and PATE.
Critical weaknesses
My main concern is the limited technical novelty compared to prior work [1]. This prior work studies privacy leakage from in-context examples in LLMs using membership inference attacks and proposes two approaches for differentially private prompt learning: using DP-SGD to optimize prompts and using PATE to construct a private student context through teacher aggregation. This paper follows essentially the same framework, and the main difference appears to be the application domain, moving from text-based LLMs to tabular foundation models, but the paper does not clearly identify or address fundamental challenges unique to tabular data that require new algorithmic solutions. Therefore, I find the overall technical contribution incremental relative to prior work.
[1] Duan, Haonan, et al. "Flocks of stochastic parrots: Differentially private prompt learning for large language models." Advances in Neural Information Processing Systems 36 (2023): 76852-76871.
Questions for authors
Could the authors clarify the key methodological novelty of this work?
3 comments
Thank you for the detailed response. I appreciate the clarification of the challenges involved in private TFM adaptation. However, I still have some reservations about the overall methodological novelty. The two main approaches remain closely related to the private prompt-learning framework in prior work, and the other components appear to be relatively straightforward adaptations to the tabular setting.
Thank you & Rating Increase
Dear Reviewer,
Thank you for the thoughtful follow-up. We understand the remaining concern and agree that the high-level frameworks of TabDPSGD and TabPATE are related to PromptDPSGD and PromptPATE, respectively. Our contribution is to develop the privacy analysis and algorithmic adaptations needed to make these techniques work for tabular foundation models, whose in-context learning paradigm introduces challenges that are absent in conventional prompt learning.
We would like to emphasize three distinctions:
-
A new privacy accountant, rather than a direct application of PATE. Our binary GNMax accountant is a methodological contribution independent of the application to TFMs. Unlike standard PATE accounting, it computes the exact Rényi divergence of binary outputs and exploits the fact that a neighboring dataset can affect only one teacher across all queries. This improves the number of privately answered queries by a median factor of 2.46 under identical votes, noise, and privacy budgets. Moreover, Theorem 6 provides a certified release using generalized propose-test-release. These contributions apply beyond tabular foundation models to binary PATE mechanisms more generally.
-
Private adaptation requires changing how TFMs consume private records. Unlike conventional prompt learning, where private examples are inputs to the language model, tabular ICL uses private records as a shared context attended to by prediction queries. Directly applying DPSGD to this setting does not provide the required per-record gradient isolation: every query can depend on every context record. TabDPSGD addresses this by replacing the private table with a learned context and treating each private record as an independent query. This requires a different adaptation formulation for typed numerical and categorical features, as well as modifications to the gradient computation to obtain correct per-record gradients for TFMs.
-
PATE without public in-distribution data for tabular ICL. PromptPATE relies on public examples to construct the student context. For sensitive tabular applications, such as healthcare and finance, an appropriate public table with matching features usually do not exist. TabPATE generates candidate queries directly from the tabular feature space, privately releases the necessary numerical feature bounds, and accounts for this preprocessing in the privacy budget. Furthermore, unlike conventional PATE teachers, which can operate on individual examples, TFM teachers require sufficiently large in-context datasets. Partitioning the private table among teachers introduces a fundamental trade-off between the number of teachers, the context size available to each teacher, and the quality of aggregated predictions. Our evaluation and ablations characterize this trade-off.
We see the contributions as complementary: a new privacy analysis that improves PATE accounting, a DPSGD formulation that enables record-level privacy for tabular ICL, and a PATE-based adaptation that removes the need for public in-distribution data while accounting for the context requirements of TFMs.
We acknowledge that these contributions build on established DP foundations. However, we believe that making these foundations applicable to a new model paradigm, with new privacy analysis and concrete algorithmic solutions to its specific constraints, represents a meaningful methodological contribution rather than a straightforward transfer between application domains.
We hope this further clarifies the distinction between the existing DP frameworks and the contributions of our work, and would appreciate the Reviewer’s reconsideration.
With kind regards,
The Authors
We thank the Reviewer for the careful evaluation and constructive feedback. Below, we address the concerns raised about our contributions and technical novelty.
To the best of our knowledge, our work is the first to adapt TFMs under DP. Beyond Duan et al. [1], it makes the following main contributions. The first is a new privacy analysis, and the other two address challenges specific to TFMs and tabular data that Duan et al. do not face.
(1) Tighter accounting of binary GNMax. Standard PATE, which PromptPATE applies unchanged, bounds the cost of each query from the probability that the noisy vote leaves the plurality class, through a bound that holds for any number of classes. It also takes a worst case over neighbors separately for each query, although one record can affect only one teacher. Our accountant in Eq. (11) computes the exact Rényi divergence of each binary label in closed form, fixes the affected teacher across all queries and only then maximizes over teachers. On identical votes and the same budget, it answers a median of 2.46 times as many labels as standard PATE (Appendix H.2). These additional labels come from the analysis alone and give the student a larger context. The accountant uses only the binary votes, the Gaussian noise and the disjoint partition of the private data, so it applies to any binary PATE beyond TFMs. Theorem 6 further allows this data-dependent cost to be tested privately before any label is released, through a generalized propose-test-release.
(2) DP-SGD for TFM adaptation. Existing fine-tuning methods for TFMs train with the real table as the context. Every query then attends to all context records, so per-record clipping no longer bounds the influence of one record, and the released model would contain the records. Private prompt tuning for LLMs does not face this, since its private examples enter only as inputs. Every TabDPSGD variant therefore replaces the real table with a small learned context in the input space of the TFM, so that each private record enters only as a query. A TFM computes each query separately, so no gradient couples two private records and the per-record guarantee of DP-SGD holds. Since the learned context is a table, it must also cover numerical and categorical features. Its numerical cells are trained directly, and each categorical cell is decoded from trainable logits through a softmax. Besides the context, TabDPSGD covers logit scaling, the head, LoRA and full fine-tuning of a TFM. Opacus does not compute correct per-record gradients for TFMs as they are, so we release a library that fixes this and makes these TFMs and adaptations compatible with Opacus and DP-SGD. Appendix E.6 now describes these fixes.
(3) Queries from the table's own feature space. PromptPATE labels public text, which exists for almost any task and may come from a different distribution. A public table with the features of a private table rarely exists. Tabular features, unlike text, are typed, bounded and low-dimensional, so TabPATE samples its candidate queries directly from the feature space of the table and requires no public data. The numerical bounds of that space depend on the private records, so both of our methods release them under DP, and every reported guarantee includes their cost. PromptPATE's teachers each hold one or a few private examples, whereas a TFM conditions its predictions on many records in its context. Each TabPATE teacher therefore holds a partition of the private table, and TabPATE releases all labeled queries as the student context. More teachers give larger vote margins against the noise but leave each teacher with fewer records. We identify this partitioning as the main utility cost of TabPATE, since the private student stays close to its noiseless teachers in almost all configurations.
We adapt four membership inference attacks to tabular ICL, one for each threat model, crossing label-only or probability access with a passive or an active adversary. With active context injection, the strongest attack reaches 75.4% TPR at 1% FPR.
Overall, TabDPSGD achieves the highest Elo among all private methods, above private training from scratch and DP synthetic data.
Section 1 now states which challenges are specific to TFMs and tabular data and how each of our methods addresses them.
If the above answers address the Reviewer's concerns, we would greatly appreciate it if they could consider updating their score.
References:
[1] Duan et al. Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Models. NeurIPS 2023.