acceptodds
Under review as a conference paper at ICLR 2027

Differentially Private Adaptations for Tabular Foundation Models

Abstract

Tabular foundation models (TFMs) achieve excellent predictive performance on novel tasks through in-context learning (ICL). Tabular ICL provides labeled records as context for pretrained models, avoiding costly per-task training or fine-tuning. However, the sensitivity of records for tabular ICL raises privacy concerns. Our work first shows that this is a real risk by instantiating highly effective membership inference attacks to infer whether specific target records were part of the private context in a TFM. To provably protect against such attacks, we propose two novel methods for differentially private adaptations of TFMs. First, we present TabDPSGD which adapts a TFM via gradient-based training on a learnable context and a subset of the model's parameters. TabDPSGD releases the trained context and the adapted weights, which can be used together for predictions informed by private data. However, gradients are not available for API-based TFMs. To remove this constraint, we also propose TabPATE, which only requires access to TFM outputs. TabPATE partitions private data across an ensemble of TFM teachers, privately aggregates their predictions on synthetic candidate queries, and releases the resulting labeled candidates as a student context for ICL. Across seven datasets and two TFMs, TabDPSGD beats the strongest evaluated private baseline on six datasets, while TabPATE outperforms private classification baselines.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Tabular Foundation Models, In-Context Learning, Membership Inference Attacks, Differential Privacy, Private Adaptations

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9tUWOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies differentially private adaptation of tabular foundation models (TFMs). The authors first show that private records used as in-context examples in TFMs are vulnerable to membership inference attacks. To address this issue, they propose two approaches: (1) TabDPSGD, which uses DP-SGD to optimize a learnable context and a subset of the TFM parameters; and (2) TabPATE, which partitions private data among TFM teachers, privately aggregates their predictions on synthetic queries, and uses the resulting labeled examples as a student context.

Critical strengths
  • The paper studies privacy concerns in tabular foundation models, which is a valid problem.

  • The paper proposes two approaches for private TFM adaptation, building on two well-established DP techniques, DP-SGD and PATE.

Critical weaknesses

My main concern is the limited technical novelty compared to prior work [1]. This prior work studies privacy leakage from in-context examples in LLMs using membership inference attacks and proposes two approaches for differentially private prompt learning: using DP-SGD to optimize prompts and using PATE to construct a private student context through teacher aggregation. This paper follows essentially the same framework, and the main difference appears to be the application domain, moving from text-based LLMs to tabular foundation models, but the paper does not clearly identify or address fundamental challenges unique to tabular data that require new algorithmic solutions. Therefore, I find the overall technical contribution incremental relative to prior work.

[1] Duan, Haonan, et al. "Flocks of stochastic parrots: Differentially private prompt learning for large language models." Advances in Neural Information Processing Systems 36 (2023): 76852-76871.

Questions for authors

Could the authors clarify the key methodological novelty of this work?

3 comments

We thank the Reviewer for the careful evaluation and constructive feedback. Below, we address the concerns raised about our contributions and technical novelty.

W1. The technical contribution appears incremental relative to Duan et al. (2023). Please clarify which challenges are specific to tabular data and how the proposed methods address them.

Q1. Could the authors clarify the key methodological novelty of this work?

To the best of our knowledge, our work is the first to adapt TFMs under DP. Beyond Duan et al. [1], it makes the following main contributions. The first is a new privacy analysis, and the other two address challenges specific to TFMs and tabular data that Duan et al. do not face.

(1) Tighter accounting of binary GNMax. Standard PATE, which PromptPATE applies unchanged, bounds the cost of each query from the probability that the noisy vote leaves the plurality class, through a bound that holds for any number of classes. It also takes a worst case over neighbors separately for each query, although one record can affect only one teacher. Our accountant in Eq. (11) computes the exact Rényi divergence of each binary label in closed form, fixes the affected teacher across all queries and only then maximizes over teachers. On identical votes and the same budget, it answers a median of 2.46 times as many labels as standard PATE (Appendix H.2). These additional labels come from the analysis alone and give the student a larger context. The accountant uses only the binary votes, the Gaussian noise and the disjoint partition of the private data, so it applies to any binary PATE beyond TFMs. Theorem 6 further allows this data-dependent cost to be tested privately before any label is released, through a generalized propose-test-release.

(2) DP-SGD for TFM adaptation. Existing fine-tuning methods for TFMs train with the real table as the context. Every query then attends to all context records, so per-record clipping no longer bounds the influence of one record, and the released model would contain the records. Private prompt tuning for LLMs does not face this, since its private examples enter only as inputs. Every TabDPSGD variant therefore replaces the real table with a small learned context in the input space of the TFM, so that each private record enters only as a query. A TFM computes each query separately, so no gradient couples two private records and the per-record guarantee of DP-SGD holds. Since the learned context is a table, it must also cover numerical and categorical features. Its numerical cells are trained directly, and each categorical cell is decoded from trainable logits through a softmax. Besides the context, TabDPSGD covers logit scaling, the head, LoRA and full fine-tuning of a TFM. Opacus does not compute correct per-record gradients for TFMs as they are, so we release a library that fixes this and makes these TFMs and adaptations compatible with Opacus and DP-SGD. Appendix E.6 now describes these fixes.

(3) Queries from the table's own feature space. PromptPATE labels public text, which exists for almost any task and may come from a different distribution. A public table with the features of a private table rarely exists. Tabular features, unlike text, are typed, bounded and low-dimensional, so TabPATE samples its candidate queries directly from the feature space of the table and requires no public data. The numerical bounds of that space depend on the private records, so both of our methods release them under DP, and every reported guarantee includes their cost. PromptPATE's teachers each hold one or a few private examples, whereas a TFM conditions its predictions on many records in its context. Each TabPATE teacher therefore holds a partition of the private table, and TabPATE releases all labeled queries as the student context. More teachers give larger vote margins against the noise but leave each teacher with fewer records. We identify this partitioning as the main utility cost of TabPATE, since the private student stays close to its noiseless teachers in almost all configurations.

We adapt four membership inference attacks to tabular ICL, one for each threat model, crossing label-only or probability access with a passive or an active adversary. With active context injection, the strongest attack reaches 75.4% TPR at 1% FPR.

Overall, TabDPSGD achieves the highest Elo among all private methods, above private training from scratch and DP synthetic data.

Section 1 now states which challenges are specific to TFMs and tabular data and how each of our methods addresses them.

If the above answers address the Reviewer's concerns, we would greatly appreciate it if they could consider updating their score.

References:

[1] Duan et al. Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Models. NeurIPS 2023.

Thank you for the detailed response. I appreciate the clarification of the challenges involved in private TFM adaptation. However, I still have some reservations about the overall methodological novelty. The two main approaches remain closely related to the private prompt-learning framework in prior work, and the other components appear to be relatively straightforward adaptations to the tabular setting.

AuthorsOpenReview ↗

Thank you & Rating Increase

Dear Reviewer,

Thank you for the thoughtful follow-up. We understand the remaining concern and agree that the high-level frameworks of TabDPSGD and TabPATE are related to PromptDPSGD and PromptPATE, respectively. Our contribution is to develop the privacy analysis and algorithmic adaptations needed to make these techniques work for tabular foundation models, whose in-context learning paradigm introduces challenges that are absent in conventional prompt learning.

We would like to emphasize three distinctions:

  1. A new privacy accountant, rather than a direct application of PATE. Our binary GNMax accountant is a methodological contribution independent of the application to TFMs. Unlike standard PATE accounting, it computes the exact Rényi divergence of binary outputs and exploits the fact that a neighboring dataset can affect only one teacher across all queries. This improves the number of privately answered queries by a median factor of 2.46 under identical votes, noise, and privacy budgets. Moreover, Theorem 6 provides a certified release using generalized propose-test-release. These contributions apply beyond tabular foundation models to binary PATE mechanisms more generally.

  2. Private adaptation requires changing how TFMs consume private records. Unlike conventional prompt learning, where private examples are inputs to the language model, tabular ICL uses private records as a shared context attended to by prediction queries. Directly applying DPSGD to this setting does not provide the required per-record gradient isolation: every query can depend on every context record. TabDPSGD addresses this by replacing the private table with a learned context and treating each private record as an independent query. This requires a different adaptation formulation for typed numerical and categorical features, as well as modifications to the gradient computation to obtain correct per-record gradients for TFMs.

  3. PATE without public in-distribution data for tabular ICL. PromptPATE relies on public examples to construct the student context. For sensitive tabular applications, such as healthcare and finance, an appropriate public table with matching features usually do not exist. TabPATE generates candidate queries directly from the tabular feature space, privately releases the necessary numerical feature bounds, and accounts for this preprocessing in the privacy budget. Furthermore, unlike conventional PATE teachers, which can operate on individual examples, TFM teachers require sufficiently large in-context datasets. Partitioning the private table among teachers introduces a fundamental trade-off between the number of teachers, the context size available to each teacher, and the quality of aggregated predictions. Our evaluation and ablations characterize this trade-off.

We see the contributions as complementary: a new privacy analysis that improves PATE accounting, a DPSGD formulation that enables record-level privacy for tabular ICL, and a PATE-based adaptation that removes the need for public in-distribution data while accounting for the context requirements of TFMs.

We acknowledge that these contributions build on established DP foundations. However, we believe that making these foundations applicable to a new model paradigm, with new privacy analysis and concrete algorithmic solutions to its specific constraints, represents a meaningful methodological contribution rather than a straightforward transfer between application domains.

We hope this further clarifies the distinction between the existing DP frameworks and the contributions of our work, and would appreciate the Reviewer’s reconsideration.

With kind regards,

The Authors

Reviewer_1rofOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper adopted DP-SGD and PATE for tabular foundation models in different settings. By accessing the performance across 7 datasets and 2 TFMs the authors demonstrated that these adaptations are effective and outperform other private baselines.

Critical strengths
  1. The methods seem effective and I especially appreciate seeing the clear benefit under low privacy budget, this makes the proposed methods a lot more practical.

  2. The paper handles both settings where gradients may or may not be available. Both methods adapt classic DP solutions but fits the actually problem well and outperforms the baselines. The proof looks accurate and sound to me too!

Critical weaknesses
  1. It lacks a bit of novelty as both DP-SGD and PATE are well studied in the DP community. The paper could sharpen its contribution more and highlight more what are the specific challenges in adapting these DP solutions to TFMs.

  2. I was hoping to see the algorithms in the main text as they are more fundamental, and gives more context of what problem the paper was trying to solve. I personally cared less about the variants than the actually algorithms.

(a few minor things)

  • A few terms also weren't defined before being use, e.g., Elo rate, R^2.
  • the results currently doesn't show the statistically significance. It'd be great to run each experiment at least a few times to establish that.
Questions for authors
  1. what's the privacy budget used for TabPATE? I wasn't sure if both TabPATE and TabDPSGD have ϵ=1\epsilon =1 and δ=10−5\delta=10^{-5}. It'd be great to show TabPATE's performance under a few different privacy budget too, though they might reflect less changes comparing the TabDPSGD results.

  2. How about training time (or computational cost in general) comparison?

  3. how many teacher models are used in TabPATE? I couldn't find the number. Also how does teach number KK affect the final results?

  4. what's the main challenge adopting DP-SGD and PATE to the TFM settings?

3 comments

We thank the Reviewer for highlighting the gains at strict privacy budgets and the practical value of covering settings for TabDPSGD and TabPATE.

W1. Please clarify the paper’s contributions and the specific challenges of adapting DP-SGD and PATE to TFMs.

Q4. What are the main challenges in adapting DP-SGD and PATE to TFMs?

To the best of our knowledge, our work is the first to adapt TFMs under DP. It makes the following main contributions beyond previous PATE- and DP-SGD-based methods. The first is a new privacy analysis, and the other two address the challenges of adapting DP-SGD and PATE to TFMs.

(1) Tighter accounting of binary GNMax. TabPATE releases many labeled queries as the student context, so the privacy cost of each label limits how many labeled queries the student context can hold. Standard PATE bounds the cost of each query from the probability that the noisy vote leaves the plurality class, through a bound that holds for any number of classes. It also takes a worst case over neighbors separately for each query, although one record can affect only one teacher. Our accountant in Eq. (11) computes the exact Rényi divergence of each binary label in closed form, fixes the affected teacher across all queries and only then maximizes over teachers. On identical votes and the same budget, it answers a median of 2.46 times as many labels as standard PATE (Appendix H.2). These additional labels come from the analysis alone and directly improve the student. The accountant uses only the binary votes, the Gaussian noise and the disjoint partition of the private data, so it applies to any binary PATE beyond TFMs.

(2) DP-SGD for TFM adaptation. The main challenge is that DP-SGD does not apply to tabular ICL directly. Existing fine-tuning methods for TFMs train with the real table as the context. Every query then attends to all context records, so per-record clipping no longer bounds the influence of one record, and the released model would contain the records. Every TabDPSGD variant therefore replaces the real table with a small learned context in the input space of the TFM, so that each private record enters only as a query. A TFM computes each query separately, so no gradient couples two private records and the per-record guarantee of DP-SGD holds. The learned context must also cover the numerical and categorical features of a table, so its numerical cells are trained directly and each categorical cell is decoded from trainable logits through a softmax. The learned context alone ranks second among all private methods, behind only TabDPSGD-LoRA. Opacus does not compute correct per-record gradients for TFMs as they are, so we release a library that fixes this and makes these TFMs and adaptations compatible with Opacus and DP-SGD. Appendix E.6 now describes these fixes.

(3) PATE without public data. The first challenge for PATE is its public data. PATE-based methods, including PromptPATE, label public data, which may come from a different distribution but must exist for the task. A public table with the features of a private table rarely exists. Tabular features, unlike text, are typed, bounded and low-dimensional, so TabPATE samples its candidate queries directly from the feature space of the table and requires no public data. The numerical bounds of that space depend on the private records, so both of our methods release them under DP within the reported budget. The second challenge is that a TFM predicts by conditioning on the records in its context. Each teacher holds a partition of the private table, so splitting the table shrinks every teacher's context. More teachers give larger vote margins against the noise but leave each teacher with fewer records. We identify this partitioning as the main utility cost of TabPATE, since the private student stays close to its noiseless teachers in almost all configurations. All teachers share one frozen TFM and TabPATE only requires its predictions, so it covers the setting without gradient access, including API-based TFMs.

Overall, TabDPSGD and TabPATE achieve the highest Elo among all private methods, above private training from scratch and DP synthetic data.

Section 1 now states these challenges and how each of our methods addresses them.

W2. Please present the core algorithms in the main text.

We have moved the TabDPSGD Algorithm (Algorithm 1) and the TabPATE algorithm (Algorithm 2) to the main paper, using the additional page for the camera-ready version.

W3. Please define Elo rating and R2R^2 before using them.

We now briefly explain both metrics in the Metrics paragraph of Section 6, before any result is reported. R2R^2 is the coefficient of determination on the test split, where 11 is a perfect prediction and 00 matches predicting the mean target for every record. The Elo rating summarizes the pairwise comparisons of all methods across all tables, with the DPSGD MLP fixed at 10001000. Appendix E.4 gives the full definitions.

W4. Please repeat the experiments across multiple runs to assess statistical significance.

We repeated the experiments across three seeds and data splits on two classification datasets (Churn and Electricity) and two regression datasets (Bike Sharing and California Housing) for both TFMs. Each method uses its best obtained search configuration across runs.

The table reports mean AUC or R2R^2 ±\pm standard error over the three seeds.

MethodElectricity AUC ↑\uparrowChurn AUC ↑\uparrowCalifornia Housing R2R^2 ↑\uparrowBike Sharing R2R^2 ↑\uparrow
TabDPSGD-Context (TabICLv2)0.844±0.0080.844 \pm 0.0080.758±0.0200.758 \pm 0.0200.770±0.0140.770 \pm 0.0140.900±0.0030.900 \pm 0.003
TabDPSGD-Context (Real-TabPFN-2.5)0.851±0.0020.851 \pm 0.0020.785±0.0280.785 \pm 0.0280.684±0.0440.684 \pm 0.0440.826±0.0280.826 \pm 0.028
TabDPSGD-Logit (TabICLv2)0.849±0.0040.849 \pm 0.0040.770±0.0160.770 \pm 0.0160.752±0.0240.752 \pm 0.0240.851±0.0030.851 \pm 0.003
TabDPSGD-Logit (Real-TabPFN-2.5)0.850±0.0050.850 \pm 0.0050.775±0.0110.775 \pm 0.0110.714±0.0240.714 \pm 0.0240.678±0.0420.678 \pm 0.042
TabDPSGD-LoRA (TabICLv2)0.855±0.0050.855 \pm 0.0050.771±0.0210.771 \pm 0.0210.749±0.0100.749 \pm 0.0100.865±0.0100.865 \pm 0.010
TabDPSGD-LoRA (Real-TabPFN-2.5)0.858±0.0030.858 \pm 0.0030.786±0.0130.786 \pm 0.0130.698±0.0250.698 \pm 0.0250.782±0.0430.782 \pm 0.043
TabDPSGD-Head (TabICLv2)0.858±0.0050.858 \pm 0.0050.795±0.0100.795 \pm 0.0100.781±0.0120.781 \pm 0.0120.468±0.0110.468 \pm 0.011
TabDPSGD-Head (Real-TabPFN-2.5)0.858±0.0030.858 \pm 0.0030.710±0.0970.710 \pm 0.0970.675±0.0290.675 \pm 0.0290.585±0.0190.585 \pm 0.019
TabDPSGD-Full (TabICLv2)0.824±0.0110.824 \pm 0.0110.818±0.0090.818 \pm 0.0090.745±0.0090.745 \pm 0.0090.736±0.0070.736 \pm 0.007
TabDPSGD-Full (Real-TabPFN-2.5)0.835±0.0040.835 \pm 0.0040.764±0.0120.764 \pm 0.0120.741±0.0110.741 \pm 0.0110.857±0.0120.857 \pm 0.012
DPSGD MLP0.851±0.0030.851 \pm 0.0030.697±0.0520.697 \pm 0.0520.669±0.0100.669 \pm 0.0100.425±0.0120.425 \pm 0.012
DPBoost0.805±0.0090.805 \pm 0.0090.634±0.0610.634 \pm 0.0610.068±0.1210.068 \pm 0.1210.034±0.0310.034 \pm 0.031
AIM0.818±0.0150.818 \pm 0.0150.704±0.0430.704 \pm 0.0430.399±0.0200.399 \pm 0.020−0.041±0.029-0.041 \pm 0.029
MST0.748±0.0040.748 \pm 0.0040.675±0.0310.675 \pm 0.0310.099±0.0790.099 \pm 0.079−0.257±0.091-0.257 \pm 0.091
DP-CTGAN0.713±0.0110.713 \pm 0.0110.512±0.0510.512 \pm 0.051−0.137±0.017-0.137 \pm 0.017−0.330±0.240-0.330 \pm 0.240
Tab-PE0.838±0.0020.838 \pm 0.0020.730±0.0500.730 \pm 0.0500.565±0.0250.565 \pm 0.0250.368±0.0420.368 \pm 0.042

TabDPSGD continues to achieve the highest mean performance on all four datasets, with TabICLv2 Context remaining best on Bike Sharing and TabICLv2 Head on California Housing. On Churn, the best variant changes from Real-TabPFN-2.5 Head to TabICLv2 Full, with the former having a large standard error across runs. On Electricity, the three best variants all reach 0.858 AUC, above the 0.851 of the DPSGD MLP. We have included the complete results in Appendix F.7.

Q1. What privacy budgets do TabPATE and TabDPSGD use? How does TabPATE perform across different (ε,δ)(\varepsilon,\delta) budgets?

Both TabPATE and TabDPSGD use (ε,δ)=(1,10−5)(\varepsilon,\delta)=(1,10^{-5}) in the main experiments. We additionally evaluate TabPATE at ε∈3,10\varepsilon\in\\{3,10\\}, keeping δ=10−5\delta=10^{-5}.

The table below reports AUC for the Churn dataset across different epsilon budgets ε∈1,3,10\varepsilon\in\\{1,3,10\\}.

TeacherChurn AUC (ε=1\varepsilon=1)Churn AUC (ε=3\varepsilon=3)Churn AUC (ε=10\varepsilon=10)
TabICLv20.8250.8360.842
TabPFN-30.8250.8310.832

The AUC rises with the budget for both teacher TFMs, and most of the gain comes between ε=1\varepsilon=1 and ε=3\varepsilon=3. We have added these results alongside the TabDPSGD budget comparison in Table 3.

Q2. How do the methods compare in training time or overall computational cost?

We did not optimize either method for speed, so the times below leave room for speed-ups. One TabPATE release on ACS Employ, with 32,768 candidate queries on one GPU, consists of the teachers' votes on the candidate queries, the GNMax labeling and fitting the student.

TeachersTeacher votesGNMax labelsStudentTotal
TabICLv2526 s8 s105 s10.7 min
TabPFN-3355 s10 s133 s8.3 min

The next table reports the time of one TabDPSGD step on ACS Employ, measured on one NVIDIA A100 with 64 context records and an expected batch size of 2,048. One TabPATE release costs as much as 35 to 120 TabDPSGD steps on TabICLv2.

MethodReal-TabPFN-2.5TabICLv2
TabDPSGD-Context25.6 s10.8 s
TabDPSGD-Logit28.9 s12.0 s
TabDPSGD-Head10.0 s5.4 s
TabDPSGD-LoRA14.0 s5.6 s
TabDPSGD-Full30.8 s18.4 s

The last table reports the peak GPU memory in GiB of one TabDPSGD step on an NVIDIA A40, with 32 context records and a batch size of 32. TabDPSGD-Context trains the learned context, and every other variant trains its own weights with the learned context frozen. The peak depends on this micro batch and not on the expected batch size, since DPSGD accumulates the clipped per-record gradients across micro batches. We compare it without (Standard) and with (Optimized) gradient checkpointing, which recomputes activations in the backward pass, and CPU offloading, which keeps the tensors saved for the backward pass in CPU memory. Optimized values use both techniques, unless †^\dagger marks gradient checkpointing alone or ‡^\ddagger marks CPU offloading alone.

MethodReal-TabPFN-2.5 StandardReal-TabPFN-2.5 OptimizedTabICLv2 StandardTabICLv2 Optimized
TabDPSGD-Context7.270.95$^\dagger$2.531.38$^\ddagger$
TabDPSGD-Logit0.250.250.760.76
TabDPSGD-Head0.260.260.770.77
TabDPSGD-LoRA8.451.000.770.77
TabDPSGD-Full11.392.5513.0810.37$^\ddagger$

With both techniques, every variant except Full on TabICLv2 stays below 3 GiB. They cost time, for instance a Full step on Real-TabPFN-2.5 at a batch size of 32 takes 0.99 s instead of 0.50 s. We have explained both optimizations more clearly and added these measurements to Appendix G.

Q3. How many teachers does TabPATE use, and how does the teacher count affect the results?

The selected number of teachers varies by dataset and backbone, ranging from 99 to 545 in our main experiments. We ablate the number of teachers (KK) on Churn at (ε,δ)=(1,10−5)(\varepsilon,\delta)=(1,10^{-5}).

Teacher count KK3264128256512768
Churn AUC ↑\uparrow0.6570.7710.7630.7900.8240.798

Performance generally improves with more teachers and reaches its highest AUC at K=512K=512, after which it decreases again at K=768K=768. We have included the teacher count ablation in Appendix H.4.

If these responses address your concerns, we would appreciate it if you would consider raising your score. We remain happy to answer any further questions.

Reviewer_SVXiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates membership leakage from tabular foundation models (TFMs) and proposes two differentially private adaptations. TabDPSGD privately optimizes a learned context and optionally model parameters. TabPATE partitions sensitive records into teacher contexts, privately labels synthetic queries, and releases a student context. Its main theoretical contribution is tighter binary GNMax accounting that retains the identity of the affected teacher across queries, together with a propose-test-release (PTR) certification mechanism. Experiments demonstrate promising utility across seven datasets.

Critical strengths

(1) The evaluation includes multiple adaptation strategies, predictive and synthetic-data baselines, classification and regression, and informative ablations. Accounting for private numerical preprocessing is a strength. The partitioning analysis helps explain TabPATE’s utility loss. (2) The motivation, method overview, and organization are accessible. Detailed appendices make the mathematical reasoning inspectable.

Critical weaknesses

(1) TabDPSGD largely transfers established private prompt tuning and private fine-tuning techniques to TFMs; TabPATE builds closely on PromptPATE. The stronger methodological contribution appears to be the binary accounting and certification construction. Claims of two broadly novel methods should distinguish adaptation engineering from new privacy analysis. (2) API-based teachers receive raw private contexts; the guarantee therefore does not protect those records from the API provider. The trusted-party boundary should be explicit. Minor presentation errors include “We use consider” and the inconsistent TabPATE Elo values of 1128 and 1123.

Questions for authors

(1) What privacy guarantee applies after the actual hyperparameter search and final retraining? (2) Were Table 4 and Figure 9 generated using Theorem 6’s PTR mechanism? Please provide all budget components, acceptance rates, and results including rejection outcomes. (3) How much accounting improvement comes from exact binary divergences versus fixing the affected teacher across queries? (4) Do the main rankings persist across seeds and splits, with uncertainty estimates and a predetermined adaptation-selection rule? (5) Which parties must be trusted, particularly when teacher inference uses a commercial API?

4 comments

We thank the Reviewer for recognizing the breadth of our evaluation, the strength of our accounting of private numerical preprocessing, and our promising utility. We address the comments one by one below:

W1. Claims of two broadly novel methods should distinguish adaptation engineering from new privacy analysis.

To the best of our knowledge, our work is the first to adapt TFMs under DP. It makes the following main contributions beyond previous PATE- and DP-SGD-based methods. The first is a new privacy analysis, and the other two address challenges specific to TFMs and tabular data.

(1) Tighter accounting of binary GNMax. Standard PATE, which PromptPATE applies unchanged, bounds the cost of each query from the probability that the noisy vote leaves the plurality class, through a bound that holds for any number of classes. It also takes a worst case over neighbors separately for each query, although one record can affect only one teacher. Our accountant in Eq. (11) computes the exact Rényi divergence of each binary label in closed form, fixes the affected teacher across all queries and only then maximizes over teachers. On identical votes and the same budget, it answers a median of 2.46 times as many labels as standard PATE. These additional labels come from the analysis alone and directly improve the student. The accountant uses only the binary votes, the Gaussian noise and the disjoint partition of the private data, so it applies to any binary PATE beyond TFMs.

(2) DP-SGD for TFM adaptation. DP-SGD does not apply to tabular ICL directly. Existing fine-tuning methods for TFMs train with the real table as the context. Every query then attends to all context records, so per-record clipping no longer bounds the influence of one record, and the released model would contain the records. Every TabDPSGD variant therefore replaces the real table with a small learned context in the input space of the TFM, so that each private record enters only as a query. A TFM computes each query separately, so no gradient couples two private records and the per-record guarantee of DP-SGD holds. Besides the context, TabDPSGD covers logit scaling, the head, LoRA and full fine-tuning of a TFM. Opacus does not compute correct per-record gradients for TFMs as they are, so we release a library that fixes this and makes these TFMs and adaptations compatible with Opacus and DP-SGD. Appendix E.6 now describes these fixes.

(3) PATE without public data. PATE-based methods, including PromptPATE, label public data, which may come from a different distribution but must exist for the task. A public table with the features of a private table rarely exists for sensitive domains, like the medical or financial, where privacy actually matters. Tabular features, unlike text, are typed, bounded and low-dimensional, so TabPATE samples its candidate queries directly from the feature space of the table and requires no public data. The numerical bounds of that space depend on the private records, so both of our methods release them under DP within the reported budget. Since a TFM requires many records in its context, each teacher holds a partition of the private table, all teachers share one frozen TFM, and TabPATE releases all labeled queries as the student context. Partitioning is the main utility cost of TabPATE, since more teachers give larger vote margins but leave each teacher with fewer records, while the private student stays close to its noiseless teachers in almost all configurations.

Overall, TabDPSGD and TabPATE achieve the highest Elo among all private methods, above private training from scratch and DP synthetic data.

We have made this split between adaptations and privacy analysis explicit in the contributions in Section 1.

W2. API-based teachers receive raw private contexts; the guarantee therefore does not protect those records from the API provider. The trusted-party boundary should be explicit. Q5. Which parties must be trusted, particularly when teacher inference uses a commercial API?

Our DP guarantee protects the private records against parties that query the adapted model or observe its release. This is the threat model that Hanke et al. [1] formalize for private adaptations such as PromptPATE [2], so the answers to queries and the released parameters or student context do not leak the private records, and any API can serve the student context. Each teacher conditions on its partition as context, so the party that runs the teachers must be trusted, as in PromptPATE and as for non-private ICL through the same TFM. In our experiments, the TFMs are open and all teachers run locally at the data holder. Section 5 now states these trusted parties.

Q1. What privacy guarantee applies after the actual hyperparameter search and final retraining?

Following prior work [2, 3, 4], the privacy cost does not include the hyperparameter search. Every reported run, including the final retraining, is calibrated to the reported privacy budget for its selected configuration. This budget also covers the preprocessing, because the numerical bounds of each table depend on its private records and are released under DP. The baselines are selected by the same search. We clarified this in Appendix E.5.

Q2. Were Table 4 and Figure 9 generated using Theorem 6's PTR mechanism? Please provide all budget components, acceptance rates, and results including rejection outcomes.

Our experiments in Table 4 and Figure 9 keep the reporting of standard PATE, which allows a direct comparison with prior work. The acceptance rates and rejection outcomes do not arise, and the reported privacy budget covers the scaling bounds, the optional marginal queries and the labels together.

Our accountant in Eq. (11) builds on the same assumptions as standard PATE and computes this cost more tightly, so the additional labels it answers come from the analysis alone. Theorem 6 extends this analysis with a certified release based on generalized propose-test-release, which privately tests the data-dependent cost against a cost proposed before observing the votes and releases the labels only if the test passes. Section 5 now states that our experiments follow the reporting of standard PATE and that Theorem 6 provides a certified release.

Q3. How much accounting improvement comes from exact binary divergences versus fixing the affected teacher across queries?

Our accountant combines the exact binary divergence with a fixed affected teacher and answers 1.66 to 3.99 times as many labels as standard PATE at (ε,δ)=(1,10−5)(\varepsilon,\delta)=(1,10^{-5}), with a median of 2.46 times. To separate the two, we compare three accountants on identical votes, noise, query order and budget. The intermediate accountant computes the exact divergence of each binary label but still takes the worst-case teacher separately for each query. Our accountant also fixes the affected teacher across all queries. The table reports the number of labels answered.

TeachersDatasetStandard PATEExact divergence, teacher per queryExact divergence, teacher fixed
TabICLv2Churn519397
TabICLv2Credit249611634
TabICLv2ACS Employ4931,3291,367
TabICLv2Polish100165166
TabICLv2Electricity5331,4541,479
TabPFN-3Churn166321322
TabPFN-3Credit5791,3041,379
TabPFN-3ACS Employ4491,2341,256
TabPFN-3Polish366465
TabPFN-3Electricity6672,6542,662
Median ratio to standard PATE1.002.352.46

The exact binary divergences give most of the improvement, with a median of 2.35 times as many labels as standard PATE. Fixing the affected teacher across queries answers 0.3% to 5.8% more labels on every configuration and raises the median to 2.46 times. We added this ablation to Appendix H.2.

Q4. Do the main rankings persist across seeds and splits, with uncertainty estimates and a predetermined adaptation-selection rule?

We repeated the evaluation across three seeds and data splits on Churn, Electricity (Classification Task), Bike Sharing, and California Housing (Regression Task) for both TFMs, reporting mean performance ±\pm standard error. For each method, we select the configuration and use that across the runs.

The table reports mean AUC or R2R^2 ±\pm standard error over the three seeds.

MethodElectricity AUC ↑\uparrowChurn AUC ↑\uparrowCalifornia Housing R2R^2 ↑\uparrowBike Sharing R2R^2 ↑\uparrow
TabDPSGD-Context (TabICLv2)0.844±0.0080.844 \pm 0.0080.758±0.0200.758 \pm 0.0200.770±0.0140.770 \pm 0.0140.900±0.0030.900 \pm 0.003
TabDPSGD-Context (Real-TabPFN-2.5)0.851±0.0020.851 \pm 0.0020.785±0.0280.785 \pm 0.0280.684±0.0440.684 \pm 0.0440.826±0.0280.826 \pm 0.028
TabDPSGD-Logit (TabICLv2)0.849±0.0040.849 \pm 0.0040.770±0.0160.770 \pm 0.0160.752±0.0240.752 \pm 0.0240.851±0.0030.851 \pm 0.003
TabDPSGD-Logit (Real-TabPFN-2.5)0.850±0.0050.850 \pm 0.0050.775±0.0110.775 \pm 0.0110.714±0.0240.714 \pm 0.0240.678±0.0420.678 \pm 0.042
TabDPSGD-LoRA (TabICLv2)0.855±0.0050.855 \pm 0.0050.771±0.0210.771 \pm 0.0210.749±0.0100.749 \pm 0.0100.865±0.0100.865 \pm 0.010
TabDPSGD-LoRA (Real-TabPFN-2.5)0.858±0.0030.858 \pm 0.0030.786±0.0130.786 \pm 0.0130.698±0.0250.698 \pm 0.0250.782±0.0430.782 \pm 0.043
TabDPSGD-Head (TabICLv2)0.858±0.0050.858 \pm 0.0050.795±0.0100.795 \pm 0.0100.781±0.0120.781 \pm 0.0120.468±0.0110.468 \pm 0.011
TabDPSGD-Head (Real-TabPFN-2.5)0.858±0.0030.858 \pm 0.0030.710±0.0970.710 \pm 0.0970.675±0.0290.675 \pm 0.0290.585±0.0190.585 \pm 0.019
TabDPSGD-Full (TabICLv2)0.824±0.0110.824 \pm 0.0110.818±0.0090.818 \pm 0.0090.745±0.0090.745 \pm 0.0090.736±0.0070.736 \pm 0.007
TabDPSGD-Full (Real-TabPFN-2.5)0.835±0.0040.835 \pm 0.0040.764±0.0120.764 \pm 0.0120.741±0.0110.741 \pm 0.0110.857±0.0120.857 \pm 0.012
DPSGD MLP0.851±0.0030.851 \pm 0.0030.697±0.0520.697 \pm 0.0520.669±0.0100.669 \pm 0.0100.425±0.0120.425 \pm 0.012
DPBoost0.805±0.0090.805 \pm 0.0090.634±0.0610.634 \pm 0.0610.068±0.1210.068 \pm 0.1210.034±0.0310.034 \pm 0.031
AIM0.818±0.0150.818 \pm 0.0150.704±0.0430.704 \pm 0.0430.399±0.0200.399 \pm 0.020−0.041±0.029-0.041 \pm 0.029
MST0.748±0.0040.748 \pm 0.0040.675±0.0310.675 \pm 0.0310.099±0.0790.099 \pm 0.079−0.257±0.091-0.257 \pm 0.091
DP-CTGAN0.713±0.0110.713 \pm 0.0110.512±0.0510.512 \pm 0.051−0.137±0.017-0.137 \pm 0.017−0.330±0.240-0.330 \pm 0.240
Tab-PE0.838±0.0020.838 \pm 0.0020.730±0.0500.730 \pm 0.0500.565±0.0250.565 \pm 0.0250.368±0.0420.368 \pm 0.042

TabDPSGD retains the highest mean performance on all four datasets, with TabICLv2 Context remaining best on Bike Sharing and TabICLv2 Head on California Housing. Every best variant is above the strongest baseline by more than its standard error. On Churn, the best variant changes from Real-TabPFN-2.5 Head to TabICLv2 Full, with the former showing a large standard error across runs. On Electricity, the three best variants all reach 0.858 AUC, above the 0.851 of the DPSGD MLP. We have included the complete results in Appendix F.7.

We have also corrected the mentioned minor typos in this revision.

If the above answers address the Reviewer's concerns, we would greatly appreciate it if they could consider updating their score. We are also happy to address any remaining questions.

References:

[1] Hanke et al. Open LLMs are Necessary for Current Private Adaptations and Outperform their Closed Alternatives. NeurIPS 2024.

[2] Duan et al. Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Models. NeurIPS 2023.

[3] Li et al. Large Language Models Can Be Strong Differentially Private Learners. ICLR 2022.

[4] Tramèr and Boneh. Differentially Private Learning Needs Better Features (or Much More Data). ICLR 2021.

AuthorsOpenReview ↗

Have concerns been addressed?

We would like to follow up on our responses. Do they adequately address the Reviewer's concerns?

Comments on the paper

1 comment

We would like to thank all the Reviewers for their valuable feedback and insightful comments, which greatly helped us further improve our submission. Reviewers recognized that our work addresses a valid and practically relevant problem, privacy in tabular foundation models (TFMs) (Reviewer 9tUW), and that our methods fit this problem well and outperform the private baselines (Reviewer 1rof). The Reviewers acknowledged our broad evaluation across multiple adaptation strategies, predictive and synthetic data baselines, and classification and regression tasks, with informative ablations (Reviewers SVXi and 1rof). Additionally, Reviewer 1rof highlighted the "clear benefit under a low privacy budget", which "makes the proposed methods a lot more practical".

Building on the Reviewers' feedback, we conducted additional experiments and provided detailed clarifications. Here, we summarize the highlights of the rebuttal:

  • Contributions and TFM specific challenges: We revised Section 1 to clarify our contributions, and the challenges specific to TFMs and tabular data: 1) We provide a tighter privacy accounting for binary GNMax, which enables our TabPATE to answer substantially more labels than standard PATE, improving its privacy-utility trade-off. 2) We lift the most significant limitation of PATE-based methods, namely their need for labeled public in-distribution data. Our TabPATE releases the numerical feature bounds under DP and samples its queries from within these bounds instead of the public labeled data. 3) We release a new library for privacy-preserving TFM adaptations, described in the new Appendix E.6.
  • Accounting ablation: To show where our accounting gain comes from, as asked by Reviewer SVXi, we compared standard PATE, an accountant with exact binary divergences alone, and our full accountant on identical votes, noise and budget for both teacher TFMs on all classification tables. Both parts consistently increase the number of labels answered at the same budget, with the exact binary divergences contributing most of the gain and fixing the affected teacher adding labels on every configuration. We highlighted these gains with an ablation in Appendix H.2.
  • Statistical significance: Following the recommendation of Reviewers SVXi and 1rof, we repeated the evaluation of every TabDPSGD variant on both TFMs and of all baselines across multiple seeds and data splits on two classification and two regression tables, each method with its selected configuration. The ranking of the main paper persists, as TabDPSGD outperforms every private baseline on all four tables by more than the standard error. We added the complete results with standard errors to Appendix F.7.
  • TabPATE across privacy budgets and teacher counts: As suggested by Reviewer 1rof, we further evaluated TabPATE at additional privacy budgets and across a range of teacher counts. The utility of TabPATE rises with the privacy budget for both teacher TFMs, with most of the gain when moving away from the strictest budget. The utility also increases with the number of teachers up to a point and then drops. This reflects the partitioning trade-off of TabPATE, since more teachers give larger vote margins but leave each teacher with fewer records. We added these results to Table 3 and Appendix H.4.

We believe these improvements substantially strengthen both the contribution and the empirical support of our work. Once again, we thank the Reviewers for their constructive feedback. We look forward to further discussions, and please let us know if we can provide any additional information to further enhance your assessment of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.