Differentially Private Adaptations for Tabular Foundation Models

Tabular foundation models (TFMs) achieve excellent predictive performance on novel tasks through in-context learning (ICL). Tabular ICL provides labeled records as context for pretrained models, avoiding costly per-task training or fine-tuning. However, the sensitivity of records for tabular ICL raises privacy concerns. Our work first shows that this is a real risk by instantiating highly effective membership inference attacks to infer whether specific target records were part of the private context in a TFM. To provably protect against such attacks, we propose two novel methods for differentially private adaptations of TFMs. First, we present TabDPSGD which adapts a TFM via gradient-based training on a learnable context and a subset of the model's parameters. TabDPSGD releases the trained context and the adapted weights, which can be used together for predictions informed by private data. However, gradients are not available for API-based TFMs. To remove this constraint, we also propose TabPATE, which only requires access to TFM outputs. TabPATE partitions private data across an ensemble of TFM teachers, privately aggregates their predictions on synthetic candidate queries, and releases the resulting labeled candidates as a student context for ICL. Across seven datasets and two TFMs, TabDPSGD beats the strongest evaluated private baseline on six datasets, while TabPATE outperforms private classification baselines.