acceptodds
Under review as a conference paper at ICLR 2027

Training Fair Tabular Foundation Models

Abstract

Tabular Foundation Models (TFMs) have emerged as leading methods for tabular predictive tasks, leveraging in-context learning to predict on new data without task-specific training. Despite the increased use of TFMs in high-stakes decision-making, their fairness properties remain largely unexplored. In this work, we incorporate fairness constraints directly into TFM training, enabling fair predictions in a single forward pass. Our approach addresses two key challenges: limited access to sensitive attributes in training data, and the incompatibility of existing fairness techniques with the in-context learning paradigm. We propose FairTFM, a scalable training strategy based on synthetic fairness tasks and a fairness-aware architecture using a gradient reversal layer, which encourages the model to learn representations invariant to sensitive attributes. Experiments on 120 fairness tasks show consistent improvements in fairness while maintaining competitive accuracy.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Tabular foundation models, fairness

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YX7MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces address the problem of fairness in context learning for tabular foundation models. The transitional method fails in the in-context learning setting as we cannot train on a specific task. Given this difficulty, this paper proposes FairTFM. The first step is to random pick a feature as sensitive attribute and convert this into categorical data. The second step is to train the backbone to predict the task label but not predict the selected sensitive attribute. FairTFM is evaluated in 120 fairness tasks from ACS and datapoint are lies in the Pareto frontier of accuracy vs group fairness metrics.

Critical strengths
  1. The setting of in-context learning fairness is interesting. A few shot learning with a consideration for fairness is a good setting.

  2. The method is easy to understand and the debiasing part make sense.

  3. The current results lie on the Pareto frontier.

Critical weaknesses
  1. The current debiasing objective function is well developed in the literature. The overall objective function for the network is ∇θL=∇θLy−λ∇θLs\nabla_ \theta \mathcal{L} = \nabla_\theta \mathcal{L} _ y - \lambda \nabla_{\theta} \mathcal{L}_s. This function is identical with the paper in 2018. https://arxiv.org/pdf/1807.00199

  2. The current method reduces a lot of performance even compared with LR or RF. Such baselines are vanilla baselines.

  3. The random choice of a feature as a sensitive attribute is problematic. For example, the existing fairness issues arise from the correlation between the sensitive attribute and the task label, like occupation bias. However, a randomized selection cannot mimic the correlation between attributes.

Questions for authors
  1. I am curious if the sensitive attribute is coincidental chose to the core feature in the pertaining, would a very strong noise being injected to the noise, did any prescreen method is used to prevent the random chose to the core feature for the task.

  2. In Figure 1 or Algorithm 1, I am not sure the role of sctxs_{ctx}. It seems that gives a few shot examples of the sensitive group protect. I want to make sure I am understanding correctly.

  3. It is unclear how the proposed method simultaneously reduces both DP and EOd differences, given their potential trade-offs under unequal group base rates. In Figure 5, Both DP difference and EOD difference almost 0.

  4. In lines 419, the method claims the method is a fair representation learning method. I would like to see the TSNE of its representation like Figure 4 in https://arxiv.org/pdf/2211.07907

3 comments
AuthorsOpenReview ↗

Response to Reviewer YX7M (1/2)

We thank the reviewer for the constructive feedback, and for positively highlighting that “the setting of in-context learning fairness is interesting”, that the method is “easy to understand”, and that our results “lie on the Pareto frontier”. We address each point below; changes in the revised paper are shown in blue.

W1: The adversarial objective already exists. We agree, and do not claim a new debiasing objective. Our contribution is making this kind of objective work during pretraining of a tabular foundation model (TFM), so that fairness transfers to unseen tasks in a single forward pass without task-specific training. This raised problems that standard adversarial training does not face:

  • No real sensitive labels for pretraining. We synthesize fairness tasks with the stick-breaking group construction (Section 3.1).
  • The sensitive attribute changes identity in every pretraining task. A separately trained adversary would need re-optimizing for each task. Our gradient-reversal formulation avoids this, and the sensitive attributes of the context rows are given as in-context tokens (Section 3.2).

The cited work, like other task-specific adversarial methods, trains a new model for each task. We compare FairTFM against two such methods, LAFTR and Adversarial Debiasing, in the new Appendix E.6 (Figures 15 and 16). FairTFM provides competitive Pareto points without any task-specific training.

W2: Accuracy relative to LR and RF. Over the 120 tasks (Table 2, Appendix E.1):

  • vs. LR: FairTFM-0.7 matches LR in accuracy (0.729 vs. 0.728), has higher AUCROC (0.754 vs. 0.733), and is fairer on all three metrics (DP 0.077 vs. 0.107, EOD 0.098 vs. 0.134, EOP 0.065 vs. 0.094). It is better than or equal to LR on every metric.
  • vs. RF: FairTFM-0.7 trails RF by 1.6 accuracy points while substantially reducing all three disparities.
  • Fairness-constrained baselines: LR, RF and XGBoost are unconstrained here. When they are trained with Exponentiated Gradient fairness constraints, they move along the same trade-off, and FairTFM remains competitive with them (Figure 5).

Part of the remaining gap is due to model scale and pretraining budget, not the fairness objective. In the new scaling ablation (Table 4, Appendix E.7), a FairTFM-0.1 with 12 layers and 12 attention heads, pretrained for longer, reaches 0.745 accuracy and 0.778 AUCROC. This matches RF (0.744 and 0.772) while being fairer on all three metrics (e.g., DP 0.102 vs. 0.113).

W3: Can a randomly chosen attribute mimic real bias? If we understand the concern correctly, it is that a randomly designated attribute may not reproduce the attribute–label correlations behind real biases. Randomly chosen features in our pretraining data are not independent of the label. The TabICL prior generates datasets from structural causal models, in which features are causally connected to each other and to the label. A designated sensitive attribute therefore inherits whatever correlation the generating model induces, both with the label and with other features that act as proxies; the attribute itself is removed from the feature set, so only its proxies remain (Section 3.1, Figure 1a). Because the designated feature varies across thousands of tasks, pretraining covers a wide range of attribute–label correlations, from little to very strong. Real biases, such as occupation bias, are one point in this range.

The evaluation supports this: FairTFM never sees a real sensitive attribute during pretraining, yet reduces disparities for real gender, race and age attributes across 120 ACS tasks (Figure 2) and 12 tasks from six other benchmarks (Figure 3).

AuthorsOpenReview ↗

Response to Reviewer YX7M (2/2)

Q1: What if the designated attribute is a core feature? We do not prescreen features, and this is deliberate. When the designated attribute is highly predictive of the label, the task has a strong fairness–accuracy tension, and the objective trades the two off according to λ. Such tasks are realistic: in practice, sensitive attributes are often correlated with the target, which is exactly when fairness interventions are needed and costly. The model is not injected with noise. The attribute is removed from the input features and provided only as a separate token for the context rows, and the adversarial term discourages the representation from encoding it or its proxies.

Q2: The role of s in Figure 1 and Algorithm 1. The reviewer’s understanding is correct. s_ctx is the sensitive attribute of each context example, provided as a separate token (Figure 1b; Algorithm 1). It tells the model which attribute is protected in the current task and which group each context example belongs to, i.e., few-shot examples of the protected groups. The query’s sensitive attribute is never given; it is replaced by a learnable mask token. The model therefore needs sensitive attributes only for the labeled context data, not at prediction time.

Q3: How can DP and EOD both be reduced under unequal base rates? When base rates differ across groups, exact DP and exact equalized odds cannot both hold, except for trivial predictors such as a constant one. Reducing both, however, is possible. Unconstrained models carry excess disparity from proxies of the sensitive attribute, and removing that information lowers both gaps without making either exactly zero. This is what we observe at moderate λ: FairTFM-0.7 reduces DP and EOD by 36% and 32% relative to TabPFNv2.5, without driving either to zero (Table 2).

The points in Figure 5 where both are close to zero come from FairTFM-25 at early pretraining checkpoints. These predict very few positives on this task, consistent with the trivial-predictor case. Figure 5 did not show which λ each point uses. We now indicate this in the figure caption, and report group-wise positive prediction rates against a constant predictor in Appendix E.5.

Q4: t-SNE of the representations. We thank the reviewer for the suggestion. We added t-SNE visualizations of FairTFM’s representations alongside the fair-representation baselines (CR, LFR, LAFTR): Figure 7 for gender, and Figure 12 for race and age. Coloring by sensitive attribute and by label shows that FairTFM’s embeddings separate the classes while making the sensitive groups hard to distinguish, and this effect strengthens as λ increases. This complements the linear probes in Table 1, where FairTFM-25 reduces sensitive-attribute AUC from 0.783 to 0.542 while keeping the label predictable.

We hope these clarifications and additions address the reviewer’s concerns and would greatly appreciate it if they would increase their score accordingly.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer YX7M,

As the discussion period is coming to an end, we wanted to kindly check whether our response has addressed your concerns. In particular:

  • (W1) We clarified our contribution relative to existing adversarial objectives, and added a comparison with task-specific adversarial methods (Appendix E.6).
  • (W2) We showed that FairTFM-0.7 matches or exceeds LR on all metrics (Table 2), and added a scaling ablation in which a larger FairTFM matches RF and XGB in accuracy while remaining fairer (Appendix E.7).
  • (W3, Q1) We explained why randomly designated sensitive attributes inherit realistic correlations with the label and its proxies.
  • (Q3) We explained how DP and EOD can both be reduced, and clarified which FairTFM checkpoints appear in Figure 5.
  • (Q4) We added the t-SNE visualizations you requested (Figures 7 and 12).

We would be grateful for any further feedback, and are happy to clarify anything that remains unclear.
Thank you again for your time and constructive review.

Reviewer_e2t7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces FairTFM, a tabular foundation model for fair prediction system and it uses in-context learning. FairTFM tires to address the limited availability of sensitive attributes and the incompatibility issue of existing fairness techniques within the in-context learning approaches. The authors evaluate on 120 tasks from ACS PUMS and 12 tasks from six other fairness benchmarks against classical models, TabPFNv2.5, TabICLv2, FairPFN, Exponentiated Gradient (EG) baselines, and fair representation methods. They report fairness improvements of 32-75% over TabPFNv2.5 at an accuracy cost of 2-11%.

Critical strengths
  1. The paper addresses an important issue and the single forward pass without task specific training is interesting.
  2. The model only needs sensitive attributes for the context rows, so they don't have to be available at the test time.
  3. Any feature can serve as sensitive attribute, so pretraining does not need real labeled sensitive attributes. Also, since the backbone is invariant to column order of the dataset, it does not matter which column is picked.
  4. The evaluation is extensive, covering different classical models, foundation models and fair representation methods.
Critical weaknesses
  1. The technical novelty is limited. Adversarial removal of sensitive attribute using Gradient Reversal Layer (GRL) is already established [1] . Kenfack et. al already address group fairness in Tabular Foundation model in-context learning [2], and meta-learning already trains models to be fair on new tasks from a few samples [3].
  2. The method handles one sensitive attribute per task and the evaluation covers only binary labels, so intersectional, multi-class settings are not addressed. Users still need ss for every context row at deployment, and missing ss values are not discussed.
  3. Some of the fairness gains may come from the model predicting almost no positives. In Figure 5, several FairTFM points score 0.0 on DP difference at about 0.695-0.70 accuracy. This matches an all-negative classifier if about 69.5% of the test set is negative. I would suggest to report group-wise positive prediction rates and compare them against a constant predictor baseline.
  4. Each λ\lambda requires a separate pretrained model, so the users of this model cannot choose the trade-off at inference time.
  5. There is already a in-context learning based TFM available [2]. A comparison with the method of [2] would strengthen the evaluation.
  6. The approach is shown only on a small backbone trained from scratch. It is unclear whether it can be added to a stronger TFM such as TabICLv2 or TabPFNv2.5 through fine-tuning, which could close much of the accuracy gap.

References

[1] Edwards, Harrison, and Amos Storkey. "Censoring representations with an adversary." arXiv preprint arXiv:1511.05897 (2015).

[2] Kenfack, Patrik, et al. “Towards Fair In-Context Learning with Tabular Foundation Models.” Transactions on Machine Learning Research, 2026, https://openreview.net/forum?id=AsBhwD0sqo.

[3] Slack, Dylan, Sorelle A. Friedler, and Emile Givental. "Fairness warnings and fair-maml: learning fairly with minimal data." Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. 2020.

Questions for authors
  1. What context size do you use for the ACS tasks? Do all TFMs receive the same context rows, and which models see ss as an input feature?
  2. Since the sensitive-attribute head never sees yy, how do you expect the GRL to improve EOD and EOP?
  3. In section 3.2, the paper claims that the learned invariance is "conditional and in-context", but no experiment tests this. On the same task, how do the DP difference change if you mark a different column as ss or if you shuffle the context ss values across rows?
6 comments
AuthorsOpenReview ↗

Response to Reviewer e2t7 1/2

We thank the reviewer for the constructive feedback, and for positively highlighting that the paper “addresses an important issue”, that “the single forward pass without task specific training is interesting”, and that “the evaluation is extensive”. We address each point below; changes in the revised paper are shown in blue.

W1: Novelty. We agree that adversarial removal of sensitive information is established, and we do not claim a new fairness mechanism. Our contribution is making group fairness pretrainable into a tabular foundation model (TFM), so that it transfers to new tasks in a single forward pass. This required solving problems that, to our knowledge, have no prior solution:

  1. generating group-fairness structure at pretraining scale without any labeled sensitive attributes (the stick-breaking group construction, Section 3.1);
  2. handling a sensitive attribute whose identity changes in every pretraining task, which rules out the separately trained, task-specific adversaries of [1] and motivates our gradient-reversal formulation (Section 3.2);
  3. preventing sensitive-attribute leakage at inference through a learnable mask token for the query’s sensitive attribute;
  4. showing that statistical group fairness learned from purely synthetic pretraining transfers zero-shot to real tasks (Figures 2 and 3).

The other two works address different settings:

  • [2] studies how to manipulate the context given to an existing model to improve fairness; it does not pretrain a TFM for group fairness. The two approaches are complementary: [2]’s context strategies could be applied with FairTFM as the base model.
  • Fair-MAML [3] is a meta-learning approach: it learns an initialization from which a fair model is obtained by gradient-based fine-tuning on each new task. FairTFM is a fair in-context learner. It adapts to a new task purely by conditioning on the labeled context examples and their sensitive attributes, producing fair predictions in a single forward pass with no parameter updates or task-specific optimization at test time. The two approaches therefore rely on fundamentally different adaptation mechanisms.

W2: Scope (sensitive attributes, multi-class, intersectionality, missing values). We agree these are limitations of the current evaluation.

  • Multi-class labels: the TabICL prior we pretrain on supports up to 10 classes, but our pretraining and evaluation use binary labels. Extending to multi-class requires no architectural change.
  • Non-binary and intersectional sensitive attributes: the stick-breaking construction is not restricted to two groups. Intersectional attributes can be generated by designating several features at once and pretraining on their intersection.
  • Missing values: FairTFM needs s only for context rows, never for test rows.

Our goal in this paper was to establish whether fair prediction on real tasks can be learned from purely synthetic pretraining at all.

W3: Are the gains driven by near-constant predictions? We thank the reviewer for this careful observation. Figure 5 did not show which fairness weight (λ) or checkpoint each FairTFM point corresponds to, so we clarify it here; the revised caption now states it. The points with near-zero DP at about 0.695–0.70 accuracy come from FairTFM-25 at earlier pretraining checkpoints. The lower settings (λ = 0.1 and 0.7) and the final FairTFM-25 checkpoint lie above them.

To make this precise, we computed group-wise positive prediction rates (PR) for the final checkpoints on this task (ACSIncome, Alabama, gender) and compared them with a constant all-negative predictor:

MethodAccDPPR (male)PR (female)
Ground truth––0.4030.214
All-negative0.6880.0000.0000.000
FairTFM-0.10.7610.0650.1780.113
FairTFM-0.70.7600.0630.1700.107
FairTFM-250.7070.0160.0320.017

FairTFM-0.1 and FairTFM-0.7 are well above the constant baseline, with positive rates around 0.11–0.18 and accuracy around 0.76. Only the strongest regularization (λ = 25) predicts few positives on this task.

Even FairTFM-25 still ranks examples well. Its AUCROC is 0.731 averaged over the 120 tasks (Table 2), and on this task FairTFM provides the best Pareto front when performance is measured by AUCROC (Figure 14). This suggests its low positive rate comes from the default 0.5 decision threshold rather than a loss of predictive signal. The t-SNE plots (Figure 7) are consistent with this: as λ grows, sensitive groups become harder to separate while the classes remain separable. We have added this analysis to Appendix E.5.

AuthorsOpenReview ↗

Response to Reviewer e2t7 2/2

W4: Choosing the trade-off at inference. This is a fair limitation. A single FairTFM checkpoint does not expose a trade-off knob at inference; users choose among the released models trained with different λ values and checkpoints. Our comparisons with Exponentiated Gradient (Figures 5, 13 and 14) and with adversarial methods (Appendix E.6) show that these checkpoints already cover a broad range of the trade-off curve. Exposing the trade-off at inference, for example by conditioning the model on λ during pretraining, is a promising extension.

W6: Larger backbones and fine-tuning. We added a scaling ablation (Table 4, Appendix E.7). It pretrains FairTFM-0.1 with 12 layers and 12 attention heads (instead of the original 6-layer and 6-head configuration), for longer (20,000 steps), on continuously sampled datasets. Accuracy rises from 0.725 to 0.745 and AUCROC from 0.758 to 0.778. This matches XGBoost and random forest while remaining fairer than both on all three metrics, so part of the accuracy gap is due to scale rather than the fairness objective. Fine-tuning existing TFMs such as TabPFNv2.5 or TabICLv2 is also promising, but would require integrating a sensitive-attribute encoder and adversarial head into their architectures; we leave this to future work.

Q1: Context size and inputs. For each task, we use a 70%/30% split for context and query. All TFMs receive the same context rows and are evaluated on the same query rows. In the main comparison, no baseline sees s as an input feature. FairTFM sees s only for the context rows, never for the query. We added an experiment where every baseline also receives s as an input feature (Table 4, Appendix E.7). Their accuracy is essentially unchanged, but every baseline becomes less fair, so excluding s is the conservative choice.

Q2: How can the GRL improve EOD and EOP? The sensitive-attribute head does not observe the query label, which is masked. It sees labels only indirectly, through the transformer’s attention to the context examples, which carry both their labels and sensitive attributes. The objective therefore pushes the query representation towards independence from s, which most directly targets demographic parity. We do not condition on the query label, as would be required to target EO explicitly. This is deliberate: at pretraining time the downstream fairness definition is unknown, so we aim for a representation that performs well across criteria. Empirically, the conditional criteria also improve substantially: relative to TabPFNv2.5, FairTFM-25 reduces EOD by 50% and EOP by 74% (Table 2). Conditioning the adversary on the query label, which is available during pretraining, would target EO directly; we now state this in Section 3.2 and the conclusion (Section 5).

Q3: Is the invariance conditional and in-context? By “conditional and in-context” we mean that FairTFM suppresses information about the attribute the context designates as sensitive, rather than degrading the embedding globally.

  • Probing (Table 1): the sensitive attribute becomes much harder to predict from FairTFM’s embeddings (AUC 0.783 for raw features vs. 0.542 for FairTFM-25), while the label stays predictable (0.834 vs. 0.787).
  • t-SNE (Figures 7 and 12): on the same task, with gender, race or age designated in turn, FairTFM’s embeddings make the designated groups hard to distinguish while preserving the class structure.

We hope the above answers address the Reviewer's concerns and would greatly appreciate it if they would increase their score accordingly.

Thank you for the detailed response and the analysis. However, several of my main concerns remain:

  1. The new table supports my concern rather than resolving it. For FairTFM-25, the female/male positive-rate ratio is the same as the ground truth. So, the low DP comes from predicting almost no positives. Since you attribute this to the 0.5 threshold, I would suggest reporting DP/EOD/EOP at a threshold that matches the base rate.
  2. The author's response does not address W5. Without this comparison, it is unclear whether fairness-aware pretraining does better than simple interventions on a stronger TFM.
  3. The 70/30 split in the response conflicts with the 80/20 split in Appendix B.2. The question on missing ss values was about context rows, not test rows.
Reviewer_fVJ9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes to enforce group fairness criteria such as demographic parity, equalized odds and equal opportunity on Tabular foundation models. The architecture adds an adversarial component that tries to predict the sensitive attribute to a nanoTabPFN trained from scratch, with a gradient reversal layer. The training is done on synthetic data where the sensitive attribute is randomly chosen among all of the features (if continuous it is discretized). Experiments are done on more than 100 tasks and show that it helps to mitigate unfairness, with a trade-off in accuracy.

Critical strengths

The paper tackles an important problem of enforcing group fairness criteria for Tabular foundation models which remains underexplored. I find the idea of randomly choosing the sensitive attribute during pretraining well motivated, avoiding the need of real sensitive labels, and results show the reduction of unfairness in real world tasks.

Critical weaknesses
  • I have technical concerns regarding the training, there is no distinction of the fairness criterion, it always keeps the sensitive attribute prediction to the adversarial objective which is as traditionally done for demographic parity only. I was expecting for example for enforcing equalized odds or equal opportunity to have the condition on Y.

  • Also in experiments, there is no comparison to state of the arts methods with traditional in-processing adversarial fairness such as [1,2]

[1] Madras, David, et al. "Learning adversarially fair and transferable representations." International Conference on Machine Learning. PMLR, 2018.

[2] Zhang, Brian Hu, Blake Lemoine, and Margaret Mitchell. "Mitigating unwanted biases with adversarial learning." Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. 2018.

Questions for authors
  1. What distinguishes the fairness criteria during training? How to enforce more EO than DP?
  2. Could you add other state-of-the-art methods to compare, such as [1,2]?
2 comments
AuthorsOpenReview ↗

Response to Reviewer fVJ9

We thank the reviewer for the constructive feedback, and for positively highlighting that we tackle “an important problem” that “remains underexplored”, that randomly choosing the sensitive attribute during pretraining is “well motivated, avoiding the need of real sensitive labels”, and that our results “show the reduction of unfairness in real world tasks”.

We address both concerns below; all changes are shown in blue in the revised paper.

W1 / Q1: Distinguishing fairness criteria (DP vs. EO).

The reviewer is right that our adversarial objective does not condition on the query label. The sensitive-attribute head predicts s_qy from the query representation, and the query label y_qy is masked. The adversary therefore sees labels only indirectly: the transformer builds the query representation by attending to the context examples, which carry their labels y_ctx and sensitive attributes s_ctx. This lets the representation account for how s relates to y within each task. However, it does not condition the adversary on the query’s label, as would be required to target equalized odds or equal opportunity explicitly [2]. The gradient reversal layer thus pushes the query representation towards independence from s, which most directly targets demographic parity. We now state this explicitly in Section 3.2.

This choice is deliberate for a foundation model. At pretraining time, neither the downstream task nor the fairness definition it calls for is known. We therefore aim for a representation that performs well on several criteria at once rather than one tuned to a single notion. Empirically, the objective also substantially reduces the conditional criteria. Relative to TabPFNv2.5 (Table 2, Appendix E.1; Table 3, Appendix E.2):

  • FairTFM-0.7 reduces DP, EOD and EOP by 36%, 32% and 34%;
  • FairTFM-25 reduces them by 51%, 50% and 74%.

To enforce EO more strongly, the adversary can be conditioned on the query label. Query labels are known during pretraining, so the sensitive-attribute head can receive y_qy as an additional input, as in [1, 2]. The adversary then targets s | y rather than s, with no other change to the pipeline. We have added this as a direction for future work in the conclusion.

W2 / Q2: Comparison with adversarial in-processing methods [1, 2].

We thank the reviewer for the suggestion and have added both methods:

  • LAFTR [1] as a fair-representation baseline (Section 4.2 and Figure 6; Appendix E.4, Figures 9–11). LAFTR is the strongest task-specific competitor and contributes several points to the joint Pareto front. FairTFM reaches comparable trade-offs from a single forward pass, without fitting an encoder for each task.
  • End-to-end comparison with LAFTR [1] and Adversarial Debiasing [2] (new Appendix E.6, Figures 15–16). Each uses the architecture from its original paper, and we sweep its adversarial weight over 20 values for LAFTR and 19 for Adversarial Debiasing. FairTFM provides competitive Pareto points for gender, race and age, and Pareto-dominant points when performance is measured by AUCROC. Both baselines must be retrained for every task and every trade-off setting.

The revision also adds a scaling ablation (Table 4, Appendix E.7). A FairTFM with twice as many layers and attention heads, pretrained for longer, matches XGBoost and random forest in accuracy while remaining fairer than both.

We hope the above answers address the Reviewer's concerns and would greatly appreciate it if they would increase their score accordingly.


[1] Madras, David, et al. "Learning adversarially fair and transferable representations." International Conference on Machine Learning. PMLR, 2018.

[2] Zhang, Brian Hu, Blake Lemoine, and Margaret Mitchell. "Mitigating unwanted biases with adversarial learning." Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. 2018

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer fVJ9,

As the discussion period is coming to an end, we wanted to kindly check whether our response has addressed your concerns. In particular, we clarified that the adversary sees labels only through the context examples, which carry both y and s, and not the query label (W1/Q1). FairTFM’s objective therefore most directly targets demographic parity, while also substantially reducing EOD and EOP (Section 3.2). We added both baselines you suggested: LAFTR as a fair-representation baseline (Section 4.2, Appendix E.4), and an end-to-end comparison with LAFTR and Adversarial Debiasing (Appendix E.6) (W2/Q2).

We would be grateful for any further feedback, and are happy to clarify anything that remains unclear.

Thank you again for your time and constructive review.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their thoughtful and constructive feedback. We are glad reviewers found the problem important and underexplored (fVJ9, e2t7), the random designation of sensitive attributes well motivated, as it avoids the need for real sensitive labels (fVJ9, e2t7), the single-forward-pass design interesting (e2t7, YX7M), the method easy to follow (YX7M), and the evaluation extensive (e2t7). We have substantially revised the paper in response; all changes are shown in blue. The main revisions are:

1. New baselines and comparisons

  • Adversarial fairness methods (fVJ9, YX7M): LAFTR [1] as a fair-representation baseline (Section 4.2, Appendix E.4), and an end-to-end comparison with LAFTR and Adversarial Debiasing [2] (Appendix E.6). FairTFM provides competitive Pareto points without any task-specific training.
  • Interventions on stronger TFMs (e2t7): we apply context-selection strategies (group-balanced and uncertainty-based [3]), feature pre-processing (CR, LAFTR), and FairTFM embeddings to TabPFNv2.5 and TabICLv2, and also with FairTFM as the base model (Appendix E.10). Averaged over 27 tasks (15 ACS tasks and 12 tasks from six non-ACS benchmarks; Figures 28–29):
    • interventions on stronger TFMs only cover the high-accuracy end of the Pareto front, reducing DP modestly (at best from ≈0.097\approx 0.097 to ≈0.08\approx 0.08);
    • all lower-unfairness operating points come from FairTFM, alone (e.g., DP ≈0.025\approx 0.025 at 0.7370.737 accuracy) or combined with an intervention (e.g., FairTFM-10 + LAFTR: DP ≈0.035\approx 0.035–0.0450.045 at 0.7450.745–0.760.76 accuracy);
    • simple interventions such as CR, which worsen fairness on stronger TFMs, become effective with FairTFM, showing that fairness-aware pretraining and model-agnostic interventions are complementary. The same trends hold for EOD, EOP, and AUCROC.

2. Clarifications of the method

  • Which fairness notion the objective targets (fVJ9, e2t7): the adversary sees labels only through the context examples, which carry both yy and ss, not through the query label. The objective therefore most directly targets demographic parity, while also substantially reducing EOD and EOP. We state this explicitly in Section 3.2, and discuss conditioning the adversary on the query label as an extension in Section 5.
  • Novelty (YX7M, e2t7): we clarified that our contribution is not a new loss function or debiasing objective, but making group fairness pretrainable into a TFM, so that fairer predictions are obtained in a single forward pass through in-context learning, without any model update. This involves synthesizing fairness tasks without real sensitive labels; extending the TFM architecture with a dedicated encoder that provides the sensitive attribute of each context example as an in-context token, and an adversarial sensitive-attribute head with gradient reversal; handling a sensitive attribute that changes in every pretraining task; and preventing leakage at inference through a learnable mask token for the query's sensitive attribute.

3. Analyses of FairTFM's behavior

  • Near-constant predictions at very high λ\lambda (e2t7, YX7M): group-wise positive rates show that FairTFM-0.1 and FairTFM-0.7 are well above a constant predictor and closer to parity than the ground truth. FairTFM-25 can predict few positives at the default threshold. With a threshold matching the base rate, its accuracy rises from 0.7070.707 to 0.7770.777 with disparities still about half those of TabPFNv2.5, so this is a calibration effect rather than a loss of predictive signal (Appendix E.5).
  • Representations (YX7M): t-SNE visualizations of FairTFM's embeddings alongside fair-representation baselines (Figures 7 and 12). FairTFM's embeddings separate the classes while making the sensitive groups hard to distinguish, and this effect strengthens as λ\lambda increases.

4. Additional experiments

  • Scaling (e2t7, YX7M): a FairTFM with 12 layers and heads, pretrained for longer, matches XGBoost and RF in accuracy while remaining fairer on all three metrics, so part of FairTFM's accuracy cost is a matter of scale (Appendix E.7).
  • Baselines with access to ss (e2t7): giving every baseline the sensitive attribute as an input leaves accuracy essentially unchanged but makes every baseline less fair, so excluding ss is the conservative choice (Appendix E.7).

We believe these revisions substantially strengthen the paper, and thank the reviewers again for their time and insightful suggestions.


[1] Madras, David, et al. "Learning adversarially fair and transferable representations." ICML, 2018.

[2] Zhang, Brian Hu, Blake Lemoine, and Margaret Mitchell. "Mitigating unwanted biases with adversarial learning." AIES 2018

[3] Kenfack et al. “Towards Fair In-Context Learning with Tabular Foundation Models.” TMLR, 2026

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.