- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces address the problem of fairness in context learning for tabular foundation models. The transitional method fails in the in-context learning setting as we cannot train on a specific task. Given this difficulty, this paper proposes FairTFM. The first step is to random pick a feature as sensitive attribute and convert this into categorical data. The second step is to train the backbone to predict the task label but not predict the selected sensitive attribute. FairTFM is evaluated in 120 fairness tasks from ACS and datapoint are lies in the Pareto frontier of accuracy vs group fairness metrics.
Critical strengths
-
The setting of in-context learning fairness is interesting. A few shot learning with a consideration for fairness is a good setting.
-
The method is easy to understand and the debiasing part make sense.
-
The current results lie on the Pareto frontier.
Critical weaknesses
-
The current debiasing objective function is well developed in the literature. The overall objective function for the network is . This function is identical with the paper in 2018. https://arxiv.org/pdf/1807.00199
-
The current method reduces a lot of performance even compared with LR or RF. Such baselines are vanilla baselines.
-
The random choice of a feature as a sensitive attribute is problematic. For example, the existing fairness issues arise from the correlation between the sensitive attribute and the task label, like occupation bias. However, a randomized selection cannot mimic the correlation between attributes.
Questions for authors
-
I am curious if the sensitive attribute is coincidental chose to the core feature in the pertaining, would a very strong noise being injected to the noise, did any prescreen method is used to prevent the random chose to the core feature for the task.
-
In Figure 1 or Algorithm 1, I am not sure the role of . It seems that gives a few shot examples of the sensitive group protect. I want to make sure I am understanding correctly.
-
It is unclear how the proposed method simultaneously reduces both DP and EOd differences, given their potential trade-offs under unequal group base rates. In Figure 5, Both DP difference and EOD difference almost 0.
-
In lines 419, the method claims the method is a fair representation learning method. I would like to see the TSNE of its representation like Figure 4 in https://arxiv.org/pdf/2211.07907
3 comments
Response to Reviewer YX7M (2/2)
Q1: What if the designated attribute is a core feature? We do not prescreen features, and this is deliberate. When the designated attribute is highly predictive of the label, the task has a strong fairness–accuracy tension, and the objective trades the two off according to λ. Such tasks are realistic: in practice, sensitive attributes are often correlated with the target, which is exactly when fairness interventions are needed and costly. The model is not injected with noise. The attribute is removed from the input features and provided only as a separate token for the context rows, and the adversarial term discourages the representation from encoding it or its proxies.
Q2: The role of s in Figure 1 and Algorithm 1. The reviewer’s understanding is correct. s_ctx is the sensitive attribute of each context example, provided as a separate token (Figure 1b; Algorithm 1). It tells the model which attribute is protected in the current task and which group each context example belongs to, i.e., few-shot examples of the protected groups. The query’s sensitive attribute is never given; it is replaced by a learnable mask token. The model therefore needs sensitive attributes only for the labeled context data, not at prediction time.
Q3: How can DP and EOD both be reduced under unequal base rates? When base rates differ across groups, exact DP and exact equalized odds cannot both hold, except for trivial predictors such as a constant one. Reducing both, however, is possible. Unconstrained models carry excess disparity from proxies of the sensitive attribute, and removing that information lowers both gaps without making either exactly zero. This is what we observe at moderate λ: FairTFM-0.7 reduces DP and EOD by 36% and 32% relative to TabPFNv2.5, without driving either to zero (Table 2).
The points in Figure 5 where both are close to zero come from FairTFM-25 at early pretraining checkpoints. These predict very few positives on this task, consistent with the trivial-predictor case. Figure 5 did not show which λ each point uses. We now indicate this in the figure caption, and report group-wise positive prediction rates against a constant predictor in Appendix E.5.
Q4: t-SNE of the representations. We thank the reviewer for the suggestion. We added t-SNE visualizations of FairTFM’s representations alongside the fair-representation baselines (CR, LFR, LAFTR): Figure 7 for gender, and Figure 12 for race and age. Coloring by sensitive attribute and by label shows that FairTFM’s embeddings separate the classes while making the sensitive groups hard to distinguish, and this effect strengthens as λ increases. This complements the linear probes in Table 1, where FairTFM-25 reduces sensitive-attribute AUC from 0.783 to 0.542 while keeping the label predictable.
We hope these clarifications and additions address the reviewer’s concerns and would greatly appreciate it if they would increase their score accordingly.
Follow-up on Our Response
Dear Reviewer YX7M,
As the discussion period is coming to an end, we wanted to kindly check whether our response has addressed your concerns. In particular:
- (W1) We clarified our contribution relative to existing adversarial objectives, and added a comparison with task-specific adversarial methods (Appendix E.6).
- (W2) We showed that FairTFM-0.7 matches or exceeds LR on all metrics (Table 2), and added a scaling ablation in which a larger FairTFM matches RF and XGB in accuracy while remaining fairer (Appendix E.7).
- (W3, Q1) We explained why randomly designated sensitive attributes inherit realistic correlations with the label and its proxies.
- (Q3) We explained how DP and EOD can both be reduced, and clarified which FairTFM checkpoints appear in Figure 5.
- (Q4) We added the t-SNE visualizations you requested (Figures 7 and 12).
We would be grateful for any further feedback, and are happy to clarify anything that remains unclear.
Thank you again for your time and constructive review.
Response to Reviewer YX7M (1/2)
We thank the reviewer for the constructive feedback, and for positively highlighting that “the setting of in-context learning fairness is interesting”, that the method is “easy to understand”, and that our results “lie on the Pareto frontier”. We address each point below; changes in the revised paper are shown in blue.
W1: The adversarial objective already exists. We agree, and do not claim a new debiasing objective. Our contribution is making this kind of objective work during pretraining of a tabular foundation model (TFM), so that fairness transfers to unseen tasks in a single forward pass without task-specific training. This raised problems that standard adversarial training does not face:
The cited work, like other task-specific adversarial methods, trains a new model for each task. We compare FairTFM against two such methods, LAFTR and Adversarial Debiasing, in the new Appendix E.6 (Figures 15 and 16). FairTFM provides competitive Pareto points without any task-specific training.
W2: Accuracy relative to LR and RF. Over the 120 tasks (Table 2, Appendix E.1):
Part of the remaining gap is due to model scale and pretraining budget, not the fairness objective. In the new scaling ablation (Table 4, Appendix E.7), a FairTFM-0.1 with 12 layers and 12 attention heads, pretrained for longer, reaches 0.745 accuracy and 0.778 AUCROC. This matches RF (0.744 and 0.772) while being fairer on all three metrics (e.g., DP 0.102 vs. 0.113).
W3: Can a randomly chosen attribute mimic real bias? If we understand the concern correctly, it is that a randomly designated attribute may not reproduce the attribute–label correlations behind real biases. Randomly chosen features in our pretraining data are not independent of the label. The TabICL prior generates datasets from structural causal models, in which features are causally connected to each other and to the label. A designated sensitive attribute therefore inherits whatever correlation the generating model induces, both with the label and with other features that act as proxies; the attribute itself is removed from the feature set, so only its proxies remain (Section 3.1, Figure 1a). Because the designated feature varies across thousands of tasks, pretraining covers a wide range of attribute–label correlations, from little to very strong. Real biases, such as occupation bias, are one point in this range.
The evaluation supports this: FairTFM never sees a real sensitive attribute during pretraining, yet reduces disparities for real gender, race and age attributes across 120 ACS tasks (Figure 2) and 12 tasks from six other benchmarks (Figure 3).