- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a method to train fairer tabular foundation models. Their method is based on light-weight fine-tuning of a query scaling network such that attention is re-weighted to minority groups. Group labels are not used, instead pseudo groups are defined based on clustering of the foundation model embeddings. They show experiments across 5 tabular benchmarks where their method improves worst-group accuracy.
Critical strengths
The paper was relatively easy to follow and clearly motivated. They did extensive comparison to other methods (though I am unclear about when these are actually fair comparisons, see weaknesses), across multiple different datasets, and across multiple types of foundation models. I also appreciated the reporting of computational costs and parameters fine-tuned.
Critical weaknesses
Major weaknesses.
- Comparison to baselines: I am unsure whether fair baselines are compared to (eg see question 1). Am I correct to understand that most of the methods in your tables have not been fine-tuned on your 6 datasets? I don't see the point of having so many. Or in table 5 for example, am I correct to understand that you are comparing zero-shot foundation models to the same models fine-tuned with your method? Surely you should also compare to comparable parameter efficient fine-tuning of those models as well?
- Along those lines, please clarify how your data is split, which splits are used for which models, and respective sizes. For instance, are the standard models trained on the same data that the foundation models are fine-tuned on? I didn't find this information in the main body or the appendix.
- Did not discuss limitations of your work.
- I think it would be nice to also show an analysis of how your identified groups overlap (or do not) with the labelled groups in the dataset. It may help yield insight on whether the groups in the dataset are actually the most meaningful way of splitting the data (see [1]).
- I found the method not very clearly explained (see q3).
Minor weaknesses:
- I found the first section of the related work not very well structured. It reads as a list of methods. I would clarify that you first: explain the importance of tabular foundation models, then discuss how they can be adapted, then discuss how they can be adapted for fairness specifically.
Questions for authors
- Do you compare to a baseline where you do fine-tuning in the same way (i.e. with training context and query scaling) but without the robustness aspect? This seems essential to have a fair comparison.
- Fig 1: What dataset is this result on? Also what is the point of showing both accuracy and loss? They seem to show the exact same trend.
- I don't fully understand what the query represents in your method. Nor the labelled context examples. It is unclear to me what happens during fine-tuning vs. what happens during inference. And where do the group clusters come into play? I don't see them in Fig 2.
- How do you explain that your method does consistently so much better than any other in Table 1?
9 comments
| Method | Params (Trainable %) | Mean | Worst | Seconds | Memory (GB) |
|---|---|---|---|---|---|
| TabPFN-3.0 | |||||
| LoRA | 7,488 (0.014%) | 11.2 | 1.30 | ||
| MLP head | 8,242 (0.016%) | 77.77 | 67.99 | 3.6 | 0.43 |
| Query scaling | 8,320 (0.016%) | 79.31 | 72.57 | ||
| TabPFN-3.5 | |||||
| LoRA | 8,448 (0.004%) | 79.49 | 71.67 | 17.1 | 2.43 |
| MLP head | 8,218 (0.004%) | 79.04 | 71.01 | 7.9 | 1.27 |
| Query scaling | 8,320 (0.004%) | ||||
| EXAONE | |||||
| LoRA | 4,224 (0.020%) | 45.5 | |||
| MLP head | 4,097 (0.020%) | 76.29 | 64.73 | 9.0 | 0.08 |
| Query scaling | 4,192 (0.020%) | 79.49 | 69.65 | ||
| Causilo | |||||
| LoRA | 16,384 (0.045%) | 76.3 | |||
| MLP head | 16,482 (0.046%) | 64.71 | 41.91 | 5.3 | 0.18 |
| Query scaling | 16,576 (0.046%) | 78.47 | 63.56 | ||
| TabFM | |||||
| LoRA | 32,768 (0.002%) | 432.9 | 8.72 | ||
| MLP head | 32,818 (0.002%) | 73.64 | 56.56 | 87.0 | 3.17 |
| Query scaling | 33,088 (0.002%) | 79.43 | 66.47 |
Table 2. Comparison of lightweight adaptation methods. Params denotes trainable parameters. We exclude estimated-group construction time from the reported runtimes, as this procedure is identical across all adaptation methods within each TFM. Bold and underlined values indicate the best and second-best results within each TFM.
Compared with LoRA under comparable trainable parameter budgets, query scaling achieves higher mean-group and worst-group accuracies on four of the five TFMs. Furthermore, query scaling achieves 4–19% shorter runtimes while maintaining comparable or lower GPU memory usage across all five TFMs.
Compared with the MLP head, query scaling achieves higher worst-group accuracy on all five TFMs. For example, on Causilo and TabFM, query scaling achieves 63.56% and 66.47%, compared with 41.91% and 56.56% for the MLP head, respectively. Although the MLP head achieves the lowest runtime and GPU memory usage by bypassing the pretrained decoder, this comes at the cost of lower subgroup robustness.
These comparisons demonstrate that query scaling provides a favorable balance between subgroup robustness and computational efficiency, consistently outperforming the MLP head in worst-group accuracy and achieving competitive or superior performance to LoRA with lower computational costs.
W2: Data Splits, Sample Sizes, and Training Protocols
All methods follow the same benchmark training, validation, and test splits of Tong et al. (2025) [1], as described in Section 5.2.1 and Appendix A.3.2. However, they differ in how the training data are utilized.
Specifically, standard classifiers and robust learning methods use the training set for model training according to their respective objectives. Pretrained TFMs without fine-tuning use as labeled context for in-context prediction. For DR-TFM, we further divide into a context set and a loss-evaluation set in a 70:30 ratio, using the former as labeled context and the latter to optimize the query-scaling network. This procedure is detailed in Section 3 and Appendix A.2.1.
For example, the Adult dataset contains 26,048 training, 6,513 validation, and 16,281 test samples. DR-TFM further divides the 26,048 training samples into approximately 18,234 context samples (70%) and 7,814 loss-evaluation samples (30%). The validation set is used for model selection, and the test set is reserved for final evaluation.
The exact training, validation, and test sample sizes for all datasets are reported in Tables 6 and 7 of Appendix A.3.2. Further details on baseline training procedures and hyperparameter selection are provided in Appendices A.3.6–A.3.7.
W3: Limitations of DR-TFM
We acknowledge that an explicit discussion of limitations was missing from the manuscript.
One limitation of DR-TFM is that it requires fine-tuning for each downstream dataset, which introduces additional computational costs and limits its flexibility in new applications. As future work, we aim to develop more general TFMs that can adapt their predictions to distribution shifts without requiring dataset-specific fine-tuning. We will explicitly discuss this limitation and future research direction in the revised manuscript.
W4: Overlap Between Estimated Groups and Evaluation Groups
We thank the reviewer for highlighting this valuable point. To examine the overlap between estimated groups and predefined evaluation groups, we report their normalized mutual information (NMI), which measures the alignment between these groupings. Higher NMI values indicate greater alignment. Table 3 presents the results for Bank, where NMI is computed within each class and averaged across classes.
| Dataset | Evaluation attribute | NMI |
|---|---|---|
| Bank | Duration s | 0.2560 |
| Bank | Housing loan | 0.0600 |
| Bank | Age | 0.0020 |
| Bank | Married | 0.0010 |
Table 3. Alignment between estimated groups and evaluation attributes on Bank. NMI is computed within each class and averaged across classes.
As shown in Table 3, the estimated groups exhibit varying degrees of alignment with different evaluation attributes, with the highest NMI of 0.2560 for Duration. However, greater alignment with a particular evaluation attribute does not necessarily indicate that the corresponding partition is the most suitable for robust adaptation. As discussed by Alloula et al. (2025) [2], even when evaluation groups are known, alternative group partitions may be more effective for improving robustness across those groups. Thus, identifying the most meaningful subpopulations for robust adaptation remains challenging, particularly without true group annotations.
Therefore, rather than assessing estimated-group quality solely through overlap with predefined evaluation groups, we evaluate their practical effectiveness in achieving subgroup robustness. Following the evaluation protocol of Tong et al. (2025) [1], we assess worst-group accuracy across multiple evaluation attributes and average the results. This allows us to evaluate whether adaptation using estimated groups achieves robust performance across diverse subpopulation partitions, rather than relying on alignment with any single predefined grouping.
W5, Q3: Queries, Labeled Context, Clustering, and Adaptation versus Inference
■ Context and Queries
As described in Section 3 and Appendix A.2.1, we divide the training dataset into a labeled context set (70%) and a loss-evaluation set (30%). The labeled context consists of input–label pairs that the pretrained TFM uses as references for prediction. A query is simply the input to be predicted. During adaptation, queries are drawn from , where their labels are used to compute the training loss. During inference, queries are new test inputs.
■ Clustering and Adaptation
In DR-TFM (GR), we first extract pretrained embeddings from the training data and perform class-conditional clustering to estimate group assignments. These estimated groups serve as proxies for the underlying subpopulations against which we seek robustness. They are used to construct the GroupDRO objective, which places greater emphasis on groups with higher losses. In DR-TFM (MS), we instead construct multiple source subsets from the labeled context and use their predictions to formulate the DRO objective. In both cases, we optimize only the query-scaling network while keeping the pretrained backbone frozen. The estimated groups are used to construct the DRO training objective and are not directly involved in the attention computation shown in Figure 2. Further details are provided in Sections 4.1–4.2 and Appendix A.2.
■ Inference
Once adaptation is complete, the adapted model is used in the same way as a standard pretrained TFM. Given a new test query and the retained labeled context , it directly produces a prediction using the learned query-scaling parameters. No additional clustering, group assignment, or DRO optimization is required during inference.
Figure 2 focuses on the query-scaling mechanism rather than the entire adaptation pipeline. To make this distinction clearer, we will clarify the roles of labeled context, queries, and estimated groups during adaptation and inference in the revised manuscript. We will also add a framework overview flowchart to the supplementary material, illustrating the data preparation, DRO formulations, adaptation, and inference stages.
We thank the reviewer for the thoughtful reading and comments. In response to these comments, we will improve the clarity of the manuscript and incorporate the additional experimental results. Below, we provide a point-by-point response.
■ Comparison between ERM and DRO
We explicitly provide the requested baseline comparison in Table 4 of the manuscript. Both ERM-based and DRO-based query scaling use the same training data, pretrained TabPFN, and query-scaling architecture. Both keep the pretrained backbone frozen and update only the query-scaling network. Their training objectives are standard empirical risk minimization and distributionally robust optimization, respectively. Detailed training protocols are provided in Appendix A.6.
Table 1. Comparison of ERM and DRO using query scaling, extracted from Table 4 of the manuscript (average worst-group accuracy over five benchmarks and three seeds).
As shown above, DRO improves worst-group accuracy by 25.23 percentage points under the same query-scaling architecture. This result highlights the robustness gains of the DRO-based adaptation protocol over standard ERM fine-tuning.
■ Training of Baseline Methods
Our comparisons include both models trained or adapted on the benchmark datasets and pretrained TFMs evaluated through in-context learning without parameter updates. As described in Section 5.2.1, we organize the baselines into three categories: standard classifiers, pretrained TFMs and their adapted variants, and robust learning methods. This broad evaluation allows us to compare DR-TFM against diverse predictive and robust learning approaches, rather than only fine-tuned TFMs.
Specifically, standard classifiers, such as XGBoost and CatBoost, are trained on the benchmark training sets. Pretrained TFMs, such as TabPFN and TabICL, use labeled training examples as context without updating model parameters. We also include fine-tuned or adapted TFMs, such as TabPFN (fine-tuned) and BETA, as well as robust learning methods, such as GEORGE and SPARE, which follow their respective training procedures. For several robust baselines, we use the results reported by Tong et al. (2025) [1] on the same benchmark settings.
The baseline categories and methods are detailed in Appendix A.3.3, with training procedures further documented in Appendices A.3.6–A.3.7 and Table 11.
■ Purpose of Table 5
Yes, Table 5 of the manuscript compares pretrained TFMs without parameter updates against the same models adapted using DR-TFM. However, the pretrained TFMs are not evaluated in a zero-shot setting; both variants use labeled training examples as context for in-context prediction. The purpose of Table 5 is to demonstrate that DR-TFM generalizes beyond TabPFN to other pretrained TFMs. As shown, DR-TFM consistently improves average worst-group accuracy across all four additional TFMs.
■ Comparison with Other PEFT Methods
To further isolate the contribution of query scaling, we compare it with LoRA and an MLP prediction head under comparable trainable parameter budgets across five TFMs. All methods follow the same training and evaluation protocols as DR-TFM (GR). The methods differ only in their adaptation architectures. The results are summarized in Table 2.