acceptodds
Under review as a conference paper at ICLR 2027

Parameter-Efficient Distributionally Robust Adaptation of Tabular Foundation Models under Subpopulation Shift

Abstract

Despite strong mean accuracy, tabular foundation models (TFMs) can perform poorly on underrepresented groups under subpopulation shift, where group proportions change between training and deployment. We propose DR-TFM, a parameter-efficient distributionally robust adaptation framework that requires no true group annotations. DR-TFM adjusts attention to labeled context examples by fine-tuning an existing query scaling network or adding and training one, while keeping all other parameters fixed. We instantiate the framework with two robust objectives using estimated groups or source conditional distributions derived from training data. For TabPFN-3, adaptation updates only of the pretrained model's parameters. Across five tabular benchmarks, DR-TFM achieves substantially higher average worst-group accuracy than pretrained TFMs and the compared robust baselines without true group annotations, while maintaining competitive mean group accuracy. DR-TFM also improves average worst-group accuracy on ACS Income and across four additional TFMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Tabular foundation models, Distributionally robust optimization, Subpopulation shift, TabPFN, Distribution shift

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_suyoOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method to train fairer tabular foundation models. Their method is based on light-weight fine-tuning of a query scaling network such that attention is re-weighted to minority groups. Group labels are not used, instead pseudo groups are defined based on clustering of the foundation model embeddings. They show experiments across 5 tabular benchmarks where their method improves worst-group accuracy.

Critical strengths

The paper was relatively easy to follow and clearly motivated. They did extensive comparison to other methods (though I am unclear about when these are actually fair comparisons, see weaknesses), across multiple different datasets, and across multiple types of foundation models. I also appreciated the reporting of computational costs and parameters fine-tuned.

Critical weaknesses

Major weaknesses.

  1. Comparison to baselines: I am unsure whether fair baselines are compared to (eg see question 1). Am I correct to understand that most of the methods in your tables have not been fine-tuned on your 6 datasets? I don't see the point of having so many. Or in table 5 for example, am I correct to understand that you are comparing zero-shot foundation models to the same models fine-tuned with your method? Surely you should also compare to comparable parameter efficient fine-tuning of those models as well?
  2. Along those lines, please clarify how your data is split, which splits are used for which models, and respective sizes. For instance, are the standard models trained on the same data that the foundation models are fine-tuned on? I didn't find this information in the main body or the appendix.
  3. Did not discuss limitations of your work.
  4. I think it would be nice to also show an analysis of how your identified groups overlap (or do not) with the labelled groups in the dataset. It may help yield insight on whether the groups in the dataset are actually the most meaningful way of splitting the data (see [1]).
  5. I found the method not very clearly explained (see q3).

Minor weaknesses:

  1. I found the first section of the related work not very well structured. It reads as a list of methods. I would clarify that you first: explain the importance of tabular foundation models, then discuss how they can be adapted, then discuss how they can be adapted for fairness specifically.

[1] https://arxiv.org/abs/2505.21363

Questions for authors
  1. Do you compare to a baseline where you do fine-tuning in the same way (i.e. with training context and query scaling) but without the robustness aspect? This seems essential to have a fair comparison.
  2. Fig 1: What dataset is this result on? Also what is the point of showing both accuracy and loss? They seem to show the exact same trend.
  3. I don't fully understand what the query represents in your method. Nor the labelled context examples. It is unclear to me what happens during fine-tuning vs. what happens during inference. And where do the group clusters come into play? I don't see them in Fig 2.
  4. How do you explain that your method does consistently so much better than any other in Table 1?
9 comments

We thank the reviewer for the thoughtful reading and comments. In response to these comments, we will improve the clarity of the manuscript and incorporate the additional experimental results. Below, we provide a point-by-point response.


W1, Q1: Comparison to Baselines

■ Comparison between ERM and DRO

We explicitly provide the requested baseline comparison in Table 4 of the manuscript. Both ERM-based and DRO-based query scaling use the same training data, pretrained TabPFN, and query-scaling architecture. Both keep the pretrained backbone frozen and update only the query-scaling network. Their training objectives are standard empirical risk minimization and distributionally robust optimization, respectively. Detailed training protocols are provided in Appendix A.6.

AdaptationERM (%)DRO (%)Improvement
Query scaling47.3472.57+25.23 pp

Table 1. Comparison of ERM and DRO using query scaling, extracted from Table 4 of the manuscript (average worst-group accuracy over five benchmarks and three seeds).

As shown above, DRO improves worst-group accuracy by 25.23 percentage points under the same query-scaling architecture. This result highlights the robustness gains of the DRO-based adaptation protocol over standard ERM fine-tuning.

■ Training of Baseline Methods

Our comparisons include both models trained or adapted on the benchmark datasets and pretrained TFMs evaluated through in-context learning without parameter updates. As described in Section 5.2.1, we organize the baselines into three categories: standard classifiers, pretrained TFMs and their adapted variants, and robust learning methods. This broad evaluation allows us to compare DR-TFM against diverse predictive and robust learning approaches, rather than only fine-tuned TFMs.

Specifically, standard classifiers, such as XGBoost and CatBoost, are trained on the benchmark training sets. Pretrained TFMs, such as TabPFN and TabICL, use labeled training examples as context without updating model parameters. We also include fine-tuned or adapted TFMs, such as TabPFN (fine-tuned) and BETA, as well as robust learning methods, such as GEORGE and SPARE, which follow their respective training procedures. For several robust baselines, we use the results reported by Tong et al. (2025) [1] on the same benchmark settings.

The baseline categories and methods are detailed in Appendix A.3.3, with training procedures further documented in Appendices A.3.6–A.3.7 and Table 11.

■ Purpose of Table 5

Yes, Table 5 of the manuscript compares pretrained TFMs without parameter updates against the same models adapted using DR-TFM. However, the pretrained TFMs are not evaluated in a zero-shot setting; both variants use labeled training examples as context for in-context prediction. The purpose of Table 5 is to demonstrate that DR-TFM generalizes beyond TabPFN to other pretrained TFMs. As shown, DR-TFM consistently improves average worst-group accuracy across all four additional TFMs.

■ Comparison with Other PEFT Methods

To further isolate the contribution of query scaling, we compare it with LoRA and an MLP prediction head under comparable trainable parameter budgets across five TFMs. All methods follow the same training and evaluation protocols as DR-TFM (GR). The methods differ only in their adaptation architectures. The results are summarized in Table 2.

MethodParams (Trainable %)Mean ↑\uparrowWorst ↑\uparrowSeconds ↓\downarrowMemory (GB) ↓\downarrow
TabPFN-3.0
LoRA7,488 (0.014%)78.96‾\underline{78.96}71.52‾\underline{71.52}11.21.30
MLP head8,242 (0.016%)77.7767.993.60.43
Query scaling8,320 (0.016%)79.3172.579.1‾\underline{9.1}1.07‾\underline{1.07}
TabPFN-3.5
LoRA8,448 (0.004%)79.4971.6717.12.43
MLP head8,218 (0.004%)79.0471.017.91.27
Query scaling8,320 (0.004%)79.08‾\underline{79.08}71.51‾\underline{71.51}15.3‾\underline{15.3}2.41‾\underline{2.41}
EXAONE
LoRA4,224 (0.020%)78.47‾\underline{78.47}68.87‾\underline{68.87}45.50.14‾\underline{0.14}
MLP head4,097 (0.020%)76.2964.739.00.08
Query scaling4,192 (0.020%)79.4969.6540.3‾\underline{40.3}0.14‾\underline{0.14}
Causilo
LoRA16,384 (0.045%)73.11‾\underline{73.11}60.84‾\underline{60.84}76.30.79‾\underline{0.79}
MLP head16,482 (0.046%)64.7141.915.30.18
Query scaling16,576 (0.046%)78.4763.5665.1‾\underline{65.1}0.79‾\underline{0.79}
TabFM
LoRA32,768 (0.002%)79.41‾\underline{79.41}66.46‾\underline{66.46}432.98.72
MLP head32,818 (0.002%)73.6456.5687.03.17
Query scaling33,088 (0.002%)79.4366.47415.8‾\underline{415.8}8.71‾\underline{8.71}

Table 2. Comparison of lightweight adaptation methods. Params denotes trainable parameters. We exclude estimated-group construction time from the reported runtimes, as this procedure is identical across all adaptation methods within each TFM. Bold and underlined values indicate the best and second-best results within each TFM.

Compared with LoRA under comparable trainable parameter budgets, query scaling achieves higher mean-group and worst-group accuracies on four of the five TFMs. Furthermore, query scaling achieves 4–19% shorter runtimes while maintaining comparable or lower GPU memory usage across all five TFMs.

Compared with the MLP head, query scaling achieves higher worst-group accuracy on all five TFMs. For example, on Causilo and TabFM, query scaling achieves 63.56% and 66.47%, compared with 41.91% and 56.56% for the MLP head, respectively. Although the MLP head achieves the lowest runtime and GPU memory usage by bypassing the pretrained decoder, this comes at the cost of lower subgroup robustness.

These comparisons demonstrate that query scaling provides a favorable balance between subgroup robustness and computational efficiency, consistently outperforming the MLP head in worst-group accuracy and achieving competitive or superior performance to LoRA with lower computational costs.


W2: Data Splits, Sample Sizes, and Training Protocols

All methods follow the same benchmark training, validation, and test splits of Tong et al. (2025) [1], as described in Section 5.2.1 and Appendix A.3.2. However, they differ in how the training data are utilized.

Specifically, standard classifiers and robust learning methods use the training set D\mathcal{D} for model training according to their respective objectives. Pretrained TFMs without fine-tuning use D\mathcal{D} as labeled context for in-context prediction. For DR-TFM, we further divide D\mathcal{D} into a context set D(c)\mathcal{D}^{(c)} and a loss-evaluation set D(e)\mathcal{D}^{(e)} in a 70:30 ratio, using the former as labeled context and the latter to optimize the query-scaling network. This procedure is detailed in Section 3 and Appendix A.2.1.

For example, the Adult dataset contains 26,048 training, 6,513 validation, and 16,281 test samples. DR-TFM further divides the 26,048 training samples into approximately 18,234 context samples (70%) and 7,814 loss-evaluation samples (30%). The validation set is used for model selection, and the test set is reserved for final evaluation.

The exact training, validation, and test sample sizes for all datasets are reported in Tables 6 and 7 of Appendix A.3.2. Further details on baseline training procedures and hyperparameter selection are provided in Appendices A.3.6–A.3.7.

W3: Limitations of DR-TFM

We acknowledge that an explicit discussion of limitations was missing from the manuscript.

One limitation of DR-TFM is that it requires fine-tuning for each downstream dataset, which introduces additional computational costs and limits its flexibility in new applications. As future work, we aim to develop more general TFMs that can adapt their predictions to distribution shifts without requiring dataset-specific fine-tuning. We will explicitly discuss this limitation and future research direction in the revised manuscript.


W4: Overlap Between Estimated Groups and Evaluation Groups

We thank the reviewer for highlighting this valuable point. To examine the overlap between estimated groups and predefined evaluation groups, we report their normalized mutual information (NMI), which measures the alignment between these groupings. Higher NMI values indicate greater alignment. Table 3 presents the results for Bank, where NMI is computed within each class and averaged across classes.

DatasetEvaluation attributeNMI
BankDuration >200>200 s0.2560
BankHousing loan0.0600
BankAge >40>400.0020
BankMarried0.0010

Table 3. Alignment between estimated groups and evaluation attributes on Bank. NMI is computed within each class and averaged across classes.

As shown in Table 3, the estimated groups exhibit varying degrees of alignment with different evaluation attributes, with the highest NMI of 0.2560 for Duration. However, greater alignment with a particular evaluation attribute does not necessarily indicate that the corresponding partition is the most suitable for robust adaptation. As discussed by Alloula et al. (2025) [2], even when evaluation groups are known, alternative group partitions may be more effective for improving robustness across those groups. Thus, identifying the most meaningful subpopulations for robust adaptation remains challenging, particularly without true group annotations.

Therefore, rather than assessing estimated-group quality solely through overlap with predefined evaluation groups, we evaluate their practical effectiveness in achieving subgroup robustness. Following the evaluation protocol of Tong et al. (2025) [1], we assess worst-group accuracy across multiple evaluation attributes and average the results. This allows us to evaluate whether adaptation using estimated groups achieves robust performance across diverse subpopulation partitions, rather than relying on alignment with any single predefined grouping.


W5, Q3: Queries, Labeled Context, Clustering, and Adaptation versus Inference

■ Context and Queries

As described in Section 3 and Appendix A.2.1, we divide the training dataset D\mathcal{D} into a labeled context set D(c)\mathcal{D}^{(c)} (70%) and a loss-evaluation set D(e)\mathcal{D}^{(e)} (30%). The labeled context consists of input–label pairs that the pretrained TFM uses as references for prediction. A query is simply the input to be predicted. During adaptation, queries are drawn from D(e)\mathcal{D}^{(e)}, where their labels are used to compute the training loss. During inference, queries are new test inputs.

■ Clustering and Adaptation

In DR-TFM (GR), we first extract pretrained embeddings from the training data and perform class-conditional clustering to estimate group assignments. These estimated groups serve as proxies for the underlying subpopulations against which we seek robustness. They are used to construct the GroupDRO objective, which places greater emphasis on groups with higher losses. In DR-TFM (MS), we instead construct multiple source subsets from the labeled context and use their predictions to formulate the DRO objective. In both cases, we optimize only the query-scaling network while keeping the pretrained backbone frozen. The estimated groups are used to construct the DRO training objective and are not directly involved in the attention computation shown in Figure 2. Further details are provided in Sections 4.1–4.2 and Appendix A.2.

■ Inference

Once adaptation is complete, the adapted model is used in the same way as a standard pretrained TFM. Given a new test query and the retained labeled context D(c)\mathcal{D}^{(c)}, it directly produces a prediction using the learned query-scaling parameters. No additional clustering, group assignment, or DRO optimization is required during inference.

Figure 2 focuses on the query-scaling mechanism rather than the entire adaptation pipeline. To make this distinction clearer, we will clarify the roles of labeled context, queries, and estimated groups during adaptation and inference in the revised manuscript. We will also add a framework overview flowchart to the supplementary material, illustrating the data preparation, DRO formulations, adaptation, and inference stages.

Reviewer_46bKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DR-TFM, a parameter-efficient framework for adapting tabular foundation models to subpopulation shift without true group annotations. It combines existing GroupDRO or multi-source DRO objectives with a lightweight query-scaling network, while freezing the remaining model parameters. Experiments demonstrate improvements in average worst-group accuracy across several tabular benchmarks and multiple TFMs.

Critical strengths

1.Strong empirical improvements and broad applicability. DR-TFM substantially improves average worst-group accuracy on the five tabular benchmarks. Additional experiments on ACS Income and multiple TFMs support its applicability beyond a single model or dataset. 2.Simple and parameter-efficient adaptation. Updating only 0.016% of TabPFN’s parameters provides an attractive alternative to full-model fine-tuning. The ablations comparing ERM and DRO across adaptation locations help establish the effectiveness of the robust objective.

Critical weaknesses
  1. The robust objectives and class-conditional clustering build on established methods, while query scaling already exists in TabPFN. Table 4 suggests that much of the improvement comes from DRO: query scaling only marginally outperforms decoder fine-tuning, with nearly identical runtime and memory. Comparisons with other lightweight adaptations would help establish the specific value of this design.
  2. Since the framework already estimates groups, a natural baseline would balance or resample context examples using those same estimated groups, without updating model parameters. The comparison with Fair-TabICL using true group labels does not resolve this question. This baseline would clarify whether learned query scaling is necessary beyond correcting context imbalance.
Questions for authors

See Weaknesses.

5 comments

We are grateful to the reviewer for the thoughtful reading and comments. Below, we present additional experiments and analyses to address the reviewer's comments. We will incorporate these additions into the revised manuscript.


W1: Clarification of Contributions and Comparison with Alternative Adaptation Methods

Our contribution is a parameter-efficient framework that integrates pretrained TFM representations with distributionally robust adaptation to address subpopulation shifts without true group annotations. We highlight three key aspects of this contribution: mechanistic interpretability, computational efficiency, and competitive subgroup robustness.

■ Mechanistic Interpretability

Our work views attention adjustment over labeled context examples as a key mechanism for robust adaptation of pretrained TFMs. Motivated by this perspective, we adopt query scaling as a lightweight, targeted adapter that directly adjusts query–key attention scores while preserving pretrained representations. Although different backbone architectures and adaptation methods, including decoder fine-tuning, can achieve reasonable predictive performance, identifying which parameter changes contribute to improved robustness is challenging, particularly in TFMs with complex and tightly coupled architectures. By focusing on query scaling, we provide insight into how attention redistribution relates to subgroup robustness, as illustrated in Figure 3 of the manuscript. Furthermore, applying this attention-based adaptation approach to four additional TFMs beyond TabPFN consistently improves worst-group accuracy, as shown in Table 5 of the manuscript. We believe that these findings provide an attention-based perspective that can inform future research on robust adaptation of pretrained TFMs.

■ Computational Efficiency and Subgroup Robustness

To evaluate the computational efficiency and subgroup robustness of query scaling, we additionally compare it with decoder fine-tuning, LoRA, and an MLP prediction head across five TFMs. LoRA and the MLP head use trainable parameter budgets comparable to query scaling. All methods follow the same training and evaluation protocols as DR-TFM (GR). The methods differ only in their adaptation architectures. To provide a clearer comparison of computational efficiency, we exclude the time required for estimated-group construction, as this procedure is identical across all adaptation methods within each TFM. The results are summarized in Table 1.

MethodParams (Trainable %)Mean ↑\uparrowWorst ↑\uparrowSeconds ↓\downarrowMemory (GB) ↓\downarrow
TabPFN-3.0
Decoder FT427,392 (0.804%)79.03‾\underline{79.03}71.93‾\underline{71.93}9.31.10
MLP head8,242 (0.016%)77.7767.993.60.43
LoRA7,488 (0.014%)78.9671.5211.21.30
Query scaling8,320 (0.016%)79.3172.579.1‾\underline{9.1}1.07‾\underline{1.07}
TabPFN-3.5
Decoder FT5,015,936 (2.291%)79.6670.5620.14.09
MLP head8,218 (0.004%)79.0471.017.91.27
LoRA8,448 (0.004%)79.49‾\underline{79.49}71.6717.12.43
Query scaling8,320 (0.004%)79.0871.51‾\underline{71.51}15.3‾\underline{15.3}2.41‾\underline{2.41}
EXAONE
Decoder FT524,554 (2.521%)78.90‾\underline{78.90}68.99‾\underline{68.99}53.80.15
MLP head4,097 (0.020%)76.2964.739.00.08
LoRA4,224 (0.020%)78.4768.8745.50.14‾\underline{0.14}
Query scaling4,192 (0.020%)79.4969.6540.3‾\underline{40.3}0.14‾\underline{0.14}
Causilo
Decoder FT3,221,866 (8.931%)77.37‾\underline{77.37}62.15‾\underline{62.15}98.90.85
MLP head16,482 (0.046%)64.7141.915.30.18
LoRA16,384 (0.045%)73.1160.8476.30.79‾\underline{0.79}
Query scaling16,576 (0.046%)78.4763.5665.1‾\underline{65.1}0.79‾\underline{0.79}
TabFM
Decoder FT67,187,210 (4.098%)79.1669.37519.910.03
MLP head32,818 (0.002%)73.6456.5687.03.17
LoRA32,768 (0.002%)79.41‾\underline{79.41}66.46432.98.72
Query scaling33,088 (0.002%)79.4366.47‾\underline{66.47}415.8‾\underline{415.8}8.71‾\underline{8.71}

Table 1. Comparison of decoder fine-tuning and lightweight adaptation methods across five TFMs (averages over five benchmarks and three seeds; accuracy in %). The lightweight methods (LoRA, MLP head, and query scaling) use comparable trainable parameter budgets within each TFM. Params denotes trainable parameters. Bold and underlined values indicate the best and second-best results for accuracy, runtime, and memory within each TFM.

(cont.)

Compared with decoder fine-tuning, query scaling achieves higher worst-group accuracy on four of the five TFMs, while using approximately 51–2,030 times fewer trainable parameters. Across the four additional TFMs, it also reduces runtime by 20–34% and GPU memory usage by 7–41%.

Compared with LoRA under comparable trainable parameter budgets, query scaling achieves higher mean-group and worst-group accuracies on four of the five TFMs, with 4–19% shorter runtimes and comparable or lower GPU memory usage.

The MLP head achieves the lowest runtime and memory usage by bypassing the pretrained decoder, but exhibits lower worst-group accuracy than query scaling across all five TFMs. For example, worst-group accuracy decreases from 63.56% to 41.91% on Causilo and from 66.47% to 56.56% on TabFM.


W2: Context Balancing and Resampling Using Estimated Groups

We thank the reviewer for raising this important point. To address this concern, we conducted additional experiments comparing estimated-group-based context resampling strategies with DR-TFM (GR), examining how their performance changes as the severity of within-class subpopulation shift increases.

■ Experimental Setup

We consider a controlled within-class subgroup shift setting. Let ρ(A)=12∑ymax⁡aPtr(A=a∣Y=y)\rho(A)=\frac{1}{2}\sum_y\max_a P^{\mathrm{tr}}(A=a\mid Y=y) denote the within-class majority subgroup proportion for evaluation attribute AA.

We use the original training, validation, and test splits. For each binary attribute AA, we independently reconstruct the training and validation sets by undersampling the four (Y,A)(Y,A) subgroups without replacement. We maintain balanced class proportions and set the subgroup proportions to (ρ/2,(1−ρ)/2,(1−ρ)/2,ρ/2)(\rho/2,(1-\rho)/2,(1-\rho)/2,\rho/2) for (Y,A)=(0,0),(0,1),(1,0),(1,1)(Y,A)=(0,0),(0,1),(1,0),(1,1), respectively, where ρ∈0.5,0.8,0.9\rho\in\\{0.5,0.8,0.9\\}. For example, ρ=0.9\rho=0.9 yields subgroup proportions of (0.45,0.05,0.05,0.45)(0.45,0.05,0.05,0.45). Both training and validation sets follow the same mixture, with their sample sizes fixed across ρ\rho. The original test set remains unchanged. This controlled setting captures an important and widely studied form of subpopulation shift, as exemplified by representative benchmarks such as Waterbirds and CelebA [1].

In the controlled setting, we construct separate training and validation datasets for each attribute AA and shift severity ρ\rho. Each configuration is evaluated only on the four (Y,A)(Y,A) subgroups corresponding to that attribute. Results are averaged across attributes within each dataset, followed by averaging across five datasets and three random seeds.

■ Estimated-Group-Balanced Context Construction

We evaluate three strategies for constructing balanced context sets using the estimated groups obtained by DR-TFM (GR):

  • Undersampling: Randomly reduce all estimated groups to the size of the smallest group.

  • Oversampling: Sample with replacement from smaller groups until each matches the largest group.

  • Hybrid sampling: Compute the average sample count across estimated groups, then undersample larger groups and oversample smaller groups to match this count.

Importantly, the balancing strategies and DR-TFM (GR) use exactly the same fixed estimated-group assignments within each experimental setting. We evaluate these strategies without parameter updates and compare them with DR-TFM (GR) using the original context. The results are summarized in Table 2.

Methodρ=0.5\rho=0.5│ρ=0.8\rho=0.8│ρ=0.9\rho=0.9
Mean ↑\uparrowWorst ↑\uparrow│Mean ↑\uparrowWorst ↑\uparrow│Mean ↑\uparrowWorst ↑\uparrow
TabPFN variants││
Full context80.13 (0.15)73.47 (0.57)│72.91‾\underline{72.91} (0.04)43.93 (0.43)│65.31 (0.10)22.61 (0.48)
Undersampling79.07 (0.27)71.26 (0.26)│72.12 (0.42)44.18 (1.83)│66.49‾\underline{66.49} (0.80)26.97‾\underline{26.97} (2.84)
Oversampling79.59 (0.14)73.07‾\underline{73.07} (0.18)│72.54 (0.21)44.09 (1.39)│65.51 (0.09)23.81 (0.62)
Hybrid79.63‾\underline{79.63} (0.13)72.86 (0.77)│73.01 (0.21)45.68‾\underline{45.68} (0.56)│65.48 (0.33)23.77 (1.62)
DR-TFM (GR)75.89 (0.70)69.30 (0.74)│72.89 (0.57)51.34 (0.84)│70.69 (0.60)41.56 (1.55)

Table 2. Mean-group and worst-group accuracy under varying subgroup shift severity and estimated-group resampling strategies, averaged over five benchmarks (three-seed means; SD in parentheses; accuracy in %). Bold and underlined means indicate the best and second-best results in each column.

(cont.)

As shown in Table 2, estimated-group balancing strategies achieve relatively high worst-group accuracy under mild subgroup imbalance, but their performance deteriorates substantially as the imbalance increases. At ρ=0.9\rho=0.9, their worst-group accuracies range from 23.77% to 26.97%, whereas DR-TFM (GR) achieves 41.56%, outperforming the best balancing strategy by 14.59 percentage points, while also achieving higher mean-group accuracy. Importantly, the balancing strategies and DR-TFM (GR) use the same estimated-group information, demonstrating that learning query scaling through DRO provides substantial additional robustness.**


References.

[1] Sagawa et al., Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization. ICLR, 2020.


We sincerely thank the reviewer once again for the careful comments, and we hope that our responses adequately address the concerns.

Reviewer_EJYQOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the robustness of tabular foundation models under subpopulation shift, where training and test data contain the same groups but in different proportions. To address this challenge, the authors propose Distributionally Robust Tabular Foundation Models (DR-TFM) with two variants: (i) group-based robustness (GR), based on GroupDRO [1], and (ii) multi-source robustness (MS), based on multi-source DRO [2]. Both follow a similar high-level strategy of constructing groups or subsets from training data and optimizing against worst-case distributions. To achieve parameter-efficient adaptation while preserving pretrained representations, DR-TFM employs a lightweight query-scaling MLP to adjust attention weights over labeled training samples. Extensive experiments across multiple subpopulation-shift settings demonstrate substantial improvements in average worst-group accuracy while maintaining competitive mean-group accuracy across different tabular foundation models, including TabPFN-3.

Overall, the paper is well organized and easy to follow. It addresses an important robustness problem with a simple and effective framework, supported by comprehensive empirical evaluations. Although the individual components are largely based on established DRO techniques, their integration provides practical value through parameter-efficient adaptation. However, limited comparisons with established PEFT methods and insufficient analysis of estimated-group quality and scalability leave some questions about the distinctive advantages and limitations of the proposed approach. Considering these strengths and weaknesses, I lean toward Weak Accept in my original review. As the author substantially addressed my concerns during the rebuttal, I increased the score to Clear Accept.

[1] Sagawa et al. (2020). Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization. ICLR 2020.

[2] Kim et al. (2026). Distributionally Robust Classification for Multi-Source Unsupervised Domain Adaptation. ICLR 2026.

Critical strengths

S1. Simple and effective framework. The paper addresses an important robustness challenge for tabular foundation models under subpopulation shift, where strong average predictive performance may conceal poor performance on underrepresented groups. The proposed DR-TFM provides a straightforward approach by combining established distributionally robust optimization (DRO) objectives with lightweight query-scaling for parameter-efficient adaptation. The framework does not require true group annotations and can be applied to different tabular foundation models.

S2. Comprehensive empirical evaluation and computational efficiency. The authors provide extensive experiments across both simulated and real-world settings, with comprehensive comparisons against standard classifiers, robust learning methods, and tabular foundation models. The results demonstrate substantial improvements in average worst-group accuracy while maintaining competitive mean-group accuracy across different tabular foundation model backbones. Furthermore, the computational analysis demonstrates a favorable trade-off between robustness and efficiency, achieving strong performance with substantially fewer trainable parameters and lower computational costs.

S3. Clear writing and organization. The paper is well written and easy to follow, with clear and consistent mathematical notation and equations that support the proposed method.

Critical weaknesses

W1. Limited comparisons with parameter-efficient adaptation methods. Although Tables 3 and 4 demonstrate the computational efficiency of the proposed query-scaling adaptation compared with full fine-tuning and several adaptation strategies, comparisons with established parameter-efficient fine-tuning (PEFT) methods remain limited. In particular, comparisons with methods such as LoRA under the same DRO objectives and comparable parameter budgets would help highlight the contribution of query scaling in terms of robustness and computational efficiency.

W2. Limited analysis of estimated-group quality. The proposed DR-TFM relies on estimated groups or source subsets constructed from pretrained representations rather than true group annotations. However, it remains unclear how closely the estimated groups correspond to the actual subpopulations and how their quality affects worst-group performance. Further analysis or clarification of the relationship between group estimation and robustness would help establish the effectiveness and limitations of the proposed approach.

W3. Limited empirical validation of scalability. Section 4.2 discusses the scalability of query scaling to larger encoders and more attention heads without increasing the number of trainable parameters. However, the experiments do not directly evaluate how computational efficiency and predictive performance change with encoder size or attention-head count. Additional scaling experiments or discussion would help substantiate the claimed scalability advantages.

W4. Minor presentation and organization issues. Although the paper is generally well written, Section 4 could benefit from a clearer overview of the proposed framework. Figure 2 primarily illustrates query scaling without showing how the two DRO objectives interact with the adaptation process. A more comprehensive framework diagram would help clarify this relationship. Additionally, some abbreviations (e.g., GR and MS) are not explicitly defined, and methodological explanations are occasionally mixed with numerical experimental findings, which could be better organized.

Questions for authors

Q1. Could the authors provide comparisons with established PEFT methods, such as LoRA, under the same DRO objectives and comparable parameter budgets to better establish the contribution of query scaling?

Q2. Could the authors further analyze or discuss how estimated-group quality affects worst-group performance, particularly for the GR variant and settings where adaptation degrades performance?

Q3. Could the authors provide additional evidence or discussion on how computational efficiency and predictive performance scale with increasing encoder size or attention-head count?

5 comments

We are grateful to the reviewer for the thoughtful reading and comments. Below, we present additional experiments and analyses to address the reviewer's comments. We will incorporate these additions into the revised manuscript.


W1, Q1: Comparisons with LoRA and Other PEFT Methods

We additionally compare query scaling with LoRA and an MLP prediction head under comparable trainable parameter budgets across five TFMs. All methods follow the same training and evaluation protocols as DR-TFM (GR). The methods differ only in their adaptation architectures. To provide a clearer comparison of computational efficiency, we exclude the time required for estimated-group construction, as this procedure is identical across all adaptation methods within each TFM. The results are summarized in Table 1.

MethodParams (Trainable %)Mean ↑\uparrowWorst ↑\uparrowSeconds ↓\downarrowMemory (GB) ↓\downarrow
TabPFN-3.0
Decoder FT427,392 (0.804%)79.03‾\underline{79.03}71.93‾\underline{71.93}9.31.10
MLP head8,242 (0.016%)77.7767.993.60.43
LoRA7,488 (0.014%)78.9671.5211.21.30
Query scaling8,320 (0.016%)79.3172.579.1‾\underline{9.1}1.07‾\underline{1.07}
TabPFN-3.5
Decoder FT5,015,936 (2.291%)79.6670.5620.14.09
MLP head8,218 (0.004%)79.0471.017.91.27
LoRA8,448 (0.004%)79.49‾\underline{79.49}71.6717.12.43
Query scaling8,320 (0.004%)79.0871.51‾\underline{71.51}15.3‾\underline{15.3}2.41‾\underline{2.41}
EXAONE
Decoder FT524,554 (2.521%)78.90‾\underline{78.90}68.99‾\underline{68.99}53.80.15
MLP head4,097 (0.020%)76.2964.739.00.08
LoRA4,224 (0.020%)78.4768.8745.50.14‾\underline{0.14}
Query scaling4,192 (0.020%)79.4969.6540.3‾\underline{40.3}0.14‾\underline{0.14}
Causilo
Decoder FT3,221,866 (8.931%)77.37‾\underline{77.37}62.15‾\underline{62.15}98.90.85
MLP head16,482 (0.046%)64.7141.915.30.18
LoRA16,384 (0.045%)73.1160.8476.30.79‾\underline{0.79}
Query scaling16,576 (0.046%)78.4763.5665.1‾\underline{65.1}0.79‾\underline{0.79}
TabFM
Decoder FT67,187,210 (4.098%)79.1669.37519.910.03
MLP head32,818 (0.002%)73.6456.5687.03.17
LoRA32,768 (0.002%)79.41‾\underline{79.41}66.46432.98.72
Query scaling33,088 (0.002%)79.4366.47‾\underline{66.47}415.8‾\underline{415.8}8.71‾\underline{8.71}

Table 1. Comparison of decoder fine-tuning and lightweight adaptation methods across five TFMs (averages over five benchmarks and three seeds; accuracy in %). The lightweight methods (LoRA, MLP head, and query scaling) use comparable trainable parameter budgets within each TFM. Params denotes trainable parameters. Bold and underlined values indicate the best and second-best results for accuracy, runtime, and memory within each TFM.

Compared with decoder fine-tuning, query scaling achieves higher worst-group accuracy on four of the five TFMs, while using approximately 51–2,030 times fewer trainable parameters. Across the four additional TFMs, it also reduces runtime by 20–34% and GPU memory usage by 7–41%.

Compared with LoRA under comparable trainable parameter budgets, query scaling achieves higher mean-group and worst-group accuracies on four of the five TFMs, with 4–19% shorter runtimes and comparable or lower GPU memory usage.

The MLP head achieves the lowest runtime and memory usage by bypassing the pretrained decoder, but exhibits lower worst-group accuracy than query scaling across all five TFMs. For example, worst-group accuracy decreases from 63.56% to 41.91% on Causilo and from 66.47% to 56.56% on TabFM.

W2, Q2: Estimated-Group Quality and Worst-Group Performance

We thank the reviewer for highlighting this important point.

■ Analysis of Estimated Groups

To assess estimated-group alignment, we measure within-class normalized mutual information (NMI) between estimated groups and each evaluation attribute, averaged across classes. Table 2 summarizes alignment and changes in worst-cell accuracy.

DatasetEvaluation attributeNMIWorst-cell accuracy (%)Δ\Delta (pp)
BankDuration >200>200 s0.256023.9→71.023.9\to71.0+47.1+47.1
Adultnot never-married / divorced0.101036.7→55.936.7\to55.9+19.2+19.2
MI$\to$AZMale0.012758.9→68.958.9\to68.9+10.0+10.0
MIMale0.013061.5→68.361.5\to68.3+6.8+6.8
MAMale0.001878.2→71.578.2\to71.5−6.7-6.7
MANon-White0.000775.8→70.375.8\to70.3−5.5-5.5
MA$\to$MIMale0.001570.5→63.170.5\to63.1−7.4-7.4
MA$\to$MINon-White0.000773.3→69.773.3\to69.7−3.6-3.6

Table 2. Alignment between estimated groups and evaluation attributes, and changes in worst-cell accuracy after DR-TFM (GR) adaptation. NMI is computed within each class and averaged across classes; arrows indicate accuracy before and after adaptation. Δ\Delta is calculated from the displayed accuracy values.

The results show a clear trend: settings with larger worst-cell accuracy improvements tend to exhibit higher NMI values, whereas settings with performance degradation show very low NMI values. For example, Bank's duration achieves the highest NMI (0.2560) and the largest improvement (+47.1 percentage points), while MA and MA$\to$MI exhibit consistently low NMI values and performance degradation. These findings suggest that the quality of estimated groups plays an important role in achieving subgroup robustness.

Despite its importance, identifying meaningful subpopulations without true group annotations remains challenging, not only for DR-TFM but also for existing group-free robustness methods. Indeed, as shown in Table 16 of the manuscript, all 11 non-TFM robust baselines (12 configurations, including both GEORGE variants) achieve lower worst-group accuracy than ERM-MLP on both MA and MA$\to$MI. These results highlight the broader difficulty of achieving subgroup robustness without true group annotations and motivate further research on reliable subpopulation identification.

■ Failure Analysis: Estimated-Group versus Evaluation-Group Risks

We further examine how estimated-group quality affects the relationship between validation performance and actual test robustness. Since DR-TFM optimizes worst-group risk over estimated groups, improvements in validation performance may not translate into better robustness across actual evaluation groups when group estimation is inaccurate.

Specifically, we define the change in worst-group risk as

ΔRworst=Rworst(fθ^)−Rworst(fθ0),Rworst(f)=max⁡g∈GRg(f).\Delta R_{\mathrm{worst}} = R_{\mathrm{worst}}(f_{\hat{\theta}}) - R_{\mathrm{worst}}(f_{\theta_0}), \qquad R_{\mathrm{worst}}(f)=\max_{g\in\mathcal G}R_g(f).

Here, Rg(f)R_g(f) denotes the average CE loss within group gg, while fθ0f_{\theta_0} and fθ^f_{\hat{\theta}} denote TabPFN before and after adaptation. We compute this quantity using estimated groups on the validation set and actual evaluation groups on the test set. For test evaluation, we compute the worst-group risk separately for each evaluation attribute and average the resulting values. Negative values indicate reductions in worst-group risk.

SettingValidation│Test
ΔRworst\Delta R_{\mathrm{worst}}Δ\Delta Worst-group Acc.│ΔRworst\Delta R_{\mathrm{worst}}Δ\Delta Worst-group Acc.
Bank−0.727-0.727+31.81+31.81│−0.682-0.682+37.26+37.26
Adult−0.354-0.354+16.66+16.66│−0.489-0.489+18.73+18.73
MA−0.134-0.134+8.20+8.20│+0.069+0.069−6.09-6.09
MA$\to$MI−0.108-0.108+2.97+2.97│+0.009+0.009−5.43-5.43

Table 3. Changes in worst-group CE risk and accuracy (Δ\Delta: GR minus TabPFN; three-seed averages; accuracy changes in pp).

On Bank and Adult, where group alignment is relatively high, reductions in estimated-group validation risk are accompanied by lower evaluation-group test risk and substantial improvements in worst-group accuracy. In contrast, MA and MA$\to$MI exhibit very low group alignment, and their estimated-group validation accuracy improves (+8.20 and +2.97 pp) despite declines in test worst-group accuracy (−6.09-6.09 and −5.43-5.43 pp). These results demonstrate that estimated-group validation alone cannot reliably detect failures when estimated groups poorly reflect the underlying subpopulations. This highlights the importance of accurate group estimation for reliable robustness under subpopulation shift.

(cont.)

Before adaptation, pretrained TabPFN already achieves relatively high worst-group accuracy on MA and MA$\to$MI (77.02% and 71.86%, respectively). In such settings, optimizing robustness over estimated groups that are poorly aligned with actual subpopulations can lead to overly conservative adaptation, potentially degrading performance across actual evaluation groups.


W3, Q3: Scalability Analysis

■ Scalability across Pretrained TFM Architectures

Query scaling offers a structural advantage in parameter scalability. In TabPFN, decoder fine-tuning updates the query and key projections and scaling networks. The projection parameters scale as O(dHdh)O(dHd_h), where dd, HH, and dhd_h denote the representation dimension, number of attention heads, and head dimension, respectively. In contrast, query scaling requires only 2mdh+m+dh2md_h+m+d_h trainable parameters, where mm is its hidden dimension. Since this network is shared across attention heads, its parameter count is independent of HH for fixed mm and dhd_h.

Specifically, decoder fine-tuning requires approximately 51 times more trainable parameters than query scaling on TabPFN-3.0 (427,392 vs. 8,320). Importantly, this parameter-efficiency advantage becomes substantially more pronounced when extending query scaling to other pretrained TFMs. Across four additional TFMs with model sizes ranging from 20.3M to 1.57B parameters and 4, 6, or 8 attention heads, the gap reaches approximately 2,030-fold on the larger TabFM. As shown in Table 1, query scaling also achieves higher worst-group accuracy on four of the five TFMs, while reducing runtime by 20–34% and GPU memory usage by 7–41% across the four additional TFMs. These results demonstrate that the structural parameter efficiency of query scaling translates into substantial practical advantages across diverse pretrained TFM architectures.

■ Computational Scalability with Context Size

We further evaluate the computational scalability of query scaling as the context size increases from 2,000 to 50,000 samples. We follow the simulation setting in Section 5.1, keeping all experimental conditions identical across methods except for the model components updated during adaptation.

MethodN=2×103N=2\times10^{3}│N=1×104N=1\times10^{4}│N=5×104N=5\times10^{4}
Seconds ↓\downarrowMemory (GB) ↓\downarrow│Seconds ↓\downarrowMemory (GB) ↓\downarrow│Seconds ↓\downarrowMemory (GB) ↓\downarrow
Full FT130.72.24│349.14.47│4,075.916.26
Encoder FT125.62.22│351.14.42│4,067.016.09
Decoder FT1.70.44│6.20.68│29.51.90
Query scaling1.50.38│5.90.56│27.91.76

Table 4. Training time and GPU memory of TabPFN-3.0 adaptation across context sizes. Bold values indicate the lowest value for each context size.

As shown in Table 4, query scaling consistently achieves the lowest training time and GPU memory usage across all context sizes. Its computational advantage over encoder and full-model fine-tuning becomes particularly pronounced as the context size increases. At N=50,000N=50{,}000, query scaling is approximately 146 times faster than encoder fine-tuning, requiring only 27.9 seconds and 1.76 GB of GPU memory, compared with 4,067.0 seconds and 16.09 GB. Even compared with decoder fine-tuning, query scaling achieves slightly lower training time and GPU memory usage while updating substantially fewer parameters. These results demonstrate that query scaling provides a computationally scalable adaptation strategy for large-context TFMs, maintaining low adaptation costs as context size increases without requiring updates to the pretrained backbone.


W4: Framework Overview, Abbreviations, and Presentation

We thank the reviewer for these helpful comments. We will add a framework overview flowchart to the supplementary material to clarify the overall DR-TFM pipeline, including both DRO formulations and the shared query-scaling adaptation mechanism. We will also explicitly define the GR and MS abbreviations upon their first use and improve the organization of the manuscript by more clearly separating methodological descriptions from experimental findings.


We sincerely thank the reviewer once again for the careful comments, and we hope that our responses adequately address the concerns.

Reviewer_2BHwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DR-TFM addresses worst-group robustness of tabular foundation models (TFMs) under subpopulation shift without true group annotations. It freezes most pretrained parameters and tunes a query-scaling network in decoder attention over labeled context examples: 8,320 parameters (0.016%) in TabPFN-3. The module is added to models that lack it. Adaptation uses either GroupDRO over class-wise clusters of frozen embeddings (GR) or multi-source DRO with clustered or resampled contexts (MS). Model selection uses estimated groups. Across five benchmarks, average worst-group accuracy improves from 47.94% to 72.57% (GR) and 69.31% (MS), with average gains of 4–8 percentage points across the ACS Income within-state and transfer blocks. Further experiments include a simulation, component and cluster-count ablations, true-group comparisons, and four additional TFMs.

Critical strengths
  1. Substantial average gains hold across models without true group labels.

Dataset-averaged worst-group accuracy improves for all five tested TFMs. Neither adaptation nor model selection requires true group labels: the silhouette score determines the cluster count, and worst-group validation accuracy over estimated groups guides early stopping.

  1. Adaptation has low computational overhead.

GR takes about 24 seconds and 1.07 GB of GPU memory per result, compared with 214 seconds and 9.25 GB for full TabPFN fine-tuning. This makes the approach practical if its gains hold under the controls below.

  1. The ablations and simulation provide useful evidence.

Table 4 compares ERM and DRO across five trainable components, showing DRO gains throughout. True-group comparisons and cluster-count ablations probe sensitivity to group construction. In the label-balanced simulation, minority test samples increase attention to their own group (median 0.23 to 0.56; Figure 3), supporting the proposed mechanism beyond class-imbalance correction.

Critical weaknesses
  1. Class balancing and threshold effects remain insufficiently controlled.

Evaluation groups are (attribute, label) cells, so class balancing or threshold adjustment may explain part of the real-data gains, especially on imbalanced datasets such as Shoppers (about 85% negative). Neither necessarily improves both worst-group and mean-group accuracy, and the observed mean-group gains do not resolve this ambiguity. The reported Mean metric averages group accuracies; ordinary sample-average test accuracy is missing.

The class-balanced TabPFN baseline barely improves average worst-group accuracy (48.35% vs. 47.94%), but it fine-tunes the full model and does not provide a matched control for query scaling; whether its model-selection criterion matches DR-TFM’s also needs clarification. DistPFN’s posterior adjustment is not a matched threshold control. Fair-TabICL reaches 72.00% with a group-balanced context, but uses true groups. Changing cluster counts also changes the partition and robust objective, so it does not isolate class balancing.

Matched class-only GroupDRO and validation-selected threshold or logit controls would clarify the added value of inferred subgroups. This concern does not negate the reported gains or the separate evidence from the label-balanced simulation.

  1. Methodological novelty is limited; query scaling’s main advantage over decoder tuning is parameter count.

GR follows GEORGE-style pseudo-group GroupDRO, while MS adapts Kim et al. (2026). The main contribution is the adaptation recipe across TFMs; query scaling already exists in TabPFN-3.

Under DRO, query scaling reaches 72.57% versus 71.93% for decoder tuning, with nearly identical runtime (23.8 vs. 24.0 seconds) and memory use (1.07 vs. 1.10 GB). The roughly 51-fold reduction in trainable parameters is valuable, but the 0.64-point accuracy gap needs paired analysis; averaged per-dataset standard deviations in Table 20 are insufficient. Input-adapter tuning also reaches 66.64% under DRO, showing that the objective helps at other adaptation locations.

For MS, Table 9 shows little sensitivity to the Wasserstein radius on Adult’s estimated-group validation accuracy. This does not demonstrate a benefit from feature perturbations, though it does not rule out benefits elsewhere. The authors also note that the chosen mixture radius admits all convex mixtures of sources.

  1. Efficiency and baseline comparisons involve different configurations.

Table 3 reports 9× and 43× speedups over full-model TabPFN tuning and a 16-member BETA ensemble. BETA also freezes the transformer but trains an input adapter and uses a different context configuration. These results establish efficiency for the tested configurations, but the speedups cannot be attributed to query scaling alone. Table 4 provides single-member input-adapter and decoder-tuning controls, with the latter showing similar costs to query scaling.

Comparisons with non-TFM robust baselines also reflect access to pretrained representations. The same-backbone ablations in Table 4 partly address this, but Appendix A.6 should specify learning rates and schedules for encoder and full-model DRO, and clarify whether ERM and DRO use matched context splits and validation criteria.

  1. Failure cases remain insufficiently explained.

GR reduces worst-group accuracy on ACS MA and MA→MI for all five TFMs. The drops are 6.1 and 5.4 points for TabPFN, and 12.3 and 13.3 points for TabFM (Table 25). These failures do not contradict the abstract’s claim of average improvements.

Appendix A.4 suggests estimated-to-target group mismatch as a cause of MA→MI degradation, supported by recovery with true groups. However, those variants also change training groups, validation groups, and context-split stratification, leaving the cause unresolved. Class-prior overcorrection remains untested. Whether estimated-group validation can detect these failures is also unclear, limiting practical reliability.

Questions for authors
  1. Without true group annotations, could the authors add class-only GroupDRO with the same query-scaling network, TabPFN with a class-balanced context, and frozen TabPFN with threshold or logit adjustment selected using worst-group validation accuracy over estimated groups? Please report overall test accuracy and the worst (attribute, label) cell before and after adaptation. A clear advantage over these controls would resolve my main concern.

  2. Beyond the settings in Table 11 and Appendix A.6, please clarify hyperparameter searches and selection criteria for fine-tuned and class-balanced TabPFN and the ERM/DRO rows in Table 4, particularly encoder and full-model DRO learning rates and schedules. Were context splits and validation criteria matched? How were DR-TFM’s fixed settings chosen without true group annotations or test data?

  3. How well do estimated clusters align with evaluation groups, for example through within-class NMI or ARI against each attribute? Can worst-group validation accuracy over estimated groups detect the ACS MA and MA→MI drops and support fallback to the pretrained model?

8 comments

We thank the reviewer for the thoughtful reading and comments. Below, we provide additional comparisons and clarifications addressing these comments. We will incorporate these additions into the revised manuscript.


W1, Q1: Class-Level Correction and Subgroup Robustness

Our original class-balanced full fine-tuning baseline followed TabPFN's standard procedure using FinetunedTabPFNClassifier, with a class-weighted cross-entropy objective and the default validation ROC-AUC criterion for model selection. Our aim was to evaluate the effect of a class-weighted cross-entropy objective within this standard fine-tuning framework. We agree with the reviewer's point and have updated the model-selection criterion to worst-group validation accuracy over estimated groups, using the same group estimation procedure as DR-TFM (GR). Detailed training configurations are provided in our response to Q2.

To address the reviewer's concern, we expanded our comparisons with class-balanced context selection, logit adjustment, and class-only GroupDRO, alongside the re-evaluated class-balanced full fine-tuning baseline. Class-only GroupDRO uses the same query-scaling architecture as DR-TFM (GR). For the estimated-group variants, including DR-TFM (GR), model selection uses worst-group validation accuracy over estimated groups, without requiring true group annotations.

MethodOverall ↑\uparrowMean ↑\uparrowWorst ↑\uparrowWorst-cell ↑\uparrow│Class-gap ↓\downarrowSubgroup-gap ↓\downarrow
TabPFN variants│
TabPFN86.32 (0.05)73.75 (0.08)47.94 (0.22)39.50 (0.58)│0.788 (0.002)0.172 (0.001)
Class-balanced context80.45 (0.43)79.01 (0.18)69.27 (0.68)61.85 (1.69)│0.054‾\underline{0.054} (0.021)0.169 (0.011)
Class-balanced full FT81.33‾\underline{81.33} (0.99)78.16 (0.11)66.27 (1.73)59.38 (2.09)│0.230 (0.011)0.161‾\underline{0.161} (0.004)
Logit (Estimated-group val.)81.03 (0.03)79.57 (0.07)72.39‾\underline{72.39} (0.11)66.14‾\underline{66.14} (0.21)│0.169 (0.001)0.165 (0.001)
DR-TFM variants│
GR (Class-only)81.24 (0.17)79.51‾\underline{79.51} (0.10)71.75 (0.27)63.05 (0.47)│0.052 (0.003)0.165 (0.004)
GR79.89 (0.28)79.31 (0.17)72.57 (0.64)66.60 (1.15)│0.092 (0.008)0.155 (0.004)

Table 1. Performance and risk gaps averaged over five benchmarks (mean over three seeds, with SD in parentheses; accuracy in %). For binary classification, threshold adjustment is equivalent to additive logit adjustment in terms of predicted labels; hence, only logit adjustment is reported. Bold and underlined means indicate the best and second-best values in each column.

■ Understanding Performance on the Original Benchmarks

Although class-level correction methods achieve comparable worst-group accuracy in Table 1, we find that this is driven by the dominance of class-level risk disparities over within-class subgroup disparities in the original benchmarks. Consequently, correcting class-level disparities can substantially improve worst-group accuracy without meaningfully reducing within-class subgroup risk disparities. To understand this phenomenon, we examine class-level and within-class subgroup risk disparities on the test distribution.

Let Y∈0,1Y\in\\{0,1\\} denote the class label, A\mathcal A the set of binary evaluation attributes, and pf(y∣x)p_f(y\mid x) the class probability predicted by model ff. Each attribute A∈AA\in\mathcal A takes values in 0,1\\{0,1\\}. With PteP^{\mathrm{te}} denoting the test distribution, we define the class-level and subgroup-level cross-entropy (CE) risks as

Ry(f)=EPte![−log⁡pf(y∣X)∣Y=y],Ry,aA(f)=EPte![−log⁡pf(y∣X)∣Y=y,A=a].\begin{aligned} R_y(f) &=\mathbb E_{P^{\mathrm{te}}}\\!\left[-\log p_f(y\mid X)\mid Y=y\right],\\\\ R_{y,a}^{A}(f) &=\mathbb E_{P^{\mathrm{te}}}\\!\left[-\log p_f(y\mid X)\mid Y=y,\\ A=a\right]. \end{aligned}

These expectations are estimated by the mean CE loss within the corresponding test class or subgroup. We then define the risk gaps as

Class-gap⁡(f)=∣R0(f)−R1(f)∣,Subgroup-gap⁡(f)=1∣A∣∑A∈A12∑y∈0,1∣Ry,0A(f)−Ry,1A(f)∣.\begin{aligned} \operatorname{Class\text{-}gap}(f) &=\left|R_0(f)-R_1(f)\right|,\\\\ \operatorname{Subgroup\text{-}gap}(f) &=\frac{1}{|\mathcal A|}\sum_{A\in\mathcal A} \frac{1}{2}\sum_{y\in\\{0,1\\}} \left|R_{y,0}^{A}(f)-R_{y,1}^{A}(f)\right|. \end{aligned}

The class gap measures risk disparity between the two classes, whereas the subgroup gap measures risk disparity between subpopulations within each class, averaged over the evaluated attributes.

(cont.)

Table 1 reports these metrics using each method's predicted probabilities, including the adjusted probabilities for logit adjustment. Notably, for pretrained TabPFN, the class-level risk gap (0.788) dominates the within-class subgroup risk gap (0.172). This indicates that class-level predictive risk disparities are much more pronounced than within-class subgroup disparities on the original benchmarks. Furthermore, logit adjustment reduces the class gap from 0.788 to 0.169, while the subgroup gap remains nearly unchanged (0.172 to 0.165). Nevertheless, worst-group accuracy improves from 47.94% to 72.39%. These results reveal that, contrary to our intentions, the dominant class-level disparities in the original benchmarks allow simple class-level correction to achieve comparable worst-group performance. This limits the ability of the original benchmarks to evaluate robustness to more general subpopulation shifts, including shifts in within-class subgroup proportions. To evaluate robustness beyond class-level correction, we introduce a controlled experimental setting with balanced class proportions and varying within-class subgroup mixtures, while keeping the test distribution fixed.

■ Controlled Evaluation of Within-Class Subgroup Shifts

Let γ=max⁡yPtr(Y=y)\gamma=\max_y P^{\mathrm{tr}}(Y=y) denote the majority class proportion and ρ(A)=12∑ymax⁡aPtr(A=a∣Y=y)\rho(A)=\frac{1}{2}\sum_y\max_a P^{\mathrm{tr}}(A=a\mid Y=y) denote the within-class majority subgroup proportion for attribute AA.

We first use the same training, validation, and test splits as in the original setting. For each evaluated binary attribute AA, we independently reconstruct the training and validation sets by undersampling the four (Y,A)(Y,A) subgroups without replacement. We fix the class proportions at γ=0.5\gamma=0.5 and set the subgroup proportions to (ρ/2,(1−ρ)/2,(1−ρ)/2,ρ/2)(\rho/2,(1-\rho)/2,(1-\rho)/2,\rho/2) for (Y,A)=(0,0),(0,1),(1,0),(1,1)(Y,A)=(0,0),(0,1),(1,0),(1,1), respectively, where ρ∈0.5,0.8,0.9\rho\in\\{0.5,0.8,0.9\\}. For example, ρ=0.9\rho=0.9 yields subgroup proportions of (0.45,0.05,0.05,0.45)(0.45,0.05,0.05,0.45). Both training and validation sets follow the same mixture, with their respective sample sizes held constant across ρ\rho, while the original test set remains unchanged.

Within-class subgroup shifts represent a particularly important and widely studied setting for subpopulation robustness [1, 2, 3]. For example, in the Waterbirds training set, bird type and background are correlated by construction, leaving mismatched combinations at approximately 5% of each class. In the CelebA training set, blond hair is strongly associated with being female, so males constitute only 5.7% of the blond-haired class. These examples motivate evaluating robustness to within-class subgroup mixture shifts beyond class-level shifts.

In the original setting, each method uses a single model per dataset and evaluates its performance across all evaluation attributes. In the controlled setting, we construct a separate training and validation dataset for each attribute AA and shift severity ρ\rho. Each resulting model is evaluated only on the four (Y,A)(Y,A) subgroups associated with that attribute, using the unchanged original test set. Finally, results are averaged across attributes within each dataset, followed by averaging across five datasets and three random seeds.

Methodρ=0.5\rho=0.5│ρ=0.8\rho=0.8│ρ=0.9\rho=0.9
Overall ↑\uparrowWorst ↑\uparrowClass-gap ↓\downarrowSubgroup-gap ↓\downarrow│Overall ↑\uparrowWorst ↑\uparrowClass-gap ↓\downarrowSubgroup-gap ↓\downarrow│Overall ↑\uparrowWorst ↑\uparrowClass-gap ↓\downarrowSubgroup-gap ↓\downarrow
TabPFN variants││
TabPFN (Class-balanced context)80.64 (0.25)73.47 (0.57)0.051‾\underline{0.051} (0.003)0.070 (0.000)│72.71 (0.34)43.93 (0.43)0.268 (0.010)0.846 (0.006)│66.74 (0.14)22.61 (0.48)0.443 (0.010)1.432 (0.014)
Est. group balanced context79.96 (0.50)71.26 (0.26)0.083 (0.010)0.075 (0.002)│73.28 (0.80)44.18 (1.83)0.222‾\underline{0.222} (0.040)0.781‾\underline{0.781} (0.034)│67.75‾\underline{67.75} (0.49)26.97 (2.84)0.391‾\underline{0.391} (0.011)1.221‾\underline{1.221} (0.062)
Class-balanced full FT80.10 (0.24)73.84‾\underline{73.84} (0.30)0.063 (0.009)0.069‾\underline{0.069} (0.003)│73.14 (0.41)45.35‾\underline{45.35} (1.38)0.271 (0.006)0.843 (0.014)│67.07 (0.25)24.37 (0.99)0.442 (0.004)1.426 (0.015)
Logit (Est. group val.)79.38 (1.26)72.78 (1.15)0.182 (0.059)0.070 (0.001)│74.61 (0.74)45.19 (0.89)0.315 (0.023)0.846 (0.006)│67.70 (0.73)27.42‾\underline{27.42} (1.41)0.525 (0.103)1.424 (0.007)
Logit (True-group val.)79.70 (0.26)75.87 (0.29)0.138 (0.010)0.070 (0.000)│74.44 (0.55)47.92 (1.17)0.295 (0.036)0.838 (0.008)│67.35 (0.51)29.82 (0.35)0.468 (0.070)1.412 (0.011)
DR-TFM variants││
GR (Class-only)80.13‾\underline{80.13} (0.18)74.59 (0.27)0.038 (0.003)0.070 (0.001)│72.79 (0.74)44.58 (1.49)0.277 (0.007)0.819 (0.017)│67.29 (0.78)25.82 (1.14)0.426 (0.021)1.334 (0.056)
GR75.48 (0.80)69.30 (0.74)0.092 (0.007)0.064 (0.007)│74.11‾\underline{74.11} (0.82)51.34 (0.84)0.173 (0.015)0.533 (0.033)│72.73 (0.87)41.56 (1.55)0.279 (0.020)0.785 (0.091)
GR (True-group val.)79.59 (0.19)74.89 (0.55)0.071 (0.016)0.067 (0.001)│76.99 (0.54)61.12 (1.84)0.179 (0.026)0.446 (0.046)│74.62 (0.69)53.98 (3.53)0.219 (0.034)0.657 (0.088)

Table 2. Performance and risk gaps under class-balanced subgroup shifts, averaged over five benchmarks (three-seed means; SD in parentheses; accuracy in %). True-group validation uses true annotations only for model selection. Bold/underlined means mark the best/second-best results per column, excluding true-group validation rows.

Table 2 reports Overall accuracy on the original test set, Worst accuracy across the four (Y,A)(Y,A) cells, and the corresponding class and subgroup risk gaps, averaged over attributes and datasets. The key observations from Table 2 are as follows:

  1. Risk Disparities Before Adaptation. As shown in Table 2, the frozen pretrained TabPFN exhibits a subgroup risk gap of 0.070 at ρ=0.5\rho=0.5, which increases substantially to 0.846 and 1.432 at ρ=0.8\rho=0.8 and 0.90.9, respectively. In contrast, the corresponding class gaps are 0.051, 0.268, and 0.443. Thus, subgroup risk gaps dominate class-level risk gaps under stronger shifts.

  2. Overall Trend. As ρ\rho increases from 0.5 to 0.9, the advantages of GR become more pronounced under the controlled within-class subgroup shifts. At ρ=0.9\rho=0.9, the other baselines without true-group validation achieve only 22.61–27.42% worst-group accuracy, whereas GR achieves 41.56%, demonstrating greater robustness to subgroup mixture shifts.

  3. GR vs. Class-Level Correction. At ρ=0.9\rho=0.9, GR improves worst-group accuracy over GR (Class-only) and logit adjustment by 15.74 and 14.14 percentage points, respectively. Since GR and GR (Class-only) share the same query-scaling architecture, this highlights the benefit of subgroup-aware DRO. Moreover, GR maintains a 24.16-point advantage over logit adjustment even with true-group validation (53.98% vs. 29.82%), demonstrating the limitations of class-level correction.

These results demonstrate that class-level correction alone is insufficient to achieve robustness against more general subpopulation shifts, particularly those involving changes in within-class subgroup proportions. In contrast, our DRO adaptation consistently achieves stronger worst-group performance under pronounced subpopulation shifts, demonstrating its advantages beyond class-level correction.

Comments on the paper

6 comments

We thank the reviewer for the thoughtful reading and comments. Below, we provide additional comparisons, analyses, and clarifications. We will incorporate these additions into the revised manuscript.


W1, Q1: Clarification of Contributions and Comparison with Alternative Adaptation Methods

Our method is an integrated framework for robust adaptation of tabular foundation models under subpopulation shift without true group annotations. Specifically, DR-TFM leverages pretrained TFM representations to estimate subpopulations, incorporates distributionally robust objectives based on estimated groups or constructed source subsets, and adapts the model through a lightweight query-scaling network while keeping the remaining parameters fixed. Thus, we do not attribute the performance improvements solely to query scaling, but rather to the integration of these components into a unified framework. We highlight three key aspects of this contribution: mechanistic interpretability, computational efficiency, and competitive subgroup robustness.

■ Mechanistic Interpretability

Our work views attention adjustment over labeled context examples as a key mechanism for robust adaptation of pretrained TFMs. Motivated by this perspective, we adopt query scaling as a lightweight, targeted adapter that directly adjusts query–key attention scores while preserving pretrained representations. Although different backbone architectures and adaptation methods, including decoder fine-tuning, can achieve reasonable predictive performance, identifying which parameter changes contribute to improved robustness is challenging, particularly in TFMs with complex and tightly coupled architectures. By focusing on query scaling, we provide insight into how attention redistribution relates to subgroup robustness, as illustrated in Figure 3 of the manuscript. Furthermore, applying this attention-based adaptation approach to four additional TFMs beyond TabPFN consistently improves worst-group accuracy, as shown in Table 5 of the manuscript. We believe that these findings provide an attention-based perspective that can inform future research on robust adaptation of pretrained TFMs.

■ Computational Efficiency and Subgroup Robustness

To evaluate the computational efficiency and subgroup robustness of query scaling, we additionally compare it with LoRA and an MLP prediction head under comparable trainable parameter budgets across five TFMs. All methods follow the same training and evaluation protocols as DR-TFM (GR). The methods differ only in their adaptation architectures. To provide a clearer comparison of computational efficiency, we exclude the time required for estimated-group construction, as this procedure is identical across all adaptation methods within each TFM. The results are summarized in Table 1.

MethodParams (Trainable %)Mean ↑\uparrowWorst ↑\uparrowSeconds ↓\downarrowMemory (GB) ↓\downarrow
TabPFN-3.0
LoRA7,488 (0.014%)78.96‾\underline{78.96}71.52‾\underline{71.52}11.21.30
MLP head8,242 (0.016%)77.7767.993.60.43
Query scaling8,320 (0.016%)79.3172.579.1‾\underline{9.1}1.07‾\underline{1.07}
TabPFN-3.5
LoRA8,448 (0.004%)79.4971.6717.12.43
MLP head8,218 (0.004%)79.0471.017.91.27
Query scaling8,320 (0.004%)79.08‾\underline{79.08}71.51‾\underline{71.51}15.3‾\underline{15.3}2.41‾\underline{2.41}
EXAONE
LoRA4,224 (0.020%)78.47‾\underline{78.47}68.87‾\underline{68.87}45.50.14‾\underline{0.14}
MLP head4,097 (0.020%)76.2964.739.00.08
Query scaling4,192 (0.020%)79.4969.6540.3‾\underline{40.3}0.14‾\underline{0.14}
Causilo
LoRA16,384 (0.045%)73.11‾\underline{73.11}60.84‾\underline{60.84}76.30.79‾\underline{0.79}
MLP head16,482 (0.046%)64.7141.915.30.18
Query scaling16,576 (0.046%)78.4763.5665.1‾\underline{65.1}0.79‾\underline{0.79}
TabFM
LoRA32,768 (0.002%)79.41‾\underline{79.41}66.46‾\underline{66.46}432.98.72
MLP head32,818 (0.002%)73.6456.5687.03.17
Query scaling33,088 (0.002%)79.4366.47415.8‾\underline{415.8}8.71‾\underline{8.71}

Table 1. Comparison of lightweight adaptation methods under comparable trainable parameter budgets across five TFMs (averages over five benchmarks and three seeds; accuracy in %). Params denotes trainable parameters. Bold and underlined values indicate the best and second-best results for accuracy, runtime, and memory within each TFM.

(cont.)

Compared with LoRA under comparable trainable parameter budgets, query scaling achieves higher mean-group and worst-group accuracies on four of the five TFMs. Furthermore, query scaling achieves 4–19% shorter runtimes while maintaining comparable or lower GPU memory usage across all five TFMs.

Compared with the MLP head, query scaling achieves higher worst-group accuracy on all five TFMs. For example, on Causilo and TabFM, query scaling achieves 63.56% and 66.47%, compared with 41.91% and 56.56% for the MLP head, respectively. Although the MLP head achieves the lowest runtime and GPU memory usage by bypassing the pretrained decoder, this comes at the cost of lower subgroup robustness.

These comparisons demonstrate that query scaling provides a favorable balance between subgroup robustness and computational efficiency, consistently outperforming the MLP head in worst-group accuracy and achieving competitive or superior performance to LoRA with lower computational costs.


W2, Q2: Estimated-Group and Failure-Case Analysis

We thank the reviewer for highlighting this important point.

■ Analysis of Estimated Groups

To assess estimated-group alignment, we measure within-class normalized mutual information (NMI) between estimated groups and each evaluation attribute, averaged across classes. Table 2 summarizes alignment and changes in worst-cell accuracy.

DatasetEvaluation attributeNMIWorst-cell accuracy (%)Δ\Delta (pp)
BankDuration >200>200 s0.256023.9→71.023.9\to71.0+47.1+47.1
Adultnot never-married / divorced0.101036.7→55.936.7\to55.9+19.2+19.2
MI$\to$AZMale0.012758.9→68.958.9\to68.9+10.0+10.0
MIMale0.013061.5→68.361.5\to68.3+6.8+6.8
MAMale0.001878.2→71.578.2\to71.5−6.7-6.7
MANon-White0.000775.8→70.375.8\to70.3−5.5-5.5
MA$\to$MIMale0.001570.5→63.170.5\to63.1−7.4-7.4
MA$\to$MINon-White0.000773.3→69.773.3\to69.7−3.6-3.6

Table 2. Alignment between estimated groups and evaluation attributes, and changes in worst-cell accuracy after DR-TFM (GR) adaptation. NMI is computed within each class and averaged across classes; arrows indicate accuracy before and after adaptation. Δ\Delta is calculated from the displayed accuracy values.

The results show a clear trend: settings with larger worst-cell accuracy improvements tend to exhibit higher NMI values, whereas settings with performance degradation show very low NMI values. For example, Bank's duration achieves the highest NMI (0.2560) and the largest improvement (+47.1 percentage points), while MA and MA$\to$MI exhibit consistently low NMI values and performance degradation. These findings suggest that the quality of estimated groups plays an important role in achieving subgroup robustness.

Despite its importance, identifying meaningful subpopulations without true group annotations remains challenging, not only for DR-TFM but also for existing group-free robustness methods. Indeed, as shown in Table 16 of the manuscript, all 11 non-TFM robust baselines (12 configurations, including both GEORGE variants) achieve lower worst-group accuracy than ERM-MLP on both MA and MA$\to$MI. These results highlight the broader difficulty of achieving subgroup robustness without true group annotations and motivate further research on reliable subpopulation identification.

■ Failure Analysis: Estimated-Group versus Evaluation-Group Risks

We further examine how estimated-group quality affects the relationship between validation performance and actual test robustness. Since DR-TFM optimizes worst-group risk over estimated groups, improvements in validation performance may not translate into better robustness across actual evaluation groups when group estimation is inaccurate.

Specifically, we define the change in worst-group risk as

ΔRworst=Rworst(fθ^)−Rworst(fθ0),Rworst(f)=max⁡g∈GRg(f).\Delta R_{\mathrm{worst}} = R_{\mathrm{worst}}(f_{\hat{\theta}}) - R_{\mathrm{worst}}(f_{\theta_0}), \qquad R_{\mathrm{worst}}(f)=\max_{g\in\mathcal G}R_g(f).

Here, Rg(f)R_g(f) denotes the average CE loss within group gg, while fθ0f_{\theta_0} and fθ^f_{\hat{\theta}} denote TabPFN before and after adaptation. We compute this quantity using estimated groups on the validation set and actual evaluation groups on the test set. For test evaluation, we compute the worst-group risk separately for each evaluation attribute and average the resulting values. Negative values indicate reductions in worst-group risk.

SettingValidation│Test
ΔRworst\Delta R_{\mathrm{worst}}Δ\Delta Worst-group Acc.│ΔRworst\Delta R_{\mathrm{worst}}Δ\Delta Worst-group Acc.
Bank−0.727-0.727+31.81+31.81│−0.682-0.682+37.26+37.26
Adult−0.354-0.354+16.66+16.66│−0.489-0.489+18.73+18.73
MA−0.134-0.134+8.20+8.20│+0.069+0.069−6.09-6.09
MA$\to$MI−0.108-0.108+2.97+2.97│+0.009+0.009−5.43-5.43

Table 3. Changes in worst-group CE risk and accuracy (Δ\Delta: GR minus TabPFN; three-seed averages; accuracy changes in pp).

On Bank and Adult, where group alignment is relatively high, reductions in estimated-group validation risk are accompanied by lower evaluation-group test risk and substantial improvements in worst-group accuracy. In contrast, MA and MA$\to$MI exhibit very low group alignment, and their estimated-group validation accuracy improves (+8.20 and +2.97 pp) despite declines in test worst-group accuracy (−6.09-6.09 and −5.43-5.43 pp). These results demonstrate that estimated-group validation alone cannot reliably detect failures when estimated groups poorly reflect the underlying subpopulations. This highlights the importance of accurate group estimation for reliable robustness under subpopulation shift.

Before adaptation, pretrained TabPFN already achieves relatively high worst-group accuracy on MA and MA$\to$MI (77.02% and 71.86%, respectively). In such settings, optimizing robustness over estimated groups that are poorly aligned with actual subpopulations can lead to overly conservative adaptation, potentially degrading performance across actual evaluation groups.


We sincerely thank the reviewer once again for the careful comments, and we hope that our responses adequately address the concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.