acceptodds
Under review as a conference paper at ICLR 2027

TabFSA: Feature-to-Strategy Adaptation for Tabular In-Context Learning

Abstract

Tabular foundation models (TFMs) enable in-context prediction across datasets, but target settings involving temporal transfer, unseen groups, or rare or unseen categorical levels may require targeted adaptation. We study how target data can improve their generalization to new prediction settings without full-model fine-tuning or pretraining a new backbone. We propose Feature-to-Strategy Adaptation (FSA), which uses support–query training tasks representing the target settings to learn a bank of low-rank expert adapters, each specialized to one task family. With the backbone and adapters frozen, controlled comparisons show that favorable adaptation strengths and allocations vary across tasks, and some weighted combinations outperform directly applying any one expert at maximum strength. These observations motivate learning the adaptation strategy from the current task context. FSA constructs a dual-view strategy feature from labeled support examples and unlabeled query inputs, comparing support and query through feature and sample views. We jointly learn this strategy feature and its mapping to adaptation strength and expert allocation by minimizing query prediction loss, enabling task-dependent composition without further parameter optimization on unseen tasks. We instantiate FSA as TabFSA with TabICLv2 and synthetic target training tasks. On BeyondArena, TabFSA raises aggregate Elo from 1178 to 1216 and improves average rank from 4.40 to 3.90. On 200 TALENT classification datasets, it records 136 wins, 15 ties, and 49 losses against the frozen backbone. Retraining FSA on frozen TabPFN also improves prediction, demonstrating applicability across backbones.

Primary area:
foundation or frontier models, including LLMs
Keywords:
tabular foundation models, in-context learning, task-conditioned adaptation, synthetic data

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ADMoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Feature-to-Strategy Adaptation (FSA), a framework for improving tabular foundation models (TFMs) under distribution shifts by dynamically combining specialized LoRA adapters. The method consists of two training stages.

Stage 1: Training specialized LoRA experts. The authors extend TabICLv2's synthetic task generator to construct four specialized task distributions:

1. Extended-IID: Nonlinear relationships, missingness, and class imbalance.
2. Temporal: Covariate shift, label shift, and concept drift.
3. Grouped: Group-specific relationships and generalization to unseen groups.
4. High-cardinality: Rare and unseen categorical levels.

A separate LoRA adapter is trained on each task distribution while keeping the TabICLv2 backbone frozen.

Stage 2: Learning task-dependent adapter composition. The authors introduce a strategy network that predicts the adaptation strength ss and the mixture coefficients π\pi for the four LoRA experts. The architecture works as follows:

1. Extract intermediate feature-level and sample-level representations from the frozen TabICLv2 backbone, using both label-aware and label-free representation paths.
2. Feed these representations into two separate Set Transformer modules, producing a 256-dimensional feature-view representation and a 256-dimensional sample-view representation.
3. Concatenate the two representations into a 512-dimensional dataset-level representation.
4. Feed this representation into a two-hidden-layer MLP to predict the adaptation strength $s$ and expert allocation weights $\pi$.
5. Use the predicted coefficients to construct a weighted combination of the four LoRA updates, which is then applied to the frozen backbone for prediction.

The Set Transformers and strategy MLP are trained jointly by minimizing query NLL on additional synthetic tasks, while the backbone and LoRA experts remain frozen. At inference, the strategy network predicts the adapter coefficients without requiring any test-time optimization.

Experimental results. The authors evaluate TabFSA on BeyondArena and TALENT. On BeyondArena, TabFSA achieves an overall Elo of 1216, compared with 1178 for TabICLv2 and 1212 for TabPFN-2.6. The largest improvements occur on grouped and temporal tasks, while IID performance remains nearly unchanged. On TALENT, TabFSA achieves 136 wins, 15 ties, and 49 losses against TabICLv2 across 200 classification datasets.

The ablation studies demonstrate that task-conditioned adapter composition achieves lower NLL than fixed single-expert selection, uniform LoRA weight averaging, and a learned global mixture. The authors also provide additional experiments using TabPFN-2.5 as the backbone, showing that the framework can be applied beyond TabICLv2.

Critical strengths

S1. An interesting application of dynamic LoRA composition to tabular foundation models.

The idea of dynamically combining specialized LoRA adapters is interesting in the context of tabular foundation models, particularly for handling different types of distribution shifts. Although related approaches have been explored in language models and other domains, adapting this idea to tabular in-context learning introduces additional challenges, including representing variable-sized datasets and capturing support–query distribution differences. The proposed method provides one possible solution to these challenges.

An additional practical advantage is that the method does not require test-time optimization or dataset-specific hyperparameter searches for the adapter coefficients. Once trained, the strategy network directly predicts the adapter composition for a new dataset.

S2. Strong empirical performance on two established tabular benchmarks.

The experimental results are promising, particularly on non-IID datasets. On BeyondArena, TabFSA achieves the highest overall Elo among the evaluated models, with substantial improvements over TabICLv2 on grouped and temporal tasks. On TALENT, the method outperforms TabICLv2 on 136 of 200 classification datasets.

The authors also provide ablation studies showing that the specialized experts learn different behaviors and that task-dependent composition achieves lower NLL than several fixed adapter-composition strategies. These results support the potential usefulness of dynamically combining specialized adapters.

Critical weaknesses

W1. It is unclear whether the improvements come from the proposed architecture or the expanded synthetic training distributions.

This is my primary concern. The authors construct four additional synthetic task distributions covering temporal shifts, grouped observations, high-cardinality categorical features, and extended-IID settings. These additional training distributions could themselves explain a substantial portion of the performance improvements.

However, the paper does not provide a direct comparison against simply continuing to train TabICLv2 on the same expanded synthetic task distribution. It also does not compare against training a single LoRA adapter on the mixture of all four task distributions. These are important baselines because the primary objective is to improve the generalization of existing TFMs to these distribution shifts. It is unclear why the backbone cannot learn these behaviors directly when exposed to the same synthetic training tasks. Without these comparisons, it is difficult to determine whether the performance gains originate from the newly introduced training distributions, the specialized LoRA experts, or the task-dependent routing mechanism.

W2. The complexity and computational overhead of the strategy network are insufficiently justified by the ablation studies.

The proposed strategy network is relatively complicated. It requires an additional TabICLv2 representation-extraction path, two Set Transformer modules, and an MLP to predict only five adaptation coefficients.

This results in approximately 1.9–2.1× inference latency and 9–14% additional peak GPU memory compared with the original TabICLv2. Despite this substantial overhead, the paper does not evaluate simpler routing architectures, such as using raw dataset statistics or pooled representations from the existing backbone to directly predict the adapter coefficients. Such alternatives could potentially achieve comparable performance at significantly lower computational cost. Moreover, the authors only compare different methods of combining LoRA weight updates, without evaluating straightforward prediction-level ensembling of the four specialized experts.

Another important limitation is that all major component and strategy ablations are evaluated using NLL rather than classification accuracy or ROC-AUC. While NLL is a meaningful metric, improvements in NLL can arise from changes in probability calibration without corresponding improvements in classification accuracy or ranking performance. Temperature scaling, for instance, can improve NLL without changing predicted classes or binary ROC-AUC.

The full-model benchmark results demonstrate improvements in accuracy and ROC-AUC-based evaluation, but the ablations do not establish whether the proposed strategy network is responsible for these improvements relative to simpler alternatives. Finally, although the authors provide additional experiments on TabPFN-2.5, these evaluations are limited to smaller selected panels rather than the full benchmark suite. The architecture also has limited flexibility for introducing additional experts, since the strategy network's output dimension depends on the number of experts and would need to be modified and retrained or adapted when new experts are added.

Overall, I find the architectural complexity difficult to justify without stronger comparisons against simpler and more computationally efficient alternatives.

Questions for authors

Q1. How much of the improvement comes from the expanded synthetic training distributions rather than the proposed adaptation architecture?

The paper introduces four additional synthetic task distributions, which could themselves improve the generalization of TabICLv2. Could the authors compare TabFSA against:

  • Continued training or fine-tuning of TabICLv2 on the same mixture of synthetic tasks.
  • A single LoRA adapter trained on the combined synthetic task distribution. Ideally, these comparisons should use comparable training data and computational budgets. This experiment would help determine whether the proposed expert decomposition and routing architecture provide benefits beyond simply improving the backbone's training distribution.

Q2. Is the proposed dual-view strategy network necessary?

The current method uses two TabICLv2 representation paths and two additional Set Transformer modules to construct a dataset-level representation for predicting the adapter coefficients. Could the authors evaluate simpler alternatives, such as:

  • Using statistics computed directly from the raw support/query data, including support labels, as inputs to a lightweight MLP router.
  • Using mean-pooled representations from the existing TabICLv2 backbone, without the additional label-free representation path or Set Transformer modules. It would be particularly useful to compare both predictive performance and inference latency. These experiments would clarify whether the additional representation extraction and Set Transformer modules are necessary to achieve the reported improvements.

Q3. Do the conclusions from the ablation studies hold when evaluated using metrics other than NLL?

Tables 2, 3, and 4 primarily evaluate the different experts and composition strategies using query NLL. However, lower NLL does not necessarily imply better classification accuracy or ROC-AUC. Changes in probability calibration could contribute to the observed improvements. Could the authors report additional evaluation metrics, particularly:

  • Classification accuracy and per-dataset win rates.
  • ROC-AUC for binary classification datasets.
  • Comparisons against temperature-calibrated baselines where appropriate. In particular, it would be valuable to know whether the full strategy network also outperforms fixed experts and simpler composition strategies in terms of accuracy and ROC-AUC, rather than only NLL. This would provide stronger evidence that the proposed routing mechanism improves predictive performance beyond probability calibration.

Q4. How does the method compare against prediction-level ensembling of the four LoRA experts?

The paper compares TabFSA against uniform averaging of LoRA weight updates. However, weight averaging is not equivalent to prediction ensembling because the underlying neural network is nonlinear. Could the authors additionally evaluate averaging the predicted probabilities of the four independently adapted models, or using majority voting? Although prediction ensembling would likely incur additional inference costs, it would provide a useful baseline for determining whether the learned weight-composition mechanism offers advantages over a straightforward ensemble of specialized predictors.

8 comments
AuthorsOpenReview ↗

Response to Reviewer ADMo (Part 1/4)

We thank the reviewer for the detailed feedback. We address each point below.


W1/Q1: Expanded training distributions versus task-conditioned expert reuse

"How much of the improvement comes from the expanded synthetic training distributions rather than the proposed adaptation architecture?"

The expanded distributions alone do not explain the observed Real-40 gains: both mixed-training and fixed-bank controls support task-conditioned expert reuse.

First, we train a single LoRA on the combined mechanism families at ranks 8 and 32, and continue backbone training on that mixture. Here, “single LoRA” means one adapter shared across mixed training tasks, distinct from the per-dataset LoRA baseline in Table 6. The results are reported in Appendix F.2, Table 25 (p. 33):

Table R1. Mixed-training controls on Synthetic-40 and Real-40. NLL columns report relative reductions; accuracy is Real-40 macro accuracy and ROC-AUC covers its 25 binary tasks.

MethodSynthetic NLL reduction (%)Real NLL reduction (%)Real accuracy (%)Real ROC-AUC
TabFSA2.2502.24082.4010.886
Single mixed-task LoRA, rank 80.620-7.14080.8920.853
Single mixed-task LoRA, rank 320.070-7.52080.2530.859
Continued backbone training2.320-11.37079.4820.854

FSA leads all three Real-40 metrics under the reported configurations. Continued backbone training gives a 2.320% Synthetic-40 NLL reduction, but a -11.370% Real-40 reduction. FSA gives 2.240% on Real-40 with 82.401% accuracy and 0.886 binary ROC-AUC. Fitting the synthetic mixture therefore does not by itself establish improved transfer to real tasks.

Second, the fixed-bank comparison keeps the backbone and learned expert directions unchanged and varies only how they are applied. FSA reduces Real-40 NLL by 2.240%, versus 0.080% for a learned global mixture, 1.240% for hard selection, and 1.590% for the best fixed expert (Table 3, p. 8). Since all these controls have the same learned update directions, the difference identifies the benefit of task-conditioned application. Accordingly, the architecture-specific conclusion rests on the fixed-bank comparison, rather than an assumption that the different training procedures have identical total computational budgets.

Together, these controls show why the contribution involves both organizing specialized updates and learning their task-conditioned reuse. Mixed training tests direct absorption of the expanded task families; the fixed-bank comparison isolates the application rule. Table 8 (p. 10) summarizes the added controls.

AuthorsOpenReview ↗

Response to Reviewer ADMo (Part 2/4)

W2/Q2: Why the dual-view strategy is useful: simpler-router comparisons

"The complexity and computational overhead of the strategy network are insufficiently justified by the ablation studies."

"It would be particularly useful to compare both predictive performance and inference latency."

The dual-view task representation gives higher predictive quality than both simpler routers using the same frozen backbone and expert bank. The strategy predicts one strength logit and four expert-allocation logits, but these coefficients must summarize a variable-sized labeled support set and its relation to unlabeled query inputs. The resulting weights also retain a zero-update component. Representing the feature and sample axes before and after cross-feature interaction provides the task context needed to decide both how much to adapt and which experts to use (Sections 3.4-3.5, pp. 5-6).

The statistics router uses raw support-query summaries and support-label information; the mean-pooled router uses pooled backbone support and query row representations and their difference. They test whether simpler summaries recover the full strategy's predictive benefit (Appendix F.1, Table 24, p. 32):

Table R2. Real-40 expert-bank controls: accuracy covers 40 tasks and ROC-AUC covers 25 binary tasks. Relative NLL reduction is measured against frozen TabICLv2.

MethodRel. NLL reduction (%)Accuracy (%)ROC-AUC
TabFSA2.24082.4010.886
Frozen TabICLv20.00082.2120.876
Fixed Extended-IID0.10081.8270.868
Fixed temporal-6.63082.2220.869
Fixed grouped-0.43081.7520.867
Fixed high-cardinality1.59082.2900.873
Statistics router0.50082.2770.876
Mean-pooled router-0.95081.9530.871
Probability mean-0.07082.2860.871

FSA leads all three Real-40 metrics, so the richer representation improves classification decisions and binary ranking as well as NLL. View ablations further support combining both levels: feature-only and sample-only strategies give Real-40 NLL reductions of 1.100% and 1.370%, versus 2.240% for full FSA (Table 4, p. 8). Synthetic-40 shows the same ordering over the simpler routing and aggregation controls:

Table R3. Synthetic-40 routing and aggregation controls, reporting relative NLL reduction.

MethodSynthetic NLL reduction (%)
TabFSA2.250
Statistics router0.640
Mean-pooled router-0.230
Probability mean0.050

The consistent gains on both panels establish the predictive contribution of the dual-view construction; the following cost comparisons quantify what is paid for that benefit.

AuthorsOpenReview ↗

Response to Reviewer ADMo (Part 3/4)

W2/Q2: Inference cost, cross-backbone coverage, and expert expansion (continued)

The predictive benefit is evaluated together with inference latency and peak memory. Matched single-view inference isolates the overhead relative to frozen TabICLv2 (Appendix E.8, Table 21, p. 31):

Table R4. Matched single-view inference with 256 queries on one RTX 3090. Columns report latency in milliseconds and peak CUDA-allocated memory in MB.

SupportFeaturesFrozen msTabFSA msFrozen MBTabFSA MB
1,024830.563.4175.4199.1
1,0243037.977.9275.5303.2
1,02410066.9130.1580.1662.8
4,096853.8102.6300.6328.7
4,0963075.2160.4621.1681.1
4,096100145.0290.11,654.71,869.6

Across these six configurations, FSA uses 1.91-2.13 times the frozen latency and 9.4-14.3% more peak memory. The task-level comparison below includes feature computation, strategy generation, and prediction for both the full and simpler routers:

Table R5. Inference latency and peak CUDA-allocated memory on one RTX 3090, fp32, 256 queries, and one prediction view. Cardiotocography uses 1,544 support rows and Acquire uses 2,048. Latency is the mean of five timed calls after one warm-up; full FSA uses four strategy task views.

MethodCardiotocography latency (ms)Peak memory (GiB)Acquire latency (ms)Peak memory (GiB)
TabFSA172.5000.350459.7001.320
Frozen TabICLv246.9000.28098.7000.960
Statistics router53.0000.280105.8000.960
Mean-pooled router80.6000.280194.7000.960
Probability mean205.1000.280414.9000.960

The simpler routers are faster, but neither matches FSA's Real-40 NLL, accuracy, or binary ROC-AUC. Tables R2 and R5 directly quantify the predictive benefit and its computational cost, answering the architectural-complexity and latency questions together.

"these evaluations are limited to smaller selected panels rather than the full benchmark suite."

The cross-backbone panels test portability across diverse task mechanisms (Appendix E.7, Table 20, p. 30):

Table R6. Cross-backbone transfer on the TabPFN-2.5-supported panels, with 40 synthetic and 40 real tasks. Reductions use frozen TabPFN-2.5 as reference.

MethodSynthetic NLL reduction (%)Real NLL reduction (%)
TabPFN-FSA4.6402.180
Frozen TabPFN-2.50.0000.000

The selected panels provide diverse, complementary evidence for cross-backbone portability. Synthetic-40 tests extended-IID, temporal, grouped, and high-cardinality mechanisms, while Real-40 tests transfer to heterogeneous real classification tasks across IID, grouped, and temporal settings. They therefore examine both controlled task mechanisms and their transfer to real data, within TabPFN-2.5's supported classification domain of at most 10 classes. We instantiate FSA on the frozen backbone and train its adapters and strategy modules, obtaining 4.640% and 2.180% NLL reductions on the synthetic and real panels. This breadth and the consistent gains are sufficient to support the specific conclusion that FSA's adaptation procedure transfers to another frozen backbone. Repeating the complete benchmark with a second backbone would require substantial additional computation; it is not necessary to establish this portability conclusion.

"the strategy network's output dimension depends on the number of experts"

The four current experts cover representative target-task mechanisms: extended-IID, temporal, grouped, and high-cardinality settings. If a new representative direction is needed, FSA can add target training tasks for that mechanism and learn a new LoRA expert while keeping the backbone and existing experts frozen. The strategy generator's allocation head is then expanded and its strategy modules trained for the enlarged bank. Additional training is confined to the new expert and the lightweight strategy modules, avoiding retraining of the backbone and existing expert bank.

Reviewer_7igyOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Feature-to-Strategy Adaptation (FSA), which aims to improve a frozen tabular foundation model on settings such as temporal transfer, unseen groups, and rare or unseen categorical levels.

Stage 1: expert adapters. Four LoRA expert adapters are trained on synthetic support–query episodes. The episodes come from four mechanism families (extended-IID, temporal, grouped, high-cardinality) derived from TabICLv2’s SCM prior. Stage 2: strategy generator. With the backbone and experts frozen, a strategy generator maps a “dual-view strategy feature” to an adaptation strength s and an expert allocation π, which weight the experts’ low-rank updates. The feature combines label-aware support summaries with label-free support–query comparisons at the feature-group and row levels. The generator is trained by query NLL on a mixture of synthetic tasks, so no optimization is performed on unseen tasks. Results. Instantiated on TabICLv2 (TabFSA): BeyondArena Elo rises from 1178 to 1216 on 98 classification datasets, with gains on grouped (+177) and temporal (+126) splits and none on IID (+1). On 200 TALENT datasets, TabFSA records 136/15/49 wins/ties/losses in accuracy against the frozen backbone. Ablations on 40 synthetic and 40 real tasks study strategy types, information sources and training-data composition. A re-instantiation on TabPFN reduces query NLL by 4.6% and 2.2% on the two panels.

Critical strengths

A clean, practical design that targets documented failure modes of tabular foundation models. The backbone stays frozen and adaptation is amortized, with no optimization on unseen tasks. The zero-update component lets the model fall back to the frozen backbone, and all adaptation components are trained on synthetic episodes only. Separating learning update directions from deciding how to apply them is a clear decomposition. The oracle exploration of Fig. 2 is an informative diagnostic of how sensitive the backbone is to the composition. This is consequential because it makes FSA a plug-in for existing foundation models rather than a new backbone.

Critical weaknesses
  • The evidence for real-world gains or generalization is narrow, and the comparisons that would show whether learning the strategy is needed are missing :

    • Only classification tasks are tested
    • Adapter experts are constructed ad-hoc for the Beyond Arena dataset : fours adapters are indexed, the extended-IID, temporal, grouped,and high-cardinality adapters. The +38 aggregate Elo comes from 11 grouped and 10 temporal datasets. On the 77 IID datasets, TabFSA is unchanged (Table 1).
    • The methodology is not compared with other adaptative methods, such as TFM-Retouche (https://arxiv.org/pdf/2605.06047). TFM-Retouche is a Lightweight Input-Space Adapter for Tabular Foundation Models, and is is the top-ranked method on the leaderboard with light per-task tuning and ensembling, lifting aggregate Elo by +56 over the frozen TabICLv2 base. Per-dataset LoRA fine-tuning of TabICLv2 on the support set is the other natural reference.
    • Results on Talent dataset are only a win rate ratio ; having a more fine-grained metrics would be interested. Moreover, The TALENT comparison (Fig. 3) uses older released results, against which TabICLv2 already wins 72–84% of pairwise comparisons. It reports wins in accuracy without effect sizes.
    • Ablation study is not clear for me and should be reported as ELO gain, not NLL.
    • The main text compares TabFSA only with frozen TabICLv2 (Table 1) and with MLP, boosted-tree and conventional families (Fig. 4). Its standing against the other tabular foundation models among the eleven baselines is deferred to the appendix.
  • The methodological contribution is incremental, and the empirical insights that would justify it are not established. It's not clear for exemple why the dual-view strategy is needed. The vectors a_s and a_pi are not defined in the paper (mention in Sec 3.5).

Questions for authors
  • Why ony 98 datasets are tested for Beyond Arena?
  • Is it possible to have results on TabArena datasets? and for regression tasks?
  • Where do a_s and a_pi come from? What is their role?
  • What is the reasons for the dual-view strategy review? Would it be possible to rephrase line 261 ( I don't understand what you mean) ?
7 comments
AuthorsOpenReview ↗

Response to Reviewer 7igy (Part 1/4)

We thank the reviewer for the detailed feedback. We address each point below.


W1/Q1: Classification scope and complete BeyondArena coverage

"Only classification tasks are tested"

"Why ony 98 datasets are tested for Beyond Arena?"

The paper explicitly studies adaptation for tabular classification. This focus follows established TFM development: TabPFN (ICLR 2023) initially studied classification, before the later TabPFN model (Nature 2025) supported classification and regression. Similarly, TabICL (ICML 2025) focused on classification, while TabICLv2 (ICML 2026) extended to both task types.

BeyondArena contains 98 classification datasets: 73 binary and 25 multiclass. We evaluate this complete classification subset. The count reflects the benchmark's classification coverage, rather than a selection from its classification tasks (official dataset card; Appendix D.1, p. 21). Regression is a subsequent extension of the classification adaptation problem studied in this paper.

W2: Mechanism-based experts and the concentration of gains

"Adapter experts are constructed ad-hoc for the Beyond Arena dataset"

"On the 77 IID datasets, TabFSA is unchanged (Table 1)."

The experts are trained on synthetic mechanism families and shared across real evaluation datasets; they are not adapters fitted separately to BeyondArena datasets. Extended-IID, temporal, grouped, and high-cardinality families describe prediction conditions that recur across tables. The generator uses the current task context to determine the strength and allocation of these reusable update directions (Section 3.2, pp. 3-4).

The location of the gains is consistent with this design. FSA provides substantial improvement where the frozen backbone is less robust, while preserving its existing IID capability:

Table R1. BeyondArena split-specific results. Elo is fitted separately within each split.

SplitFrozen EloTabFSA EloFrozen rankTabFSA rank
All117812164.403.90
IID125812593.663.65
Grouped108512626.734.00
Temporal7809067.505.70

Grouped Elo improves by 177 and Temporal Elo by 126; IID improves slightly, from 1258 to 1259, with average rank moving from 3.66 to 3.65 (Table 1, p. 7). The IID gain is modest, as the backbone already handles these conditions well. The contribution is the ability to extend that backbone to more challenging target conditions without sacrificing its IID performance.

Transfer beyond BeyondArena is also demonstrated on the independent TALENT benchmark, where mean accuracy rises from 85.230% to 86.210% across 200 datasets [W4]. This independent transfer supports reuse of the learned mechanisms beyond BeyondArena.

AuthorsOpenReview ↗

Response to Reviewer 7igy (Part 2/4)

W3: Adaptation baselines, optimization cost, and prediction quality

"The methodology is not compared with other adaptative methods, such as TFM-Retouche (https://arxiv.org/pdf/2605.06047)."

"Per-dataset LoRA fine-tuning of TabICLv2 on the support set is the other natural reference."

TFM-Retouche, presented at the ICML 2026 FMSD workshop, optimizes a dataset-specific input-space residual adapter through a frozen TFM, with a validation-based identity guard. The paper does not provide a released open-source implementation, so its author implementation cannot be rerun for a direct, protocol-matched comparison. Its adaptation procedure also differs from FSA: TFM-Retouche optimizes an input adapter for each dataset, while FSA reuses offline-learned parameter updates and predicts their composition. Its published leaderboard gain therefore does not establish which method is better under a common adaptation protocol.

The direct full-fine-tuning and per-dataset LoRA comparisons show that FSA achieves higher binary ROC-AUC and lower multiclass NLL without new-task optimization (Table 6, p. 10):

Table R2. BeyondArena fold-0 target-task adaptation, with optimization stopped when validation NLL converges. Accuracy covers all tasks; ROC-AUC covers 73 binary tasks and NLL covers 25 multiclass tasks.

MethodAccuracy (%)ROC-AUCNLL
TabFSA89.3170.8960.341
Frozen TabICLv285.0570.8830.369
Full fine-tuning89.8200.8740.754
Per-dataset LoRA86.0450.8480.759

TabFSA remains close to full fine-tuning in accuracy (89.317% vs. 89.820%, a gap of 0.503 percentage points), while achieving much lower multiclass NLL (0.341 vs. 0.754) and higher binary ROC-AUC (0.896 vs. 0.874).

Convergence of validation NLL does not guarantee probability quality under train–test distribution shift. Confidently wrong predictions can incur a large NLL penalty without a similarly large accuracy change. The observed NLL gap is consistent with this vulnerability of task-specific optimization under shift. TabFSA achieves the stronger test probability quality without target-task gradient updates.

Both baselines are optimized independently for every dataset until validation NLL converges, with median optimization times of 164.650 s and 102.860 s (Appendix E.8, Table 23, p. 31).

On Cardiotocography, fine-tuning and LoRA optimization take 132.200 s and 92.810 s, with peak memory of 1.050 and 0.373 GiB. On the same RTX 3090, FSA inference for 256 queries takes 172.500 ms and 0.350 GiB (Table 22, p. 31). These stage-specific measurements identify the optimization burden avoided by FSA. Retaining adapted models also requires task-specific weights or adapters for the optimization baselines, whereas FSA reuses a shared expert bank.

TabFSA also leads the evaluated Temporal specialists on all three metrics (Table 7, p. 10):

Table R3. Temporal adaptation: accuracy covers 10 tasks, ROC-AUC covers the 8 binary tasks, and NLL averages finite scores.

MethodAccuracy (%)ROC-AUCNLL
TabFSA86.4370.7950.330
Frozen TabICLv286.0050.7710.360
DistPFN86.0660.7710.372
DistPFN-T85.1560.7710.399
EM correction86.0060.7710.555
Drift-Resilient TabPFN76.5760.7620.480

Together, these results establish the quality and deployment benefits of reusable, task-conditioned adaptation relative to both target-task optimization and the evaluated distribution-shift specialists.

AuthorsOpenReview ↗

Response to Reviewer 7igy (Part 3/4)

W4: TALENT effect sizes and released baselines

"It reports wins in accuracy without effect sizes."

The TALENT improvement has positive mean and median paired effect sizes across 200 datasets (Section 4.1, p. 7):

Table R4. Accuracy and effect sizes on the 200 paired TALENT datasets. Gains are in percentage points over the frozen backbone.

MethodMean accuracy (%)Mean gain (pp)Median gain (pp)
TabFSA86.2100.9800.220
Frozen TabICLv285.230--

TabFSA records 136 wins, 15 ties, and 49 losses. The mean gain quantifies the aggregate improvement, while the positive median gain shows that the paired benefit extends to the middle of the dataset distribution.

The contribution is measured directly against frozen TabICLv2 on the same paired datasets. It therefore quantifies what FSA adds to the backbone, independently of how strongly that backbone already performs against older external baselines. Figure 3 in the same section provides the broader released-baseline context.

W5: Direct prediction metrics for the ablation studies

"Ablation study is not clear for me and should be reported as ELO gain, not NLL."

We report the prediction metrics underlying the benchmark evaluation directly, and additionally report accuracy. BeyondArena's classification Elo is fitted from pairwise outcomes measured by binary ROC-AUC and multiclass log loss (NLL), as specified in the benchmark paper. The controlled studies therefore report ROC-AUC and NLL directly, making the source of the improvement visible. Accuracy additionally tests classification decisions.

FSA leads the fixed-expert, simpler-router, and prediction-averaging controls on all three Real-40 metrics (Appendix F.1, Table 24, p. 32):

Table R5. Real-40 expert-bank controls: accuracy covers 40 tasks and ROC-AUC covers 25 binary tasks. Relative NLL reduction is measured against frozen TabICLv2.

MethodRel. NLL reduction (%)Accuracy (%)ROC-AUC
TabFSA2.24082.4010.886
Frozen TabICLv20.00082.2120.876
Fixed Extended-IID0.10081.8270.868
Fixed temporal-6.63082.2220.869
Fixed grouped-0.43081.7520.867
Fixed high-cardinality1.59082.2900.873
Statistics router0.50082.2770.876
Mean-pooled router-0.95081.9530.871
Probability mean-0.07082.2860.871

The fixed-bank comparison also gives FSA a 2.240% Real-40 NLL reduction, versus 0.080% for the learned global mixture, 1.240% for hard selection, and 1.590% for the best fixed expert (Table 3, p. 8). Holding the expert directions fixed isolates their application rule. The additional accuracy and binary ROC-AUC results show that the benefit extends to classification decisions and ranking, beyond a change in NLL alone. [W7/Q4] separately tests the information sources and views.

W6: Why the main text emphasizes the frozen backbone

"Its standing against the other tabular foundation models among the eleven baselines is deferred to the appendix."

The central claim is to improve the generalization of an existing foundation model. The main-text comparison with frozen TabICLv2 therefore isolates what FSA adds to that backbone (Section 4.1, Table 1, p. 7).

The appendix supplies the broader comparison with the eleven released baselines, including other TFMs (Appendix E.2, Table 11, p. 23), complementing the direct backbone-improvement evidence in the main text.

Reviewer_SfjQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces TabFSA, a method for adapting a pretrained tabular foundation model to a new classification task without updating its parameters at test time. The authors first train four LoRA experts on synthetic tasks representing extended-IID, temporal, grouped, and high-cardinality settings. They then freeze the experts and backbone and train a strategy generator that compares labeled support examples with unlabeled query inputs at both the feature and sample levels.

Critical strengths
  1. The ablations distinguish the value of learning experts from the value of choosing how to use them.

  2. The paper directly tests whether the unlabeled query inputs provide useful information.

Critical weaknesses
  1. The aggregate BeyondArena result obscures where the method helps. On the 77 IID datasets, average rank changes only from 3.66 to 3.65.

  2. The baseline selection does not fully test the paper’s strongest result. BeyondArena provides a useful comparison with general-purpose models in their released default configurations, but not a direct comparison with methods designed for distribution shift. This matters particularly on temporal splits: despite improving over its own backbone, TabFSA ranks seventh by Elo, behind CatBoost and XGBoost, among others. The current experiments therefore establish that TabFSA improves TabICLv2 under these splits, but leave open whether it is competitive with specialized alternatives.

  3. Avoiding test-time optimization is a clear property of the method, but its cost advantage over fine-tuning is not established. Each of the four experts requires 120,000 offline updates, followed by 60,000 updates for strategy learning. Table 18 also reports 1.91–2.13× the inference latency of frozen TabICLv2 under matched single-view inference with 256 query examples. These measurements are useful, but a practitioner still cannot tell when it is cheaper to generate a new strategy for each query batch than to fine-tune once on the support set and reuse the resulting model.

Questions for authors
  1. How does TabFSA compare with methods explicitly designed for distribution shift? On the 11 grouped and 10 temporal datasets, could the authors compare against a tabular test-time adaptation baseline and, on the temporal subset, a drift-specific method such as Drift-Resilient TabPFN, which the paper already cites? Please match access to labeled support data, unlabeled query inputs, and temporal information. Where the backbones differ, reporting both absolute performance and improvement over each method’s own backbone would make the comparison interpretable.

  2. At what workload does TabFSA become preferable to fine-tuning? Could the authors compare TabFSA with both per-dataset LoRA and full fine-tuning of TabICLv2 using the same labeled support set, hardware, and ensemble size? Please report predictive performance alongside one-time offline GPU-hours, per-dataset adaptation time, per-batch inference latency, and peak memory. The most useful result would be the total cost for one query batch versus many batches on the same dataset: at comparable predictive quality, is there a break-even point between repeatedly computing a TabFSA strategy and reusing a fine-tuned model?

4 comments
AuthorsOpenReview ↗

Response to the Reviewer SfjQ(Part 1/2)

We thank the reviewer for the detailed feedback. We address each point below.


W1: Where the BeyondArena gains occur

"The aggregate BeyondArena result obscures where the method helps."

FSA's central contribution is to strengthen a pretrained tabular foundation model across target task conditions while retaining its IID capability. The concentration of gains on Grouped and Temporal tasks directly matches this objective: these settings expose generalization problems that the frozen backbone handles less well, and FSA provides substantial improvements using the same adaptation framework (Section 4.1, Table 1, p. 7).

Table R1. BeyondArena split-specific results. Elo is fitted separately within each split.

SplitFrozen EloTabFSA EloFrozen rankTabFSA rank
All117812164.403.90
IID125812593.663.65
Grouped108512626.734.00
Temporal7809067.505.70

Grouped Elo increases by 177, with average rank improving from 6.73 to 4.00; Temporal Elo increases by 126, with average rank improving from 7.50 to 5.70. On the 77 IID datasets, average rank changes from 3.66 to 3.65. Maintaining this pretrained capability while improving the more challenging target conditions is valuable: the adaptation extends the backbone's useful range without trading away its existing IID performance.

The method links these settings through a shared expert bank. Extended-IID, temporal, grouped, and high-cardinality expert directions are learned offline; with the backbone and experts frozen, the strategy generator uses the new task context to determine their strength and allocation. Thus, the split-specific gains test one task-conditioned reuse mechanism across different prediction conditions.

The benefit also extends to the independent TALENT benchmark, beyond BeyondArena's Grouped and Temporal subsets. Across 200 paired classification datasets, mean accuracy increases from 85.230% to 86.210%, with mean and median gains of 0.980 and 0.220 percentage points and 136 wins, 15 ties, and 49 losses. These results, reported in the same section, establish broader classification transfer alongside targeted non-IID improvement. [W2/Q1] adds direct comparisons with Temporal adaptation methods.

W2/Q1: Specialized distribution-shift baselines and Grouped transfer

"The current experiments therefore establish that TabFSA improves TabICLv2 under these splits, but leave open whether it is competitive with specialized alternatives."

FSA's objective is to improve the generalization of a pretrained TFM across target task types. The Temporal leaderboard compares complete released predictors, including tree-based models; the frozen-backbone comparison isolates the improvement from FSA. The added specialist comparisons further show that TabFSA leads the evaluated Temporal adaptation methods on accuracy, binary ROC-AUC, and NLL (Table 7, p. 10).

Table R2. Temporal adaptation: accuracy covers 10 tasks, ROC-AUC covers the 8 binary tasks, and NLL averages finite scores.

MethodAccuracy (%)ROC-AUCNLL
TabFSA86.4370.7950.330
Frozen TabICLv286.0050.7710.360
DistPFN86.0660.7710.372
DistPFN-T85.1560.7710.399
EM correction86.0060.7710.555
Drift-Resilient TabPFN76.5760.7620.480

DistPFN, DistPFN-T, and EM correction provide posterior-correction controls; Drift-Resilient TabPFN adds a specialist using a different backbone. Frozen TabICLv2 identifies FSA's within-backbone improvement. Positive class-prior rescaling preserves binary rankings, explaining the posterior controls' shared ROC-AUC of 0.771; FSA reaches 0.795 through task-conditioned parameter adaptation (Appendix F.4, p. 34).

For Grouped transfer, to our knowledge, FSA is the first framework to incorporate held-out-group transfer into unified, task-conditioned expert adaptation of frozen TFMs. Together with the Grouped gains, preserved IID capability, and independent TALENT transfer in [W1], the specialist results support adaptation across multiple target task types.

AuthorsOpenReview ↗

Response to the Reviewer SfjQ(Part 2/2)

W3/Q2: Predictive quality, new-task optimization, and deployment workload

"At what workload does TabFSA become preferable to fine-tuning?"

FSA combines improved prediction quality with immediate adaptation to a new dataset, avoiding the optimization that fine-tuning and LoRA repeat for each dataset. We compare all three methods after the optimization baselines reach convergence of validation NLL. FSA achieves higher binary ROC-AUC and lower multiclass NLL than both baselines (Table 6, p. 10):

Table R3. BeyondArena fold-0 target-task adaptation, with optimization stopped when validation NLL converges. Accuracy covers all tasks; ROC-AUC covers 73 binary tasks and NLL covers 25 multiclass tasks.

MethodAccuracy (%)ROC-AUCNLL
TabFSA89.3170.8960.341
Frozen TabICLv285.0570.8830.369
Full fine-tuning89.8200.8740.754
Per-dataset LoRA86.0450.8480.759

TabFSA remains close to full fine-tuning in accuracy (89.317% vs. 89.820%, a gap of 0.503 percentage points), while achieving much lower multiclass NLL (0.341 vs. 0.754) and higher binary ROC-AUC (0.896 vs. 0.874).

Convergence of validation NLL does not guarantee probability quality under train–test distribution shift. Confidently wrong predictions can incur a large NLL penalty without a similarly large accuracy change. The observed NLL gap is consistent with this vulnerability of task-specific optimization under shift. TabFSA achieves the stronger test probability quality without target-task gradient updates.

The per-dataset optimization cost is substantial (Appendix E.8, Table 23, p. 31):

Table R4. Per-dataset optimization cost after convergence of validation NLL on one RTX 3090. Wall time includes intermediate evaluations; GPU-hours are derived from mean time and dataset count.

MethodMedian time (s)Mean time (s)Total GPU-hPeak GiB: median / max
Full fine-tuning164.650370.30310.0801.220 / 15.140
Per-dataset LoRA102.860263.5877.1750.385 / 14.830

These costs recur for each new dataset. If adapted models are retained, full fine-tuning stores task-specific backbone weights and per-dataset LoRA stores task-specific adapters. FSA instead reuses one shared expert bank and obtains its composition from the current task context.

The Cardiotocography measurements separate optimization from prediction (Table 22, p. 31):

Table R5. Cardiotocography stage-specific costs on one RTX 3090. Inference uses 256 queries; optimization stops when validation NLL converges.

MethodStageWall timePeak memory (GiB)
TabFSAInference172.500 ms0.350
Frozen TabICLv2Inference46.900 ms0.280
Full fine-tuningOptimization132.200 s1.050
Per-dataset LoRAOptimization92.810 s0.373

On the same RTX 3090, FSA predicts 256 queries in 172.500 ms using 0.350 GiB. Full fine-tuning and LoRA first require 132.200 s and 92.810 s of optimization, with optimization-stage peak memory of 1.050 and 0.373 GiB. This is the new-task computation avoided by FSA; the table identifies the measured stages explicitly.

For repeated queries, fine-tuning can reuse the optimized model. The Cardiotocography online cost model for FSA versus full fine-tuning gives a crossover at approximately 1,053 batches of 256 queries, or about 270,000 query examples. Below this workload, FSA has lower cumulative online wall time:

Table R6. Calculated online wall time on Cardiotocography, with 256 queries per batch.

Query batchesTabFSA (s)Full fine-tuning, modeled (s)
10.173132.247
10017.250136.890
1,000172.500179.100
1,053181.643181.586

The online cost model (in seconds) is:

TFSA(B)=0.1725BT_{FSA}(B) = 0.1725 B

TFT(B)=132.200+0.0469BT_{FT}(B) = 132.200 + 0.0469 B

Here, BB denotes the number of query batches. The cost inputs are reported in Appendix E.8, Table 22 (p. 31). Shared offline learning is excluded. FSA is particularly advantageous when new datasets require a rapid response or receive a limited number of query batches, while also delivering the strongest binary ranking and multiclass probability quality in this comparison.


Please let us know if there are any remaining issues you'd like to discuss!

Dear Reviewer,

We hope the above clarifications and the additional experiments in the revised draft have sufficiently addressed your concerns. If you are satisfied, we kindly ask you to consider updating your score to reflect the newly added results and discussion. We remain committed to addressing any remaining points you may have during the discussion phase.

Thank you for your time and effort in reviewing our paper!

Dear Reviewer,

We would greatly appreciate your feedback on whether any concerns remain. If our clarifications and additional results have addressed your concerns, we kindly ask you to consider updating your score accordingly.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.