- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Feature-to-Strategy Adaptation (FSA), a framework for improving tabular foundation models (TFMs) under distribution shifts by dynamically combining specialized LoRA adapters. The method consists of two training stages.
Stage 1: Training specialized LoRA experts. The authors extend TabICLv2's synthetic task generator to construct four specialized task distributions:
1. Extended-IID: Nonlinear relationships, missingness, and class imbalance.
2. Temporal: Covariate shift, label shift, and concept drift.
3. Grouped: Group-specific relationships and generalization to unseen groups.
4. High-cardinality: Rare and unseen categorical levels.
A separate LoRA adapter is trained on each task distribution while keeping the TabICLv2 backbone frozen.
Stage 2: Learning task-dependent adapter composition. The authors introduce a strategy network that predicts the adaptation strength and the mixture coefficients for the four LoRA experts. The architecture works as follows:
1. Extract intermediate feature-level and sample-level representations from the frozen TabICLv2 backbone, using both label-aware and label-free representation paths.
2. Feed these representations into two separate Set Transformer modules, producing a 256-dimensional feature-view representation and a 256-dimensional sample-view representation.
3. Concatenate the two representations into a 512-dimensional dataset-level representation.
4. Feed this representation into a two-hidden-layer MLP to predict the adaptation strength $s$ and expert allocation weights $\pi$.
5. Use the predicted coefficients to construct a weighted combination of the four LoRA updates, which is then applied to the frozen backbone for prediction.
The Set Transformers and strategy MLP are trained jointly by minimizing query NLL on additional synthetic tasks, while the backbone and LoRA experts remain frozen. At inference, the strategy network predicts the adapter coefficients without requiring any test-time optimization.
Experimental results. The authors evaluate TabFSA on BeyondArena and TALENT. On BeyondArena, TabFSA achieves an overall Elo of 1216, compared with 1178 for TabICLv2 and 1212 for TabPFN-2.6. The largest improvements occur on grouped and temporal tasks, while IID performance remains nearly unchanged. On TALENT, TabFSA achieves 136 wins, 15 ties, and 49 losses against TabICLv2 across 200 classification datasets.
The ablation studies demonstrate that task-conditioned adapter composition achieves lower NLL than fixed single-expert selection, uniform LoRA weight averaging, and a learned global mixture. The authors also provide additional experiments using TabPFN-2.5 as the backbone, showing that the framework can be applied beyond TabICLv2.
Critical strengths
S1. An interesting application of dynamic LoRA composition to tabular foundation models.
The idea of dynamically combining specialized LoRA adapters is interesting in the context of tabular foundation models, particularly for handling different types of distribution shifts. Although related approaches have been explored in language models and other domains, adapting this idea to tabular in-context learning introduces additional challenges, including representing variable-sized datasets and capturing support–query distribution differences. The proposed method provides one possible solution to these challenges.
An additional practical advantage is that the method does not require test-time optimization or dataset-specific hyperparameter searches for the adapter coefficients. Once trained, the strategy network directly predicts the adapter composition for a new dataset.
S2. Strong empirical performance on two established tabular benchmarks.
The experimental results are promising, particularly on non-IID datasets. On BeyondArena, TabFSA achieves the highest overall Elo among the evaluated models, with substantial improvements over TabICLv2 on grouped and temporal tasks. On TALENT, the method outperforms TabICLv2 on 136 of 200 classification datasets.
The authors also provide ablation studies showing that the specialized experts learn different behaviors and that task-dependent composition achieves lower NLL than several fixed adapter-composition strategies. These results support the potential usefulness of dynamically combining specialized adapters.
Critical weaknesses
W1. It is unclear whether the improvements come from the proposed architecture or the expanded synthetic training distributions.
This is my primary concern. The authors construct four additional synthetic task distributions covering temporal shifts, grouped observations, high-cardinality categorical features, and extended-IID settings. These additional training distributions could themselves explain a substantial portion of the performance improvements.
However, the paper does not provide a direct comparison against simply continuing to train TabICLv2 on the same expanded synthetic task distribution. It also does not compare against training a single LoRA adapter on the mixture of all four task distributions. These are important baselines because the primary objective is to improve the generalization of existing TFMs to these distribution shifts. It is unclear why the backbone cannot learn these behaviors directly when exposed to the same synthetic training tasks. Without these comparisons, it is difficult to determine whether the performance gains originate from the newly introduced training distributions, the specialized LoRA experts, or the task-dependent routing mechanism.
W2. The complexity and computational overhead of the strategy network are insufficiently justified by the ablation studies.
The proposed strategy network is relatively complicated. It requires an additional TabICLv2 representation-extraction path, two Set Transformer modules, and an MLP to predict only five adaptation coefficients.
This results in approximately 1.9–2.1× inference latency and 9–14% additional peak GPU memory compared with the original TabICLv2. Despite this substantial overhead, the paper does not evaluate simpler routing architectures, such as using raw dataset statistics or pooled representations from the existing backbone to directly predict the adapter coefficients. Such alternatives could potentially achieve comparable performance at significantly lower computational cost. Moreover, the authors only compare different methods of combining LoRA weight updates, without evaluating straightforward prediction-level ensembling of the four specialized experts.
Another important limitation is that all major component and strategy ablations are evaluated using NLL rather than classification accuracy or ROC-AUC. While NLL is a meaningful metric, improvements in NLL can arise from changes in probability calibration without corresponding improvements in classification accuracy or ranking performance. Temperature scaling, for instance, can improve NLL without changing predicted classes or binary ROC-AUC.
The full-model benchmark results demonstrate improvements in accuracy and ROC-AUC-based evaluation, but the ablations do not establish whether the proposed strategy network is responsible for these improvements relative to simpler alternatives. Finally, although the authors provide additional experiments on TabPFN-2.5, these evaluations are limited to smaller selected panels rather than the full benchmark suite. The architecture also has limited flexibility for introducing additional experts, since the strategy network's output dimension depends on the number of experts and would need to be modified and retrained or adapted when new experts are added.
Overall, I find the architectural complexity difficult to justify without stronger comparisons against simpler and more computationally efficient alternatives.
Questions for authors
Q1. How much of the improvement comes from the expanded synthetic training distributions rather than the proposed adaptation architecture?
The paper introduces four additional synthetic task distributions, which could themselves improve the generalization of TabICLv2. Could the authors compare TabFSA against:
- Continued training or fine-tuning of TabICLv2 on the same mixture of synthetic tasks.
- A single LoRA adapter trained on the combined synthetic task distribution. Ideally, these comparisons should use comparable training data and computational budgets. This experiment would help determine whether the proposed expert decomposition and routing architecture provide benefits beyond simply improving the backbone's training distribution.
Q2. Is the proposed dual-view strategy network necessary?
The current method uses two TabICLv2 representation paths and two additional Set Transformer modules to construct a dataset-level representation for predicting the adapter coefficients. Could the authors evaluate simpler alternatives, such as:
- Using statistics computed directly from the raw support/query data, including support labels, as inputs to a lightweight MLP router.
- Using mean-pooled representations from the existing TabICLv2 backbone, without the additional label-free representation path or Set Transformer modules. It would be particularly useful to compare both predictive performance and inference latency. These experiments would clarify whether the additional representation extraction and Set Transformer modules are necessary to achieve the reported improvements.
Q3. Do the conclusions from the ablation studies hold when evaluated using metrics other than NLL?
Tables 2, 3, and 4 primarily evaluate the different experts and composition strategies using query NLL. However, lower NLL does not necessarily imply better classification accuracy or ROC-AUC. Changes in probability calibration could contribute to the observed improvements. Could the authors report additional evaluation metrics, particularly:
- Classification accuracy and per-dataset win rates.
- ROC-AUC for binary classification datasets.
- Comparisons against temperature-calibrated baselines where appropriate. In particular, it would be valuable to know whether the full strategy network also outperforms fixed experts and simpler composition strategies in terms of accuracy and ROC-AUC, rather than only NLL. This would provide stronger evidence that the proposed routing mechanism improves predictive performance beyond probability calibration.
Q4. How does the method compare against prediction-level ensembling of the four LoRA experts?
The paper compares TabFSA against uniform averaging of LoRA weight updates. However, weight averaging is not equivalent to prediction ensembling because the underlying neural network is nonlinear. Could the authors additionally evaluate averaging the predicted probabilities of the four independently adapted models, or using majority voting? Although prediction ensembling would likely incur additional inference costs, it would provide a useful baseline for determining whether the learned weight-composition mechanism offers advantages over a straightforward ensemble of specialized predictors.
8 comments
Response to Reviewer ADMo (Part 2/4)
W2/Q2: Why the dual-view strategy is useful: simpler-router comparisons
"The complexity and computational overhead of the strategy network are insufficiently justified by the ablation studies."
"It would be particularly useful to compare both predictive performance and inference latency."
The dual-view task representation gives higher predictive quality than both simpler routers using the same frozen backbone and expert bank. The strategy predicts one strength logit and four expert-allocation logits, but these coefficients must summarize a variable-sized labeled support set and its relation to unlabeled query inputs. The resulting weights also retain a zero-update component. Representing the feature and sample axes before and after cross-feature interaction provides the task context needed to decide both how much to adapt and which experts to use (Sections 3.4-3.5, pp. 5-6).
The statistics router uses raw support-query summaries and support-label information; the mean-pooled router uses pooled backbone support and query row representations and their difference. They test whether simpler summaries recover the full strategy's predictive benefit (Appendix F.1, Table 24, p. 32):
Table R2. Real-40 expert-bank controls: accuracy covers 40 tasks and ROC-AUC covers 25 binary tasks. Relative NLL reduction is measured against frozen TabICLv2.
| Method | Rel. NLL reduction (%) | Accuracy (%) | ROC-AUC |
|---|---|---|---|
| TabFSA | 2.240 | 82.401 | 0.886 |
| Frozen TabICLv2 | 0.000 | 82.212 | 0.876 |
| Fixed Extended-IID | 0.100 | 81.827 | 0.868 |
| Fixed temporal | -6.630 | 82.222 | 0.869 |
| Fixed grouped | -0.430 | 81.752 | 0.867 |
| Fixed high-cardinality | 1.590 | 82.290 | 0.873 |
| Statistics router | 0.500 | 82.277 | 0.876 |
| Mean-pooled router | -0.950 | 81.953 | 0.871 |
| Probability mean | -0.070 | 82.286 | 0.871 |
FSA leads all three Real-40 metrics, so the richer representation improves classification decisions and binary ranking as well as NLL. View ablations further support combining both levels: feature-only and sample-only strategies give Real-40 NLL reductions of 1.100% and 1.370%, versus 2.240% for full FSA (Table 4, p. 8). Synthetic-40 shows the same ordering over the simpler routing and aggregation controls:
Table R3. Synthetic-40 routing and aggregation controls, reporting relative NLL reduction.
| Method | Synthetic NLL reduction (%) |
|---|---|
| TabFSA | 2.250 |
| Statistics router | 0.640 |
| Mean-pooled router | -0.230 |
| Probability mean | 0.050 |
The consistent gains on both panels establish the predictive contribution of the dual-view construction; the following cost comparisons quantify what is paid for that benefit.
Response to Reviewer ADMo (Part 3/4)
W2/Q2: Inference cost, cross-backbone coverage, and expert expansion (continued)
The predictive benefit is evaluated together with inference latency and peak memory. Matched single-view inference isolates the overhead relative to frozen TabICLv2 (Appendix E.8, Table 21, p. 31):
Table R4. Matched single-view inference with 256 queries on one RTX 3090. Columns report latency in milliseconds and peak CUDA-allocated memory in MB.
| Support | Features | Frozen ms | TabFSA ms | Frozen MB | TabFSA MB |
|---|---|---|---|---|---|
| 1,024 | 8 | 30.5 | 63.4 | 175.4 | 199.1 |
| 1,024 | 30 | 37.9 | 77.9 | 275.5 | 303.2 |
| 1,024 | 100 | 66.9 | 130.1 | 580.1 | 662.8 |
| 4,096 | 8 | 53.8 | 102.6 | 300.6 | 328.7 |
| 4,096 | 30 | 75.2 | 160.4 | 621.1 | 681.1 |
| 4,096 | 100 | 145.0 | 290.1 | 1,654.7 | 1,869.6 |
Across these six configurations, FSA uses 1.91-2.13 times the frozen latency and 9.4-14.3% more peak memory. The task-level comparison below includes feature computation, strategy generation, and prediction for both the full and simpler routers:
Table R5. Inference latency and peak CUDA-allocated memory on one RTX 3090, fp32, 256 queries, and one prediction view. Cardiotocography uses 1,544 support rows and Acquire uses 2,048. Latency is the mean of five timed calls after one warm-up; full FSA uses four strategy task views.
| Method | Cardiotocography latency (ms) | Peak memory (GiB) | Acquire latency (ms) | Peak memory (GiB) |
|---|---|---|---|---|
| TabFSA | 172.500 | 0.350 | 459.700 | 1.320 |
| Frozen TabICLv2 | 46.900 | 0.280 | 98.700 | 0.960 |
| Statistics router | 53.000 | 0.280 | 105.800 | 0.960 |
| Mean-pooled router | 80.600 | 0.280 | 194.700 | 0.960 |
| Probability mean | 205.100 | 0.280 | 414.900 | 0.960 |
The simpler routers are faster, but neither matches FSA's Real-40 NLL, accuracy, or binary ROC-AUC. Tables R2 and R5 directly quantify the predictive benefit and its computational cost, answering the architectural-complexity and latency questions together.
"these evaluations are limited to smaller selected panels rather than the full benchmark suite."
The cross-backbone panels test portability across diverse task mechanisms (Appendix E.7, Table 20, p. 30):
Table R6. Cross-backbone transfer on the TabPFN-2.5-supported panels, with 40 synthetic and 40 real tasks. Reductions use frozen TabPFN-2.5 as reference.
| Method | Synthetic NLL reduction (%) | Real NLL reduction (%) |
|---|---|---|
| TabPFN-FSA | 4.640 | 2.180 |
| Frozen TabPFN-2.5 | 0.000 | 0.000 |
The selected panels provide diverse, complementary evidence for cross-backbone portability. Synthetic-40 tests extended-IID, temporal, grouped, and high-cardinality mechanisms, while Real-40 tests transfer to heterogeneous real classification tasks across IID, grouped, and temporal settings. They therefore examine both controlled task mechanisms and their transfer to real data, within TabPFN-2.5's supported classification domain of at most 10 classes. We instantiate FSA on the frozen backbone and train its adapters and strategy modules, obtaining 4.640% and 2.180% NLL reductions on the synthetic and real panels. This breadth and the consistent gains are sufficient to support the specific conclusion that FSA's adaptation procedure transfers to another frozen backbone. Repeating the complete benchmark with a second backbone would require substantial additional computation; it is not necessary to establish this portability conclusion.
"the strategy network's output dimension depends on the number of experts"
The four current experts cover representative target-task mechanisms: extended-IID, temporal, grouped, and high-cardinality settings. If a new representative direction is needed, FSA can add target training tasks for that mechanism and learn a new LoRA expert while keeping the backbone and existing experts frozen. The strategy generator's allocation head is then expanded and its strategy modules trained for the enlarged bank. Additional training is confined to the new expert and the lightweight strategy modules, avoiding retraining of the backbone and existing expert bank.
Response to Reviewer ADMo (Part 1/4)
We thank the reviewer for the detailed feedback. We address each point below.
W1/Q1: Expanded training distributions versus task-conditioned expert reuse
The expanded distributions alone do not explain the observed Real-40 gains: both mixed-training and fixed-bank controls support task-conditioned expert reuse.
First, we train a single LoRA on the combined mechanism families at ranks 8 and 32, and continue backbone training on that mixture. Here, “single LoRA” means one adapter shared across mixed training tasks, distinct from the per-dataset LoRA baseline in Table 6. The results are reported in Appendix F.2, Table 25 (p. 33):
Table R1. Mixed-training controls on Synthetic-40 and Real-40. NLL columns report relative reductions; accuracy is Real-40 macro accuracy and ROC-AUC covers its 25 binary tasks.
FSA leads all three Real-40 metrics under the reported configurations. Continued backbone training gives a 2.320% Synthetic-40 NLL reduction, but a -11.370% Real-40 reduction. FSA gives 2.240% on Real-40 with 82.401% accuracy and 0.886 binary ROC-AUC. Fitting the synthetic mixture therefore does not by itself establish improved transfer to real tasks.
Second, the fixed-bank comparison keeps the backbone and learned expert directions unchanged and varies only how they are applied. FSA reduces Real-40 NLL by 2.240%, versus 0.080% for a learned global mixture, 1.240% for hard selection, and 1.590% for the best fixed expert (Table 3, p. 8). Since all these controls have the same learned update directions, the difference identifies the benefit of task-conditioned application. Accordingly, the architecture-specific conclusion rests on the fixed-bank comparison, rather than an assumption that the different training procedures have identical total computational budgets.
Together, these controls show why the contribution involves both organizing specialized updates and learning their task-conditioned reuse. Mixed training tests direct absorption of the expanded task families; the fixed-bank comparison isolates the application rule. Table 8 (p. 10) summarizes the added controls.