- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
In this study, the authors examine the effects of adaptation, aggregation and context construction in recent tabular foundation model.
For adaptation, the paper introduces DiagScale, which shows that only a magnitude scaling of query-key values, we can reduce the parameter updates to lower than 0.03% compared to full fine-tuning. Next, for aggregation, the authors combine built-in ensembles generating V many views with K many different configurations. To combine these different views stemming from different configurations, the authors investigate three reducer functions: a uniform aggregation, a greedy selection heuristic that adds candidates that minimized the validation loss, or an best-single approach for selecting the candidate with the lowest val loss. Lastly, the study examines context construction, where column curation is framed for importance-based selection rules , whereas row selection is based on query-to-context attention for selecting relevant rows. In the last set of experiments, the authors investigate the effects of combining adaptation with aggregation.
Critical strengths
S1: Thorough analysis of test-time compute strategies
The paper systematically investigates the effects of three components that come into play in test-time computation for TFMs, adaptation, aggregation and context construction. The study reveals which one of them actually provide a lift for the predictive performance in the tested models, and therefore, isolates the individual design choices.
S2: DiagScale as parameter-efficient adaptation strategy
The proposed DiagScale heuristic provides an important contribution. In the study provided, it is highlighted that adapting only a small number of parameters is sufficient to achieve a performance level that is comparable to a full fine-tuning approach.
Critical weaknesses
W1: Limited explanation
Even though appreciating the extensive experimental study and the empirical evidence provided by the authors, we see which strategies could improve performance under which specific experimental setting, but at the end of the day we have limited insight into why specific strategies work/fail. So what is the recommended default configuration now where the expectation is to get good performance. Also in terms of OOD settings? For example the DiagScale heuristic seems promising, but what is the property that makes these corrections provided by the magnitude scaling sufficient?
W2: Meta-feature analysis
Thinking a bit of W1, I’d be curious to know from the authors if they have likewise systematically investigate whether there are some meta-feature constellations within the dataset where we can expect performance improvements? So, do we find some feature redudancies that might help, the intrinisic dimensionality of datasets, the authors slightly touched the class imbalance problem etc. So for the three controlled mechanics discussed (adaptation, aggregation, context construct), can we identify specific conditions that need to be met such that a benefit in perfroamance can be expected?
W3: Coverage of TFMs
In the paper, the aggregation and context construction are concentrated around the TabPFN-3 model. Therefore, the take-aways discussed here should be taken with caution if we understand the manuscript as a general investigation of TFMs. The adaptation experiments include three different backbones. The experiments could be extended to other model families to have stronger evidence that some of the identified patterns are ones that go beyond TabPFN-3.
W4: Practical advantage
In Figure 5, it seems like that a stronger backbone beats any add-ons for other competitors at lower costs. Speaking from a practical point of view, is the computational overhead then better be allocated to any of the two heuristics included here (adapt., agg), selecting a stronger backbone, or performing more intense hyperparamter optimization that is currently not in the scope of the draft. Speaking of HPO, by any chance that the authors conducted hpo to the tested settings to potentially get the best performance out of each config? From my understanding, the diagscale provides a light-weight heuristic that is also considerably low in the computational overhead, however, preferring a stronger backbone instead of leveraging the aggregation heuristic due to its expensive computational costs. But maybe the authors can elaborate a bit on their practical recommendations.
W5: Some insights are of lower novelty
The study is comprehensive in total, however, some findings also cover some insights that are established also in the respective original works. Like in inspection of larger candidate pools with effective selection strategies, the diminishing returns from increasing the ensemble size, advantages of additional training samples, etc.. So, some insights can be considered as known, however, the manuscript validates them again for TFMs. I’d have been more surprised by any novelty that specifically is valid for TFMs, that we would not expect according to the knowledge stemming from standard literature about ensemble learning or tabular machine learning in general. Maybe a paragraph of what is it that stands out from the insights that is only valid for TFMs and kind of ‘unexpected’ behavior.
Questions for authors
- Q1. Even though some deeper investigations of model-data interactions are postponed to future work, can the authors provide insights into their explanations for any lifts we can see when integrating/applying the heuristics that are examined in isolation in the proposed study? What is the expected behavior for OOD settings? (W1)
- Q2. Did the authors also inspect more thoroughly the meta features of the underlying datasets. Can we see patterns where some of the heuristics are expected to be more useful compared to settings where no benefit is seen. (W2)
- Q3: Can the authors give an explanation or provide a discussion why some settings are framed for TabPFN and why it is expected that these insights also also for other tfm families. (W3)
- Q4: Can the authors provide a discussion on the practical advantage and how computational power should be allocated? Also, did the authors conduct a more in-depth inspection of hyperparameter optimization? (W4)
- Q5. (Minor). From the authors’ experience and insights they gained from their analysis. What is the most unexpected behavior that crushed their expectation when conducting the experiments. Besides the DiagScale insights that provide some interesting reduction of parameters while keeping the performance levels, what are the crucial insights they gained for TFMs? (W5)
6 comments
Reply to Reviewer DJwj: Q2 and W2
Q2 & W2: Can dataset meta features predict when the methods improve performance?
Thank you for suggesting this analysis. We found that greater disagreement among native views is associated with larger aggregation gains, but the meta features we examined did not reliably predict gain magnitudes on new datasets. In Appendix E.3, we examine whether dataset characteristics and model behavior can help anticipate adaptation and aggregation gains. We consider 21 predictors, including dataset size, feature count, class imbalance, missingness, cardinality, feature correlation, univariate signal concentration, and disagreement across the frozen model's predictions.
The clearest pattern is that greater disagreement among TabPFN-3's native views is associated with larger configuration-aggregation gains (Spearman , globally adjusted ). Together with our fixed-pool comparison of averaging and greedy selection, this suggests that disagreement provides opportunities for improvement, while selection determines how effectively those differences are used. We did not find an equally robust association for adaptation after correcting for multiple comparisons.
We also tested whether these meta features jointly predict per-dataset gains. We used the meta features as inputs to ridge regression and gradient-boosted trees, predicting relative error reductions from DiagScale and full fine-tuning on each of the three backbones, plus 96-configuration greedy aggregation on TabPFN-3. We evaluated predictions using five-fold cross-validation over datasets, repeated 20 times. For each repeat, we computed , where is the observed gain for dataset , is its held-out prediction, and is the mean observed gain across datasets. Of the 14 prediction tasks (two regression models seven gain outcomes), 13 yield negative mean , and the remaining one is only 0.012. Thus, these meta features did not reliably predict gain magnitudes on new datasets.
For context construction, our experiments suggest examining how selection changes the context, in addition to the original dataset's characteristics. In the COMET_MC failure case, retrieval substantially increases the rare-class proportion relative to the source pool. This points to changes in context composition as a useful diagnostic to investigate, rather than treating the original class imbalance alone as a predictor of retrieval benefit.
Reply to Reviewer DJwj: Q3 and W3
Q3 & W3: Why some studies focus on TabPFN and why their insights are expected to generalize to other TFM families.
We concentrated the detailed aggregation and context-construction studies on TabPFN-3 to keep the backbone fixed while varying the intervention. We expect these insights to extend to other TFMs because they share the relevant mechanisms: aggregation combines predictions that differ in quality and errors, while context construction changes the examples and features used for in-context learning. Prediction diversity, member selection, and context choice should therefore matter beyond TabPFN-3. We have now extended both comparisons to other model families.
For aggregation, we evaluated 96 configurations on TabICL v2 across all 51 TabArena datasets. Greedy selection reduces error by 1.93%, compared with 0.75% for uniform averaging, averaged across datasets. The larger gain from greedy selection supports the value of selecting ensemble members beyond TabPFN-3. Unlike on TabPFN-3, uniform averaging also improves predictions here, showing that the benefit of averaging the full pool varies across backbones.
For context construction, we evaluated column selection on TabICL v2 and Kumo Large using the same seven wide tables and experimental settings as the TabPFN-3 study. We compare importance-based selection with each model's baseline without external column selection and with random selection of the same number of columns. The table reports relative error changes; negative is better.
| Dataset | TabICL v2: vs. baseline | TabICL v2: vs. random | Kumo Large: vs. baseline | Kumo Large: vs. random |
|---|---|---|---|---|
| Bioresponse | -11.48% | -9.50% | +0.29% | -6.07% |
| hiva_agnostic | -7.85% | +4.91% | +0.52% | +0.50% |
| QSAR-TID-11 | -6.34% | -3.93% | -0.28% | -2.15% |
| CIFAR-10 | -2.87% | -0.01% | +2.42% | +0.28% |
| micro-mass | -12.10% | -35.12% | +6.11% | -38.93% |
| 11_Tumors | -56.08% | -55.13% | -23.85% | -70.62% |
| amazon-reviews | -81.08% | -73.63% | -62.87% | -80.39% |
Kumo has a stronger baseline on all seven tables. A stronger backbone may already identify and use informative features more effectively, leaving less room for external column selection to improve predictions. On Amazon reviews, for example, column selection reduces log loss from 3.907 to 0.739 for TabICL v2 and from 1.321 to 0.490 for Kumo.
Importance-based selection outperforms random selection on the same five tables for both backbones, supporting the importance of column choice beyond TabPFN-3. The gains over both models' native baselines further show that column selection can improve predictions.
Reply to Reviewer DJwj: Q1 and W1
Thank you for your thoughtful questions. We provide additional evidence on aggregation and column selection across backbones and on adaptation under distribution shifts. We also clarify our analysis of whether dataset characteristics predict gains, discuss practical compute allocation and tuning, and explain our unexpected findings on synthetic context expansion.
Q1 & W1: Explanations for performance gains and expected behavior in OOD settings.
Thank you for raising this insightful question. Our diagnostics suggest that an important role of adaptation is to change how a pretrained TFM uses its existing representations and context. The pretrained representations may already be useful for a task, while their relative contributions to attention need adjustment. DiagScale makes this adjustment by reweighting query–key dimensions, offering a way to adapt the model without relearning those representations. In the SoftScale probes, retaining the learned direction of the scale vector preserves substantially more of the gain than retaining only its norm change. This supports dimension-specific reweighting as a useful mechanism beyond a uniform attention-temperature adjustment.
For aggregation, candidate predictions differ in both quality and the errors they make. Selection can exploit complementary predictions while limiting the influence of less useful members. Our controlled comparisons support the importance of combining diversity with selection: the same pool can improve or worsen performance depending on how its predictions are selected and combined. For context construction, retrieval can improve predictions by providing more targeted support, but it can also fail when a harmful distribution shift is introduced. Our matched-size comparisons on Covertype and Poker show that attention-selected contexts outperform random contexts. One failure case is COMET_MC, where retrieval increases the rare-class proportion from 1.7% in the source pool to a median of 40.3% across retrieved contexts, and the log loss increases from 0.0151 to 0.2302.
For OOD settings, we report results for a subset of 10 BeyondArena datasets with official temporal and grouped splits, alongside randomly split tasks as IID comparisons. We fine-tune TabPFN-3 separately on each dataset, using the learning rates selected on TabArena. The table below reports raw test scores averaged over each dataset's official splits; bold marks the best score in each row.
The following table summarizes geometric mean relative error changes for this subset by split type; negative values indicate improvement.
On the ten datasets reported here, adaptation yields larger average gains under IID splits, while gains under temporal and grouped splits are smaller. This pattern suggests that adaptation benefits from alignment between the training and deployment distributions. The learned adjustments can remain useful under temporal or group shifts, but their benefits may transfer less fully.
For aggregation, members and weights selected on validation data may be less effective if the query distribution changes. For retrieval, gains depend on whether the source pool contains relevant examples for the new queries. Validation splits that reflect the intended deployment setting can help assess whether these methods remain beneficial under such shifts.