acceptodds
Under review as a conference paper at ICLR 2027

Test-Time Compute for Tabular Foundation Models: Mechanisms, Gains, and Limits

20 views

Abstract

Which forms of test-time compute improve the predictions of strong pretrained tabular foundation models (TFMs)? We systematically study this along three axes: adaptation, aggregation, and context construction. Our evaluation spans modern TFMs across the TabArena benchmark, supplemented by experiments on wide and large-scale tables from OpenML. For adaptation, we introduce DiagScale, a diagonal query-key similarity update. It trains only %-% of model parameters and achieves gains comparable to full fine-tuning across three independently pretrained backbones. For aggregation, both pool composition and selection strategy matter. TabPFN-3 already averages predictions from different preprocessing variants of the same data, and adding more such predictions yields diminishing returns. With a broader pool of configurations, greedy selection reduces error by % relative to the default predictor, but uniform averaging increases error. For context construction, attention-guided retrieval improves TabPFN-3's predictions on some large tables and supports source pools beyond the full context memory limit. The context expansion methods we test yield no consistent improvement. Taken together, our results suggest that adaptation and selective aggregation yield consistent benchmark-level gains. The benefits of context construction depend more on the task and data regime. Adaptation and aggregation over the same backbone yield further gains when combined, but require substantially more computation than default inference. These trade-offs motivate choosing strategies according to the available computation budget.

Primary area:
foundation or frontier models, including LLMs
Keywords:
tabular foundation models, test-time compute

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_DJwjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In this study, the authors examine the effects of adaptation, aggregation and context construction in recent tabular foundation model.

For adaptation, the paper introduces DiagScale, which shows that only a magnitude scaling of query-key values, we can reduce the parameter updates to lower than 0.03% compared to full fine-tuning. Next, for aggregation, the authors combine built-in ensembles generating V many views with K many different configurations. To combine these different views stemming from different configurations, the authors investigate three reducer functions: a uniform aggregation, a greedy selection heuristic that adds candidates that minimized the validation loss, or an best-single approach for selecting the candidate with the lowest val loss. Lastly, the study examines context construction, where column curation is framed for importance-based selection rules , whereas row selection is based on query-to-context attention for selecting relevant rows. In the last set of experiments, the authors investigate the effects of combining adaptation with aggregation.

Critical strengths

S1: Thorough analysis of test-time compute strategies

The paper systematically investigates the effects of three components that come into play in test-time computation for TFMs, adaptation, aggregation and context construction. The study reveals which one of them actually provide a lift for the predictive performance in the tested models, and therefore, isolates the individual design choices.

S2: DiagScale as parameter-efficient adaptation strategy

The proposed DiagScale heuristic provides an important contribution. In the study provided, it is highlighted that adapting only a small number of parameters is sufficient to achieve a performance level that is comparable to a full fine-tuning approach.

Critical weaknesses

W1: Limited explanation

Even though appreciating the extensive experimental study and the empirical evidence provided by the authors, we see which strategies could improve performance under which specific experimental setting, but at the end of the day we have limited insight into why specific strategies work/fail. So what is the recommended default configuration now where the expectation is to get good performance. Also in terms of OOD settings? For example the DiagScale heuristic seems promising, but what is the property that makes these corrections provided by the magnitude scaling sufficient?

W2: Meta-feature analysis

Thinking a bit of W1, I’d be curious to know from the authors if they have likewise systematically investigate whether there are some meta-feature constellations within the dataset where we can expect performance improvements? So, do we find some feature redudancies that might help, the intrinisic dimensionality of datasets, the authors slightly touched the class imbalance problem etc. So for the three controlled mechanics discussed (adaptation, aggregation, context construct), can we identify specific conditions that need to be met such that a benefit in perfroamance can be expected?

W3: Coverage of TFMs

In the paper, the aggregation and context construction are concentrated around the TabPFN-3 model. Therefore, the take-aways discussed here should be taken with caution if we understand the manuscript as a general investigation of TFMs. The adaptation experiments include three different backbones. The experiments could be extended to other model families to have stronger evidence that some of the identified patterns are ones that go beyond TabPFN-3.

W4: Practical advantage

In Figure 5, it seems like that a stronger backbone beats any add-ons for other competitors at lower costs. Speaking from a practical point of view, is the computational overhead then better be allocated to any of the two heuristics included here (adapt., agg), selecting a stronger backbone, or performing more intense hyperparamter optimization that is currently not in the scope of the draft. Speaking of HPO, by any chance that the authors conducted hpo to the tested settings to potentially get the best performance out of each config? From my understanding, the diagscale provides a light-weight heuristic that is also considerably low in the computational overhead, however, preferring a stronger backbone instead of leveraging the aggregation heuristic due to its expensive computational costs. But maybe the authors can elaborate a bit on their practical recommendations.

W5: Some insights are of lower novelty

The study is comprehensive in total, however, some findings also cover some insights that are established also in the respective original works. Like in inspection of larger candidate pools with effective selection strategies, the diminishing returns from increasing the ensemble size, advantages of additional training samples, etc.. So, some insights can be considered as known, however, the manuscript validates them again for TFMs. I’d have been more surprised by any novelty that specifically is valid for TFMs, that we would not expect according to the knowledge stemming from standard literature about ensemble learning or tabular machine learning in general. Maybe a paragraph of what is it that stands out from the insights that is only valid for TFMs and kind of ‘unexpected’ behavior.

Questions for authors
  • Q1. Even though some deeper investigations of model-data interactions are postponed to future work, can the authors provide insights into their explanations for any lifts we can see when integrating/applying the heuristics that are examined in isolation in the proposed study? What is the expected behavior for OOD settings? (W1)
  • Q2. Did the authors also inspect more thoroughly the meta features of the underlying datasets. Can we see patterns where some of the heuristics are expected to be more useful compared to settings where no benefit is seen. (W2)
  • Q3: Can the authors give an explanation or provide a discussion why some settings are framed for TabPFN and why it is expected that these insights also also for other tfm families. (W3)
  • Q4: Can the authors provide a discussion on the practical advantage and how computational power should be allocated? Also, did the authors conduct a more in-depth inspection of hyperparameter optimization? (W4)
  • Q5. (Minor). From the authors’ experience and insights they gained from their analysis. What is the most unexpected behavior that crushed their expectation when conducting the experiments. Besides the DiagScale insights that provide some interesting reduction of parameters while keeping the performance levels, what are the crucial insights they gained for TFMs? (W5)
6 comments
AuthorsOpenReview ↗

Reply to Reviewer DJwj: Q1 and W1

Thank you for your thoughtful questions. We provide additional evidence on aggregation and column selection across backbones and on adaptation under distribution shifts. We also clarify our analysis of whether dataset characteristics predict gains, discuss practical compute allocation and tuning, and explain our unexpected findings on synthetic context expansion.

Q1 & W1: Explanations for performance gains and expected behavior in OOD settings.

Thank you for raising this insightful question. Our diagnostics suggest that an important role of adaptation is to change how a pretrained TFM uses its existing representations and context. The pretrained representations may already be useful for a task, while their relative contributions to attention need adjustment. DiagScale makes this adjustment by reweighting query–key dimensions, offering a way to adapt the model without relearning those representations. In the SoftScale probes, retaining the learned direction of the scale vector preserves substantially more of the gain than retaining only its norm change. This supports dimension-specific reweighting as a useful mechanism beyond a uniform attention-temperature adjustment.

For aggregation, candidate predictions differ in both quality and the errors they make. Selection can exploit complementary predictions while limiting the influence of less useful members. Our controlled comparisons support the importance of combining diversity with selection: the same pool can improve or worsen performance depending on how its predictions are selected and combined. For context construction, retrieval can improve predictions by providing more targeted support, but it can also fail when a harmful distribution shift is introduced. Our matched-size comparisons on Covertype and Poker show that attention-selected contexts outperform random contexts. One failure case is COMET_MC, where retrieval increases the rare-class proportion from 1.7% in the source pool to a median of 40.3% across retrieved contexts, and the log loss increases from 0.0151 to 0.2302.

For OOD settings, we report results for a subset of 10 BeyondArena datasets with official temporal and grouped splits, alongside randomly split tasks as IID comparisons. We fine-tune TabPFN-3 separately on each dataset, using the learning rates selected on TabArena. The table below reports raw test scores averaged over each dataset's official splits; bold marks the best score in each row.

DatasetSplit typeMetricFrozenDiagScaleFull FT
Drug-induced autoimmunityRandomROC AUC ↑0.848180.853430.85236
EmscadRandomROC AUC ↑0.992910.993500.99362
Hotel booking demandTemporalROC AUC ↑0.776470.783720.78046
Otto product classificationRandomLog loss ↓0.441710.428570.42631
Mice protein trisomy discriminantGroupedLog loss ↓1.178601.178661.17866
CardiotocographyGroupedLog loss ↓0.242980.242910.24277
Early learning predictorsGroupedRMSE ↓12.7665912.7469612.75284
Immoscout German house pricesRandomRMSE ↓0.360580.353160.35322
SAT11 algorithm runtimeGroupedRMSE ↓2.077842.048612.08087
Mercedes-Benz manufacturingTemporalRMSE ↓7.665447.691197.67161

The following table summarizes geometric mean relative error changes for this subset by split type; negative values indicate improvement.

Official splitDatasetsDiagScaleFull FT
Random4-4.23%-4.62%
Temporal2-1.47%-0.86%
Grouped4-0.40%-0.01%

On the ten datasets reported here, adaptation yields larger average gains under IID splits, while gains under temporal and grouped splits are smaller. This pattern suggests that adaptation benefits from alignment between the training and deployment distributions. The learned adjustments can remain useful under temporal or group shifts, but their benefits may transfer less fully.

For aggregation, members and weights selected on validation data may be less effective if the query distribution changes. For retrieval, gains depend on whether the source pool contains relevant examples for the new queries. Validation splits that reflect the intended deployment setting can help assess whether these methods remain beneficial under such shifts.

AuthorsOpenReview ↗

Reply to Reviewer DJwj: Q2 and W2

Q2 & W2: Can dataset meta features predict when the methods improve performance?

Thank you for suggesting this analysis. We found that greater disagreement among native views is associated with larger aggregation gains, but the meta features we examined did not reliably predict gain magnitudes on new datasets. In Appendix E.3, we examine whether dataset characteristics and model behavior can help anticipate adaptation and aggregation gains. We consider 21 predictors, including dataset size, feature count, class imbalance, missingness, cardinality, feature correlation, univariate signal concentration, and disagreement across the frozen model's predictions.

The clearest pattern is that greater disagreement among TabPFN-3's native views is associated with larger configuration-aggregation gains (Spearman ρ=0.496\rho=0.496, globally adjusted q=0.031q=0.031). Together with our fixed-pool comparison of averaging and greedy selection, this suggests that disagreement provides opportunities for improvement, while selection determines how effectively those differences are used. We did not find an equally robust association for adaptation after correcting for multiple comparisons.

We also tested whether these meta features jointly predict per-dataset gains. We used the meta features as inputs to ridge regression and gradient-boosted trees, predicting relative error reductions from DiagScale and full fine-tuning on each of the three backbones, plus 96-configuration greedy aggregation on TabPFN-3. We evaluated predictions using five-fold cross-validation over datasets, repeated 20 times. For each repeat, we computed R2=1−∑d(yd−y^d)2/∑d(yd−yˉ)2R^2=1-\sum_d(y_d-\hat y_d)^2/\sum_d(y_d-\bar y)^2, where ydy_d is the observed gain for dataset dd, y^d\hat y_d is its held-out prediction, and yˉ\bar y is the mean observed gain across datasets. Of the 14 prediction tasks (two regression models ×\times seven gain outcomes), 13 yield negative mean R2R^2, and the remaining one is only 0.012. Thus, these meta features did not reliably predict gain magnitudes on new datasets.

For context construction, our experiments suggest examining how selection changes the context, in addition to the original dataset's characteristics. In the COMET_MC failure case, retrieval substantially increases the rare-class proportion relative to the source pool. This points to changes in context composition as a useful diagnostic to investigate, rather than treating the original class imbalance alone as a predictor of retrieval benefit.

AuthorsOpenReview ↗

Reply to Reviewer DJwj: Q3 and W3

Q3 & W3: Why some studies focus on TabPFN and why their insights are expected to generalize to other TFM families.

We concentrated the detailed aggregation and context-construction studies on TabPFN-3 to keep the backbone fixed while varying the intervention. We expect these insights to extend to other TFMs because they share the relevant mechanisms: aggregation combines predictions that differ in quality and errors, while context construction changes the examples and features used for in-context learning. Prediction diversity, member selection, and context choice should therefore matter beyond TabPFN-3. We have now extended both comparisons to other model families.

For aggregation, we evaluated 96 configurations on TabICL v2 across all 51 TabArena datasets. Greedy selection reduces error by 1.93%, compared with 0.75% for uniform averaging, averaged across datasets. The larger gain from greedy selection supports the value of selecting ensemble members beyond TabPFN-3. Unlike on TabPFN-3, uniform averaging also improves predictions here, showing that the benefit of averaging the full pool varies across backbones.

For context construction, we evaluated column selection on TabICL v2 and Kumo Large using the same seven wide tables and experimental settings as the TabPFN-3 study. We compare importance-based selection with each model's baseline without external column selection and with random selection of the same number of columns. The table reports relative error changes; negative is better.

DatasetTabICL v2: vs. baselineTabICL v2: vs. randomKumo Large: vs. baselineKumo Large: vs. random
Bioresponse-11.48%-9.50%+0.29%-6.07%
hiva_agnostic-7.85%+4.91%+0.52%+0.50%
QSAR-TID-11-6.34%-3.93%-0.28%-2.15%
CIFAR-10-2.87%-0.01%+2.42%+0.28%
micro-mass-12.10%-35.12%+6.11%-38.93%
11_Tumors-56.08%-55.13%-23.85%-70.62%
amazon-reviews-81.08%-73.63%-62.87%-80.39%

Kumo has a stronger baseline on all seven tables. A stronger backbone may already identify and use informative features more effectively, leaving less room for external column selection to improve predictions. On Amazon reviews, for example, column selection reduces log loss from 3.907 to 0.739 for TabICL v2 and from 1.321 to 0.490 for Kumo.

Importance-based selection outperforms random selection on the same five tables for both backbones, supporting the importance of column choice beyond TabPFN-3. The gains over both models' native baselines further show that column selection can improve predictions.

Reviewer_MQLvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates additional task-specific computation for pretrained tabular foundation models (TFMs), organizing the inference pipeline into three components: adaptation, aggregation, and context construction. Its main methodological contribution is DiagScale, which learns dimension-wise query–key similarity multipliers while freezing the pretrained backbone. The authors report that updating approximately 0.003–0.03% of model parameters yields benchmark-level gains close to full fine-tuning across TabPFN-3, TabICL v2, and TabFM.

Critical strengths
  1. Useful decomposition and controlled comparisons.
  2. Substantial empirical support for compact adaptation.
  3. Reporting of negative results and computational trade-offs.
Critical weaknesses

W1. Limited methodological novelty. My main concern is that much of the paper applies established adaptation, ensemble-selection, and retrieval techniques to stronger TFM backbones. The manuscript itself identifies relevant precedents for these operations, including attention-guided retrieval and greedy ensemble selection. Consequently, the contribution is stronger as an empirical study than as a new methodological framework.

W2. Elo-centered presentation is insufficient to establish comparable fine-tuning gains. Although Elo is a legitimate benchmark summary, similar Elo improvements do not necessarily imply similar improvements in the underlying task metrics. The paper’s Elo procedure uses split-level pairwise wins, ties, and losses, rather than the magnitude of the performance differences. Therefore, full fine-tuning and a parameter-efficient method can have identical comparison outcomes while delivering substantially different improvements in ROC AUC, log loss, or RMSE. The central concern is not that Elo is inherently unfair, but that it is insufficient by itself to establish comparable adaptation benefits. I acknowledge that the paper does not rely exclusively on Elo: Figure 2 provides per-dataset relative improvements, and Appendix Tables 9–10 provide direct error-based comparisons and distribution summaries. Table 9 also appropriately states that its intervals do not establish statistical equivalence. Nevertheless, the headline adaptation tables emphasize ΔElo, and a complete side-by-side presentation of the absolute primary metrics would make the practical meaning of “gains close to full fine-tuning” substantially clearer.

The relevant limitation concerns unchanged pairwise outcomes, not merely unchanged dataset-average ranks. For example, consider a hypothetical split where the frozen backbone achieves AUC 0.900 and already outperforms every other reference. DiagScale improves AUC to 0.901, whereas full fine-tuning improves it to 0.920. Both adapted models win against all references on that split, so their contributions to the Elo comparison are identical despite very different absolute improvements.

Importantly, an already top-ranked backbone can still gain Elo by beating its frozen reference. What Elo cannot capture is an additional increase in the winning margin when the pairwise outcomes remain unchanged.

Questions for authors

Q1. How should the Elo comparisons be interpreted, and can the authors provide absolute primary-metric results? Which candidates were used in the calculation of elo?

Q2. How reliable is DiagScale on an individual deployment task? Figure 2 shows correlated but non-identical gains across datasets. What evidence supports choosing DiagScale rather than full fine-tuning for a new dataset when maintaining predictive performance is more important than minimizing the number of trainable parameters?

Q3. How general are the native-view scaling conclusions across TFM backbones? In my own practical evaluations, the improvement from native ensembling is often already marginal beyond approximately 16 members. This observation does not necessarily contradict the manuscript: Section 6 reports error reductions of 0.48% at 16 views and 0.60% at 32 views relative to the eight-view default, which already suggests a small additional benefit between 16 and 32 views. Could the authors provide a direct paired comparison between 16 and 32 views and extend the scaling curves to TabICL v2 and TabFM, as well as Kumo and LimiX-2 where comparable ensemble controls are available?

Q4. How is attention-based retrieval implemented within the stated memory budget? Obtaining attention scores can itself create substantial intermediate-memory requirements. Could the authors describe the implementation more explicitly, including query batch sizes, key-side tiling, how softmax normalization and head averaging are performed during streaming, and whether representations are cached, recomputed, or offloaded? A stage-by-stage peak GPU-memory profile for attention scoring, retrieval construction, and final prediction would make the scalability results easier to assess and reproduce.

Q5.Can DiagScale be compared directly with fine-tuning only WQW_Q ? My hypothesis is that DiagScale is effective because it adapts the distribution and relative weighting of query representation dimensions to the downstream task. A more targeted baseline is therefore needed to distinguish the benefit of query-side adaptation in general from the benefit of restricting that adaptation to diagonal scaling.

For a simple linear query projection, this relationship is explicit. With column-vector notation, q=WQhq=W_Qh, \quad D=I+diag⁡(δ),D=I+\operatorname{diag}(\delta), applying DiagScale gives q′=Dq=(DWQ)h.q'=Dq=(DW_Q)h. In this simplified setting, DiagScale is a constrained task-specific update to the effective query projection. However, the paper applies its multiplier after the backbone’s own query transformations, so it is not generally valid to equate it with changing the original WQW_Q when intervening query-dependent transformations are present. This makes an empirical comparison particularly useful.

Table 1 includes joint query–key projection fine-tuning, but this updates both WQW_Q and WKW_K and therefore does not isolate the proposed query-side explanation.

If the author can answer my questions, I will revise my score based on the author's response.

8 comments
AuthorsOpenReview ↗

Reply to Reviewer MQLv: W1

Thank you for your careful analysis and concrete suggestions. We provide direct error comparisons with confidence intervals and raw task scores, extend the 16-versus-32-view comparison to five backbones, and compare DiagScale directly with fine-tuning only WQW_Q. We also clarify the retrieval implementation and report GPU-memory usage at each stage.

W1: Empirical and analytical contributions beyond existing methods.

We agree that our main contribution lies in a systematic empirical investigation. We also provide a unified framework for test-time compute in modern TFMs, organizing adaptation, aggregation, and context construction to study their individual effects and interactions.

Our SoftScale diagnostics reveal the importance of dimension-wise attention reweighting, and DiagScale achieves gains close to full fine-tuning across three TFMs. For aggregation, we show that configuration diversity and selection provide complementary benefits. For context construction, we identify gains from targeted retrieval and failures associated with shifts in class composition. We also demonstrate further gains from combining adaptation with aggregation.

Together, these findings help explain where test-time compute gains come from and what limits them. They provide an empirical and analytical understanding of TFM behavior beyond evaluating existing methods on stronger backbones.

AuthorsOpenReview ↗

Reply to Reviewer MQLv: Q1 and Q2

Q1 & W2: Interpreting Elo, reporting absolute task metrics, and specifying the comparison models.

Thank you for this careful and deep analysis. We totally agree that identical pairwise outcomes can yield identical Elo contributions despite different improvements in the underlying task metric. Our claim that DiagScale achieves gains close to full fine-tuning therefore also relies on direct error comparisons.

To compare adaptation gains in both Elo and the underlying task metrics, we report Δ\DeltaElo and relative error changes across all 51 datasets in the following table. Negative error changes favor the first method in each comparison.

BackboneDiagScale: ΔEloDiagScale: Error vs. frozenFull FT: ΔEloFull FT: Error vs. frozenDiagScale vs. full FT: Error change
TabPFN-3+26-0.739% [-1.266, -0.247]+25-0.649% [-1.076, -0.267]-0.091% [-0.283, +0.084]
TabICL v2+82-1.893% [-2.804, -1.065]+77-1.441% [-2.135, -0.826]-0.458% [-1.064, -0.016]
TabFM+20-0.297% [-0.697, +0.049]+28-0.398% [-0.739, -0.126]+0.102% [-0.054, +0.286]

Frozen comparisons use the reproduced backbone with matching inference settings. Errors are 1−ROCAUC1-\mathrm{ROC\\ AUC} for binary classification, log loss for multiclass classification, and RMSE for regression. We compute percentage changes from geometric mean error ratios, with 95% confidence intervals from paired dataset bootstrapping.

The Elo and error-based comparisons show consistent overall trends: both methods improve over the frozen baseline in their point estimates, with DiagScale performing slightly better on TabPFN-3 and TabICL v2, and full fine-tuning performing slightly better on TabFM. The error-based comparisons additionally quantify the magnitude and uncertainty of these differences. These small aggregate differences support our description of the gains as close. Figure 2 shows the variation across datasets.

We provide absolute test scores below for one dataset from each task type: binary classification, multiclass classification, and regression. Scores are averaged across repeats and folds.

DatasetBackboneMetricFrozenDiagScaleFull FT
In-vehicle couponTabPFN-3ROC AUC ↑0.845810.850770.85078
In-vehicle couponTabICL v2ROC AUC ↑0.843890.852100.85145
In-vehicle couponTabFMROC AUC ↑0.847440.856330.85633
Hiva agnosticTabPFN-3Log loss ↓0.179840.177590.17767
Hiva agnosticTabICL v2Log loss ↓0.198780.176030.17713
Hiva agnosticTabFMLog loss ↓0.177440.176800.17592
Airfoil self-noiseTabPFN-3RMSE ↓0.988310.983100.98651
Airfoil self-noiseTabICL v2RMSE ↓1.078071.019921.03617
Airfoil self-noiseTabFMRMSE ↓1.071931.023841.03367

We will provide the full per-dataset scores for adaptation, aggregation, and context construction, together with their baseline scores, in the revised appendix.

The complete reference list and Elo calculation are already provided in Appendix A.4. We use 68 fixed reference entries: 67 TabArena entries plus frozen TabFM. These include Random Forest, LightGBM, CatBoost, XGBoost, neural tabular models, tabular foundation models such as TabPFN-3 and TabICL v2, and AutoGluon. The entries include default, tuned, and ensembled variants, rather than 68 distinct model families. Each evaluated candidate is added separately to this fixed set, producing a 69-entry fit, and its Δ\DeltaElo is measured against the corresponding reference baseline within that same fit.

Q2: How reliable is DiagScale on an individual deployment task?

Thank you for raising this practical question. Our direct error comparisons show that DiagScale is competitive with full fine-tuning across the three backbones: it achieves slightly lower aggregate error on TabPFN-3 and TabICL v2, while full fine-tuning performs slightly better on TabFM. Figure 2 shows that their gains are strongly correlated across datasets.

The gains from both DiagScale and full fine-tuning vary across datasets. DiagScale's benchmark-level gains show that dimension-wise attention reweighting is a competitive adaptation strategy, while choosing between the two methods for a particular task requires evidence from that task.

When predictive performance is the priority, we recommend comparing DiagScale and full fine-tuning on validation data representative of the deployment task. When their validation performance is similar, DiagScale offers a practical advantage: its task-specific parameters amount to only approximately 0.003%–0.03% of the backbone, allowing many adapted models to share the same pretrained weights. This substantially reduces the parameter storage needed for multiple tasks.

AuthorsOpenReview ↗

Reply to Reviewer MQLv: Q3 and Q5

Q3: Native-view scaling across TFM backbones.

Thank you for suggesting this direct comparison. Our results support your observation: increasing native views from 16 to 32 generally yields small additional gains. We conducted a paired comparison across all five backbones on the full 51-dataset TabArena benchmark.

BackboneError change: 32 vs. 16 views95% CI
TabPFN-3-0.12%[-0.28%, +0.06%]
TabICL v2-0.03%[-0.16%, +0.11%]
TabFM-0.03%[-0.16%, +0.10%]
LimiX-2-0.44%[-1.19%, +0.13%]
Kumo Large+0.01%[-0.15%, +0.26%]

Negative values indicate lower error with 32 views. Following the paper, we average split-level log error ratios within each dataset and then across datasets, transforming back to percentage changes. The 95% confidence intervals are obtained by bootstrapping datasets.

The extra views make little difference for TabPFN-3, TabICL v2, and TabFM, with estimated error reductions of at most 0.12%. Kumo Large is slightly worse by 0.01%. LimiX-2 has a larger estimated reduction of 0.44%, but all five confidence intervals include zero. Overall, these results agree with your observation that the gains from additional native views are often marginal. These results motivate exploring both configuration diversity and how predictions are selected and combined, rather than simply increasing the number of native views.

Q5: DiagScale versus fine-tuning only WQW_Q.

Thank you for the insightful analysis of the relationship between DiagScale and query-projection updates. We agree that directly comparing DiagScale with fine-tuning only WQW_Q is important for understanding the role of the diagonal restriction. We evaluated this baseline on TabPFN-3 across all 51 TabArena datasets and 816 test splits, updating only the query-projection weight matrices while keeping the remaining pretrained parameters frozen.

We follow the same learning-rate search range, selection criterion, and validation procedure as in the paper.

ComparisonRelative error change95% CI
WQW_Q-only vs. frozen-0.419%[-0.816%, -0.047%]
DiagScale vs. frozen-0.739%[-1.266%, -0.247%]
Full FT vs. frozen-0.649%[-1.076%, -0.267%]
DiagScale vs. WQW_Q-only-0.322%[-0.594%, -0.084%]
Full FT vs. WQW_Q-only-0.231%[-0.369%, -0.112%]

Negative values favor the first method in each comparison. The frozen baseline uses matching inference settings. We average split-level log error ratios within each dataset, then average across datasets and transform back to percentage changes. Confidence intervals use 10,000 paired bootstrap samples over datasets.

Updating WQW_Q alone lowers error relative to the frozen model, supporting the usefulness of query-side adaptation. DiagScale achieves a further reduction, with the paired confidence interval entirely below zero, while using roughly 0.2% as many trainable parameters.

These results suggest that reweighting existing query dimensions can be more effective than allowing a more flexible query-projection update, consistent with a useful regularizing effect of the diagonal constraint. As you note, DiagScale reweights query dimensions after the backbone's own query transformations, allowing it to directly adjust their contributions to attention similarity. This experiment provides further evidence for the effectiveness of dimension-wise reweighting in adapting TFMs. We appreciate the insightful question and will include these results in the revised paper.

Reviewer_iHTAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies what kinds of task-specific test-time computation can improve strong pretrained tabular foundation models. It breaks down test-time compute into three components of the inference pipeline: adaptation (changing parameters), aggregation (producing and combining multiple predictions), and context construction (choosing or adding rows and columns). Experiments are mainly on TabArena and larger OpenML tables.

For adaptation, the paper proposes DiagScale to learn to scale the query-key similarity diagonally, motivated by a sweep of 25 fine-tuning settings on TabPFN-3 for adaptation, and obtain gains close to full fine-tuning..

For aggregation, adding more ensembles saturates quickly, and over a pool of 96 configurations greedy selection improves over the default setting, while uniform average ensemble hurts performance.

For context construction, attention-guided row retrieval helps on some large tables, while column selection helps on some wide tables but hurts on others. In addition, adding constructed rows or columns does not provide consistent gains. The largest boost comes from combining DiagScale with aggregation, at a much higher cost, and the gains per dataset are hard to predict from dataset characteristics.

Critical strengths
  1. DiagScale is a natural and light-weight adaptation method. DiagScale trains only 0.003-0.03% of parameters and achieves gains close to full fine-tuning on three independently pre-trained backbones. It also appears less sensitive to the learning rate. The dataset-level cross-validation of the learning-rate choice on Table 7 also shows that the gains are not just an artifact of the rate selection on the reported benchmark. This offers a cost-effective and natural alternative to full fine-tuning.

  2. Comparisons are well controlled and negative results are reported. The paper uses appropriate control experiments to support its claims. Attention-selected contexts are compared against random contexts of the same size, and the dependence on validation data is measured and reported. The paper also reports what doesn't work (e.g., context expansion failure cases and the unpredictability of per-dataset gains) and GPU-hour costs. This leads to more reliable conclusions and useful decisions about where to spend more compute.

Critical weaknesses
  1. The three components are studied separately without a unified framework The paper presents adaptation, aggregation and context construction as three ways to use test-time compute in TFMs, but it largely studies them in isolation from each other instead of their interactions and tradeoffs under a shared budget. Context construction is never combined with the other components, although they may interact with each other: adaptation changes context-length dependent attention scaling, while context construction changes the length of the context. The paper provides separate conclusions to each component, but does not adequately answer an important question: in a practical scenario, how should we distribute a fixed test-time budget among the three components? Adding a compute-matched joint evaluation would help clarify the contribution more clearly.

  2. Reliance on existing, widely used methods without sufficient analytical depth DiagScale is essentially the same as the key-scaling component of IA3 at the attention-logit level. Configuration aggregation seems to use the same greedy tuned-and-ensembled protocol (Caruana et al., 2004) as in TabArena. The row-retrieval pipeline is adapted from LimiX. The paper claims to be a systematic study, so a new method is not strictly necessary, but then a better unified integration of existing methods or deeper related analysis might be needed to compensate.

References Caruana, R., Niculescu-Mizil, A., Crew, G., and Ksikes, A. (2004). Ensemble Selection from Libraries of Models. ICML 2004. Erickson, N., et al. (2025). TabArena: A Living Benchmark for Machine Learning on Tabular Data. NeurIPS 2025. Liu, H., et al. (2022). Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning. NeurIPS 2022. Zhang, X., et al. (2025). LimiX: Unleashing Structured-Data Modeling Capability for Generalist Intelligence. arXiv:2509.03505.

Questions for authors

Clarifications:

  • How does the 96-configuration greedy aggregation method differ from TabArena's tuned-and-ensembled protocol?
  • How does Diagscale compare with full IA3 (scaling keys, values, and FFN activations), which is not included in the 25-setting menu? This would help clarify whether it matters to restrict the update to the query-key similarity itself, or other small multiplicative adapters can achieve similar effects.

Suggestion: Training only the context-size branch recovers 93–106% of the SoftScale gain, suggesting that adaptation may mainly correct a mismatch in length-dependent attention scaling. Since context construction also changes the context size, perhaps some investigation into this mechanism could better connect the adaptation and context-construction results.

5 comments
AuthorsOpenReview ↗

Reply to Reviewer iHTA: W1

Thank you for your detailed feedback on the interactions among the methods and the depth of our analysis. We add experiments combining adaptation with retrieval, compare DiagScale with full IA$^3$ and its components, and further investigate the role of context length in adaptation. We also clarify the insights from our controlled comparisons and discuss how the results can guide compute allocation.

W1: Interactions among the three methods and compute allocation under a fixed budget.

Thank you for suggesting a joint evaluation. Our paper already evaluates adaptation combined with configuration aggregation across all 51 TabArena datasets (Section 8). We now add experiments combining adaptation with retrieval to examine how context construction interacts with parameter updates.

Combining the methods. We did not include context construction in the benchmark-wide combination because its standalone results were not consistently beneficial across the evaluated datasets (Section 7). We instead examine the combination on Covertype and Poker, using 300k training rows and 50k test queries per dataset, with eight inference views. We compare the frozen model, DiagScale, retrieval, and DiagScale + retrieval. For the combined setting, we first train DiagScale, use the adapted model to retrieve contexts, and predict with the same adapted model.

SettingCovertypePoker
Frozen0.06856±0.000380.06856\pm0.000380.22087±0.012120.22087\pm0.01212
DiagScale0.06427±0.003860.06427\pm0.003860.03878±0.030520.03878\pm0.03052
Retrieval0.06066±0.001850.06066\pm0.001850.17019±0.042410.17019\pm0.04241
DiagScale + retrieval0.06002±0.00213\mathbf{0.06002}\pm0.002130.02264±0.00525\mathbf{0.02264}\pm0.00525

Log loss (lower is better) is mean ±\pm standard deviation over three seeds.

The combination achieves the lowest mean log loss on both datasets, but the main source of improvement differs: retrieval contributes most on Covertype, while adaptation contributes most on Poker. Thus, changing parameters and selecting context examples can provide complementary gains, with their relative contributions depending on the task. On US_Accidents, retrieval followed by eight-view prediction yields 6.64% lower log loss than the single-view version, averaged over five seeds. This provides additional evidence that retrieval and aggregation can be beneficial in combination.

Allocating computation. The paper already compares performance and measured runtime across individual methods and their combinations (Section 8, Figure 5). On TabPFN-3, adding DiagScale to configuration aggregation increases total runtime by only 2.3% while reducing error by a further 0.64%. This suggests that combining the two is worthwhile when the budget already supports aggregation. Our results also show that a stronger frozen backbone can outperform an enhanced weaker model at lower cost. We therefore suggest choosing a suitable backbone before investing in adaptation or aggregation.

After choosing a backbone, adaptation is attractive when many queries are expected on the same task, because its initial fitting cost can be spread over subsequent predictions. If the budget also permits candidate evaluation and additional prediction time, configuration aggregation can be considered. Both the candidate pool size and the number of configurations used for prediction should reflect the available budget. The pool could also be tailored to the task using validation performance, rather than sampled randomly.

When full-context inference exceeds GPU memory, retrieval offers a way to use more training data within that limit and can be combined with adaptation. If the budget permits, a small validation comparison can help choose among these options, with its cost included in the total budget.

AuthorsOpenReview ↗

Reply to Reviewer iHTA: W2 and Q1

W2: Analytical insights beyond applying existing methods.

Thank you for this suggestion. The contribution of our study lies in using controlled comparisons to explain where test-time compute gains come from, when they fail, and how different methods work together.

For adaptation, we identify dimension-wise attention reweighting as an effective way to fine-tune TFMs (Section 5). Our SoftScale diagnostics show that changing the relative contributions of attention dimensions matters beyond a uniform temperature adjustment. DiagScale directly applies this finding: across three TFM backbones, it achieves gains close to full fine-tuning while updating only 0.003%–0.03% of the model's parameters. These results show that substantial adaptation gains can be obtained by adjusting how pretrained query–key dimensions contribute to attention similarity, while keeping the backbone weights fixed. We acknowledge DiagScale's equivalence to IA$^3$'s key-scaling component at the attention-logit level; the contribution lies in establishing its effectiveness for TFM fine-tuning and providing a diagnostic basis for this choice.

For aggregation, we find that configuration diversity and selection are complementary (Section 6). With the same prediction pool, uniform averaging increases error by 3.68%, while greedy selection reduces it by 2.44%.

For context construction, retrieval can improve predictions by providing more targeted support, but can also fail when selection introduces a harmful distribution shift (Section 7). With query groups, context sizes, and refitting fixed, attention-selected rows outperform random rows on Covertype and Poker but underperform them on COMET_MC, where retrieval substantially changes the context's class composition. For context expansion, the methods we tested did not consistently improve on the original context.

These findings provide a mechanistic basis for TFM fine-tuning and clarify the conditions under which aggregation and context construction are beneficial. They constitute the analytical contribution of our study: using controlled interventions to understand how pretrained TFMs respond to additional computation.

Q1: How configuration aggregation differs from TabArena's tuned-and-ensembled protocol.

Our aggregation builds on TabArena's tuned-and-ensembled strategy, using a TabPFN-3 configuration space adapted from TabArena's TabPFN-v2.5 space. We use this strategy to investigate a question beyond the final ensemble's performance: which parts of aggregation produce the gain?

We separate configuration diversity from the rule used to combine predictions. With the same 96-configuration pool, uniform averaging increases error by 3.68%, while greedy selection reduces it by 2.44%. Thus, generating more candidate predictions is not sufficient; how they are selected and combined determines whether this pool helps.

Larger configuration pools improve greedy aggregation in our experiments, but the gains weaken when less validation data is available for selecting members. Increasing native views yields diminishing returns. Combining aggregation with DiagScale further improves performance at a small additional cost, showing that the two methods provide complementary gains.

AuthorsOpenReview ↗

Reply to Reviewer iHTA: Q2 and Suggestion

Q2: How does DiagScale compare with full IA$^3$?

Thank you for suggesting this comparison. We compare DiagScale with FFN-only scaling, key-and-value scaling, key-and-FFN scaling, and full IA$^3$ on TabPFN-3. DiagScale provides the smallest update among these methods, while broader multiplicative adapters offer additional gains.

This diagnostic comparison uses eight datasets covering binary classification, multiclass classification, and regression: Marketing_Campaign, airfoil_self_noise, credit_card_clients_default, houses, physiochemical_protein, students_dropout_and_academic_success, superconductivity, and wine_quality. We follow the paper's training and checkpoint-selection protocol. The table reports the geometric mean error change across these eight datasets.

MethodTrainable parametersError change
DiagScale12,67212{,}672-1.41%
FFN-only26,88026{,}880-1.16%
K+V26,88026{,}880–27,26427{,}264-1.60%
K+FFN40,32040{,}320–40,70440{,}704-1.92%
Full IA$^3$53,76053{,}760–54,14454{,}144-1.91%

Negative values indicate lower error relative to the published TabPFN-3 default. The DiagScale parameter count is for regression. From the results, we can see that DiagScale uses the fewest trainable parameters among the tested adapters, approximately one quarter of full IA$^3$'s. Updates that also scale values or FFN activations can provide further gains, suggesting that diagonal rescaling is useful at multiple locations within the model. Together, these results support multiplicative adaptation as an effective approach to fine-tuning TFMs, which was not revealed by prior work. Our central finding is that restricting adaptation to attention similarity can already achieve gains close to full fine-tuning across three TFMs with very few trainable parameters. We will include these results and analysis in the revised paper.

Suggestion: The connection between context-length-dependent scaling and context construction.

Thank you for suggesting this direction. The paper already examines the SoftScale branch that maps context length to dimension-wise scaling factors, using separate branch training and diagnostics of the scale vectors' direction and norm (Section 5 and Appendix B.5). To further investigate the role of context length in adaptation, we conducted two additional experiments on the same three diagnostic datasets.

First, we examine what fine-tuning the context-size branch changes. For each attention head, let s0s_0 and s1s_1 denote the pretrained and fine-tuned scale vectors at the same context length. We evaluate two replacements:

\qquad s_{\mathrm{norm}} = \frac{\lVert s_1 \rVert_2}{\lVert s_0 \rVert_2}s_0.$$ We apply these replacements without further training, keeping the query-dependent gates, context examples, and evaluation data fixed. Retaining the learned scale direction while restoring the pretrained norm preserves **81–101%** of the gain from fine-tuning this branch, whereas retaining only the norm change preserves **9–25%**. This indicates that the correction mainly changes the relative weighting of query dimensions rather than only the overall attention temperature. We then test whether the learned updates remain effective at different context lengths. We fine-tuned the context-size branch with 128, 512, or 2,048 context rows and evaluated each resulting model at all three lengths, keeping the validation procedure fixed. The table reports percentage changes in error relative to the frozen model at the same evaluation context length; negative values indicate improvement. We report RMSE for Houses and Protein, and log loss for Bankruptcy. | Dataset | Training context rows | Test: 128 | Test: 512 | Test: 2,048 | | --- | --- | --- | --- | --- | | Houses | 128 | -0.37% | -0.47% | -0.65% | | | 512 | -0.27% | -1.13% | -1.67% | | | 2,048 | +0.29% | -1.12% | -2.57% | | Protein | 128 | -0.37% | -0.73% | -0.60% | | | 512 | -0.64% | -1.70% | -1.07% | | | 2,048 | -0.79% | -0.49% | -3.11% | | Bankruptcy | 128 | -2.37% | +0.05% | -0.74% | | | 512 | -0.23% | -0.69% | -1.39% | | | 2,048 | -0.43% | -6.11% | -8.47% | Updates learned at one context length can also improve predictions at other lengths, although the gains vary with the training and evaluation lengths. Changing the training context length also changes the examples available during fine-tuning, so this experiment measures transfer across lengths rather than isolating a length-scaling mismatch. These diagnostics suggest that the interaction may involve how adaptation reweights attention dimensions, together with changes in context length. Alongside these diagnostics, our Covertype and Poker experiments directly evaluate the combination of retrieval and adaptation, which achieves lower mean log loss than either method alone.
Reviewer_QN68OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper systematically investigates how test-time computation (task-specific parameter fitting, model aggregation, and context construction) affects the prediction performance of modern In-context tabular models that are mostly based on TabICLv2 architecture. The paper also introduces parameter-efficient adaptation (DiagScale) and selective configuration aggregation that shows cumulative performance gains despite higher computational cost.

Critical strengths

Overall, the paper is organized and easy to follow. The introduction of light-weight DiagScale, also seems interesting and the empirical results are clearly outlined to justify the points that authors address.

Critical weaknesses

Please refer to questions below.

Questions for authors
  • Given that different TFMs encode features and handle categorical data through different pretraining regimes, why does DiagScale generalize quite robustly? Also, are there specific failure cases?
  • There has been recent works using TFMs even on tables with non-numerical data (possibly BeyondArena, STRABLE, etc.,). How would the proposed modifications behave in such cases?
  • Could severe class-imbalance tables affect some of the analysis given?
  • How could the suggested approach be addressed for production settings (Scalability etc.,)
  • Could the authors provide raw score results?
3 comments
AuthorsOpenReview ↗

Reply to Reviewer QN68: Q1-Q2

Thank you for your thoughtful questions about the generality and practical use of our findings. We provide additional results on tables with non-numerical features, clarify how class imbalance affects our findings, and discuss cross-backbone transfer and failure cases. We also provide raw task scores and practical deployment considerations below.

Q1: Why DiagScale transfers across backbones, and when it fails.

DiagScale operates on an attention mechanism shared by the three evaluated TFMs: query–key dot products. Although these models differ in feature encoding, categorical-data processing, and pretraining, DiagScale adjusts the dimension-wise contributions to attention similarity after the model's own query transformations (Section 5). Its application therefore does not require a particular input encoding or a shared representation across models; the multipliers are learned separately for each model and task.

We consider adaptation unsuccessful on a dataset when its mean test error exceeds the frozen baseline after validation-based checkpoint selection and fallback. On Anneal, DiagScale and full fine-tuning increase log loss by 0.21% and 5.38% on TabICL v2, and by 2.91% and 3.71% on TabFM, respectively. The frozen models already achieve low log loss, while the rarest class contains only eight examples. Limited support for rare classes can make validation less reliable for selecting updates that generalize. The smaller degradation with DiagScale is consistent with its restricted parameterization helping preserve useful pretrained behavior.

Q2: How the proposed methods perform on tables with non-numerical features.

We report results on 10 BeyondArena datasets, including text and high-cardinality categorical features, using their official splits. Frozen TabPFN-3, DiagScale, and full fine-tuning share the same input preprocessing; text features use BeyondArena's semantic embedding pipeline. We transfer the learning rates and training settings from our TabArena experiments.

DatasetSettingMetricFrozenDiagScaleFull FT
Drug-induced autoimmunityIIDROC AUC ↑0.848180.853430.85236
EmscadIIDROC AUC ↑0.992910.993500.99362
Hotel booking demandOODROC AUC ↑0.776470.783720.78046
Otto product classificationIIDLog loss ↓0.441710.428570.42631
Mice protein trisomy discriminantOODLog loss ↓1.178601.178661.17866
CardiotocographyOODLog loss ↓0.242980.242910.24277
Early learning predictorsOODRMSE ↓12.7665912.7469612.75284
Immoscout German house pricesIIDRMSE ↓0.360580.353160.35322
SAT11 algorithm runtimeOODRMSE ↓2.077842.048612.08087
Mercedes-Benz manufacturingOODRMSE ↓7.665447.691197.67161

The table reports raw scores for these 10 datasets, averaged over their official splits. Both methods improve predictions on most of the ten tasks, with DiagScale achieving lower error than full fine-tuning on five. These results show that DiagScale can also perform well on tables containing non-numerical features.

These experiments also provide another perspective: how adaptation behaves in IID and OOD settings, using random splits for IID evaluation and temporal or grouped splits for OOD evaluation. The following table summarizes geometric mean error changes relative to the frozen model for the same ten datasets, grouped by their official split type; negative values indicate improvement.

Official splitDatasetsDiagScaleFull FT
Random (IID)4-4.23%-4.62%
Temporal (OOD)2-1.47%-0.86%
Grouped (OOD)4-0.40%-0.01%

On these ten datasets, adaptation yields larger average gains under IID splits, while gains under temporal and grouped splits are smaller. This pattern is consistent with adaptation benefiting from alignment between training and deployment distributions. The positive average gains under temporal and grouped splits suggest that adaptation can also help under distribution shifts. In practice, validation splits should reflect the intended deployment setting.

Aggregation and context construction can also be applied to tables with non-numerical features using the backbone's input encoding. We expect aggregation to improve benchmark-level performance on tables containing non-numerical features. For context construction, our TabArena results show no consistent benchmark-level gains (Section 7), so we expect its benefits on such tables to remain task- and method-dependent.

AuthorsOpenReview ↗

Reply to Reviewer QN68: Q3

Q3: Could severe class-imbalance tables affect some of the analysis given?

In our evaluation, severe class imbalance can affect the outcome of context construction, and our experiments identify both beneficial and harmful cases.

Anneal is an extreme case of class imbalance, which has class proportions of approximately 76.2%, 11.0%, 7.5%, 4.5%, and 0.9%, with only eight examples in the rarest class. For context expansion, adding four training-derived rows per class with small numerical perturbations yields an average relative log-loss reduction of 8.3%. The same construction does not improve mean log loss on the other datasets in this control.

On COMET_MC, attention-based retrieval substantially increases the proportion of rare-class examples in the context, alongside a deterioration in predictive performance. The rare-class proportion rises from 1.7% in the source pool to a median of 40.3% across retrieved contexts. Log loss increases from 0.0151 to 0.2302 (Appendix D.3). A potential solution is to combine attention-based relevance with class-coverage constraints, so that retrieval preserves useful examples from other classes.

More broadly, building on the dataset-level analysis in Appendix E.3, we examine class imbalance as a predictor of adaptation and aggregation gains. Here, we measure imbalance using C1, one minus normalized label entropy (Lorena et al., 2019): C1=1+∑c=1Cpclog⁡pc/log⁡CC1=1+\sum_{c=1}^{C}p_c\log p_c/\log C, where pcp_c is the training proportion of class cc and CC is the number of classes. Higher values indicate greater imbalance. The table reports Spearman correlations between C1 and per-dataset gains (relative reduction in mean test error versus the frozen backbone) across 38 classification datasets. None is statistically significant, so we do not find clear evidence that class imbalance alone predicts adaptation or aggregation gains.

BackboneMethodSpearman ρ\rhopp
TabPFN-3DiagScale+0.1660.319
TabPFN-3Full FT+0.3060.062
TabPFN-3Aggregation+0.0060.971
TabICL v2DiagScale+0.0160.925
TabICL v2Full FT-0.0960.568
TabFMDiagScale-0.2630.110
TabFMFull FT-0.0240.886

Reference. Lorena, A. C., Garcia, L. P. F., Lehmann, J., Souto, M. C. P., and Ho, T. K. (2019). How Complex Is Your Classification Problem?: A Survey on Measuring Classification Complexity. ACM Computing Surveys, 52(5), Article 107.

AuthorsOpenReview ↗

Reply to Reviewer QN68: Q4-Q5

Q4: How could the suggested approach be addressed for production settings (Scalability etc.,)

We appreciate this practical question. In production, predictive gains need to be weighed against model storage, prediction cost, and available GPU memory. Our three approaches have different implications for these requirements.

DiagScale can substantially reduce model storage when a company serves many tasks or customers. Instead of storing a full fine-tuned model for every task, it can share one pretrained model and store only the small set of learned DiagScale parameters for each task. These contain approximately 0.003%–0.03% of the original model's parameters across our evaluated models (Table 2). The learned multipliers add little inference overhead.

Aggregation shares the pretrained weights across configurations, but requires multiple predictions per query. In production, the candidate pool could be tailored to the task using validation performance, rather than sampled randomly. The number of selected configurations can then be limited to balance predictive gains against inference cost.

Context selection makes larger training pools usable within a limited GPU memory budget. We demonstrated inference from pools of up to 10M rows on one 80 GB H100 (Figure 4 and Appendix D.2). The blockwise retrieval design can accommodate larger source pools by processing additional blocks.

Q5: Could the authors provide raw score results?

Thank you for this suggestion. Below we report raw test scores for three datasets covering binary classification, multiclass classification, and regression, across all three backbones.

DatasetBackboneMetricFrozenDiagScaleFull FT
Polish bankruptcyTabPFN-3ROC AUC ↑0.984900.985620.98549
Polish bankruptcyTabICL v2ROC AUC ↑0.984350.985470.98536
Polish bankruptcyTabFMROC AUC ↑0.995300.995230.99533
Hiva agnosticTabPFN-3Log loss ↓0.179840.177590.17767
Hiva agnosticTabICL v2Log loss ↓0.198780.176030.17713
Hiva agnosticTabFMLog loss ↓0.177440.176800.17592
HousesTabPFN-3RMSE ↓0.189040.187150.18663
HousesTabICL v2RMSE ↓0.194600.187110.18826
HousesTabFMRMSE ↓0.181480.179950.18006

Test scores averaged across repeats and folds. ↑/↓ indicates whether higher/lower is better.

We will add the complete per-dataset results for adaptation, aggregation, and context construction to the revised appendix, alongside the corresponding baselines.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Rebuttal Discussion

Dear Area Chair and Reviewers,

Thank you for the careful reviews and constructive discussion. The feedback helped us clarify the paper's contributions and extend the evidence supporting its findings. We summarize the main points of our replies below.

Our contributions and the discussion. Our main contribution is a systematic empirical study of test-time computation in modern TFMs, organized around adaptation, aggregation, and context construction. We identify dimension-wise attention reweighting as an effective fine-tuning mechanism, distinguish the roles of configuration diversity and selection, and explain opportunities and failures in context construction.

Understanding the mechanisms and interactions. Reviewers iHTA and MQLv recognized the controlled comparisons and empirical support for parameter efficiency in adaptation, while DJwj highlighted DiagScale's parameter efficiency and competitive performance. To better understand the mechanism, we added comparisons with full IA$^3$ and its individual components, as well as fine-tuning only WQW_Q, and extended the paper's SoftScale diagnostics. Reviewer iHTA acknowledged that the additional diagnostics and IA$^3$ comparisons help clarify how DiagScale works. Beyond the paper's study combining adaptation and aggregation across the benchmark, the added experiments combining retrieval and adaptation also show complementary gains.

Interpreting performance and assessing generality. Reviewer DJwj appreciated the systematic analysis of TTC strategies, while QN68 recognized the empirical support for our findings. In response to questions from QN68, MQLv, and DJwj, we provided raw task metrics and paired confidence intervals. Reviewer MQLv interpreted the negative confidence intervals as conflicting with the degradations in Figure 2. We clarified that these intervals concern the average effect across datasets, not individual outcomes, and that checkpoint selection uses validation data while both sets of results report test performance. We also reported adaptation results on ten BeyondArena datasets, extended view scaling to five backbones, and evaluated column selection on TabICL v2 and Kumo. Reviewer DJwj considered the cross-backbone evaluation concern resolved.

Clarifying computational costs and practical use. Reviewers iHTA and MQLv recognized the paper's reporting of computational tradeoffs. To address Reviewer MQLv's concern about retrieval overhead, we provided memory profiles and direct comparisons of total runtime with full context inference. These measurements show that retrieval improves predictions with moderate runtime overhead and enables inference from training pools too large for full context inference on the same GPU. For the practical questions raised by QN68, iHTA, and DJwj, we provided guidance on backbone choice, allocating computation among the methods, and controlling selection and prediction costs according to the workload and available budget.

We will incorporate these additional results and clarifications into the revised manuscript. We believe they strengthen the paper's central contribution. Thank you again for your time and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.