acceptodds
Under review as a conference paper at ICLR 2027

A Controlled Study of Scaling Training Data in Tabular Machine Learning

Abstract

Despite increased attention to the field of tabular machine learning, the niche of large-scale (1M+) data remains under-explored. In this work we study training-data scaling on eight datasets with up to ten million examples, evaluating tabular ML methods under realistic distribution shifts with in-distribution (ID) controls. First, we find that GBDTs and MLP-based models generally improve with additional data and exhibit broadly similar scaling trends. Tabular foundation models vary substantially across pretrained versions, with recent models benefiting more consistently from larger contexts, in some cases up to one million examples under distribution shift. Our analysis of MLP-based models shows that, as training data scales, ID and out-of-distribution (OOD) performance can diverge: for MLP-based models selected using ID validation, additional data can improve ID performance while degrading OOD performance. We show that target-aligned validation mitigates this effect through tuning and early stopping. In contrast, tabular foundation models lack a commonly adopted mechanism for using validation as a separate feedback signal, and we demonstrate that in the presence of strong distribution shift a common strategy of adding validation examples to the context does not prevent OOD degradation as training data grows.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Tabular data, Prior Fitted Networks, MLPs, Generalization, Large Data

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3TaxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In this paper, the authors provide a thorough empirical investigation of scaling training data in the domain of tabular machine learning. They take GBDTs, MLP-based models as well as tabular foundation models into account. The experiments are framed to investigate the the effects of in-distribution and out-of-distribution generalization as the amount of training data grows. Their empirical study highlights the effect of iid performance and how it translates to generalization under some distribution shifts. Whereas GBDT and MLP-based models show better generalization with increasing data samples considered, we also see the accuracy on the wrong line effect. TFMs on the other hand are more prone towards more variation in the performance when scaling input data.

Critical strengths

S1: Thorough empirical analysis of IID and OOD performance across different model families

The authors provide a thorough empirical study on the performance of models and their iid and ood performance. We observe diverse behavior of parametric models and GBDTs compared to TFMs. For the first, we observe the dependency of the model selection to the iid validation against ood validation, where for the latter the validation samples considered in their context does show opposite behavior.

S2: Scaling analysis for TabPack

That component was missing in the analysis of the TabPack paper and also the diversity issues of the models considered in the ensemble. A take-away that needs for inspection for the community is to see how the improvements in IID performance can reliably be translated under some distributions shifts.

Critical weaknesses

W1: Some hyperparameters and setting variations are left out from the analysis

The empirical study already provides a broad inspection of hyperparameters that are validated and compared against each other. We see a thorough scaling of the models’ capacity, adapted learning rates and weight decays. What I’d be curious to see, that has also been figured out for foundation models - also from other domains - that they not only subdivide the learning procedure itself into different stages, but also be curious about the learning rate schedulers and the batch size used for training. here, we see that the e.g. the batch size is fixed across all different settings. My question would be if the authors conducted an initial analysis or runs where these choices are justified with.

W2: Meta feature analysis of the datasets

From the different benchmark datasets used in the evaluation, we see in figure 1 a rather homogenoeus behavior for parametric models, but we see more deterioations w.r.t. the performance for TFM in figure 2. The insights provided in Section 4.2. are rather very general with stating just “differ substantially” without inspecting this behavior in more detail. To answer this question, I’d like to ask the authors if they did for example a more in-depth analysis in the meta-features the datasets potentially share, i.e., do datasets like cooking time, weather and bank fraud potentially share some characteristic that lead to a similar behavior, whereas for datasets like delivery eta and istella we see that tabpfn v3.5 fast and tabicl diverge from the performance of tabpfn v3, and in jane street potentially we have a meta feature constellation not shared by any of the other datasets leading to this divergent behavior of tabicl v2 (maybe shared a bit with aelous flight delay)

W3: Scaling criteria on the iid/ood performance not fully clear

The empirical analysis is built on the idea to scale the training data starting from thounds and scaling to millions of data points. What is not fully clear to me, is how the scaling itself is done. Let’s consider for example the aeolus flight delay dataset, where it is stated that the samples from 2016-2022 were leveraged for the training phase. But which points were now selected from the training data. For simplicity, assume we take 10,20,30%,… from the data. Is the time ordering considered meaning that you first take the 10% most recent data samples and expand it backwards in time, do you start from the beginning of the considered time period (i.e. starting with the first 10% of samples starting from 2016), or do you considered 10% from more or less randomly selected short time intervals spanning the whole time period and expanding the intervals step-wise till you reach 100%? Maybe the assumption is to take the most recent once and extend it backward as the assumption is this distribution rather fits with the validation/test set. Maybe the authors can clarify this setup (potentially I missed it).

W4: Limited explanation of the diverging performance

Speaking of W2 and W3, I’d like to stress the point of explanation a bit more, as in the current version of the manuscript, one could also inspect if the models are over-confident w.r.t. to some spurious features, or even - due to the time-dependency W3 - do we also observe some changes or drifts in the feature-target relationships? Maybe the authors gained some insights already with the conducted experiments but due to no pattern that arised w.r.t. theh deteriotations, they were potentially not reported. However, I’d be curious also to see a bit more of a targeted discussion regarding this point.

Minor:

  • 033: “its”
  • 323: post-hoc
  • 377: between
  • 310: missing white space between “in (Gorishniy et al. […]”

Formatting issues:

  • Table below Figure 8 is cropped due to ending of the page…not readable
Questions for authors
  • Q1: Did the authors conduct experiments w.r.t. some hyperparameters and design choices that were not further changed and assumed to be fixed across all experiments (batch size, learning rate schedulers, learning stages for increased number of training samples)?
  • Q2: Did the authors observe or inspect and meta feature characteristics that might yield an explanation of the deterioration across some benchmarks that impacts the models’ performance?
  • Q3: Can the authors clarify the scaling choice in more detail?
  • Q4: Can the authors provide a more in-depth investigation on the effects stemming potentially also from some shifts in the feature-target relationship and how it might impact a model’s performance.
4 comments

(1/2)

We thank the reviewer for the careful reading and constructive suggestions! We address questions and weaknesses below, we've also uploaded the updated manuscript with changes highlighted in color.

Some hyperparameters and setting variations are left out from the analysis ... learning rate schedulers and batch size used for training...

Q1: Did the authors conduct experiments w.r.t. some hyperparameters and design choices that were not further changed and assumed to be fixed across all experiments

We choose batch size based on training-set size, using a grid similar to one used in (TabArena). The same rule applies throughout the main experiments, except for the model-capacity sweeps, which use a fixed batch size.

To assess sensitivity to this choice, we additionally evaluated TabPack with Muon at batch sizes 1024, 2048, 4096, and 8192 on all eight datasets at the largest training sizes. We repeated configuration and ensemble selection for each batch size while keeping the remaining protocol unchanged. The differences were generally small and dataset-dependent, with no batch size consistently performing best. Our default of 4096 was within 0.5% of the best mean test score on every dataset. We’ve added the experiment and full results to the appendix "Batch Size Parameter".

The learning rate schedule are at odds with the early-stopping which is used for every DL model in the parametric models section, so we chose not to study it. Furthermore, we already study the imporvements in optimization by comparing AdamW and Muon. The general observation that improvements transfer at this dataset scale makes us believe that scheduling could help too if the early-stopping issue is dealt with, but we believe it is out of scope for the current study and the optimizer experiments are sufficient.

Meta feature analysis of the datasets

Q2: Did the authors observe or inspect and meta feature characteristics that might yield an explanation of the deterioration across some benchmarks that impacts the models’ performance?

Limited explanation of the diverging performance

Q4: Can the authors provide a more in-depth investigation on the effects stemming potentially also from some shifts in the feature-target relationship and how it might impact a model’s performance.

Thank you for these questions. Our explanation centers on the interaction between distribution shift and training-data scale. Sections 5.2 and 5.3 were actually motivated by the non-monotonic foundation-model performance on Cooking Time and Jane Street in the begging of the project, with the additional Kumo results showing related behavior on Aeolus Flight Delay.

As we've discussed in related work, Accuracy on the Wrong Line links negative ID–OOD performance correlations to label noise and nuisance features, and shows that additional training data can exacerbate this behavior (Sanyal et al., 2025). This together with the observation of degradations of TFM performance motivates the analysis in sections 5.2 and 5.3. We first see that parametric models also degrade on the same exact datasets when the methods are optimized for ID accuracy (see Figure 11 in the updated manuscript in appendix for Cooking Time, Jane Street and Aeolus Flight delay results). Section 5.3 probes the role of distribution shift further. We restrict training to historical periods that are less representative of the evaluation conditions, while keeping validation and test sets unchanged. These restrictions reduce seasonal coverage on Weather, isolate the pandemic period on Aeolus Flight Delay, and exclude recent observations on Cooking Time; Appendix B describes the construction. Under these restrictions, deterioration appears at sub-million training sizes. Together, these experiments support distribution shift as a central factor in the observed deterioration, with its impact becoming more pronounced as training data grows.

Separately, we believe synthetic-prior coverage helps explain some dataset-specific performance differences across TFMs. Maps Routing has substantially more features than the other tasks, while Istella is a regression task derived from pointwise learning-to-rank data with only five target values. These distinctive properties help explain why some pretrained models perform poorly on these tasks while TabPFN v3.5 and Kumo perform substantially better.

We’ve revised the closing discussion in Section 4.2 and the introduction to the analysis section to make the connection between foundation-model performance declines, distribution shift and model selection explicit.

(2/2)

Scaling criteria on the iid/ood performance not fully clear

Q3: Can the authors clarify the scaling choice in more detail?

Our main experiments use random subsets of a fixed training pool. We generate one random permutation of the training rows and take its first NN entries for training size NN. Each smaller training set is therefore contained in every larger one, and all methods use the same subset at a given size. Validation and test sets remain fixed; ID holdouts are reserved before training subsets are constructed.

We've now also evaluated the alternative suggested by the reviewer: selecting the latest NN training observations and expanding backward in time. This comparison uses TabPack with Muon on the seven temporal datasets, with ten evaluation seeds, the same fixed temporal validation sets, and no refitting. Istella is excluded because its split is query-based.

We've added the results to Appendix "Training Subsampling Strategies".

Selecting the latest observations generally performs worse than random sampling on these tasks. Cooking Time is a notable exception: recent-only training improves performance at intermediate training sizes. This supports the importance of temporal representativeness on this dataset and connects to the experiment in Section 5.3, where we restrict training to the earliest 25% of observations. Adding more examples from that fixed early period does not restore the missing recent coverage, and TabPFN v3.5 exhibits increasing OOD deterioration.

More generally, proximity in time does not consistently make a training subset more useful: recent windows can also lose relevant historical or seasonal coverage. Unlike random sampling from a fixed pool, expanding a recent window changes both sample size and the periods represented. This motivates our main protocol, which separates the effect of training-set size from changes in temporal coverage.

Minor + Formatting issues

Thank you for identifying the issues. We've fixed them in the updated manuscript.

Q1: Thank you very much for providing the new insights into the batch sizes. Even though, we see now that the batch size does not have a major impact for TabPacks performance, I’m still curious to inspect the raised question a bit more in-depth if it holds across different model architectures. Recent studies have shown the impact for LLMs, and in future work we should also shine a light on the training dynamics in TFMs.

Q2: Thanks for providing the additional discussion. So, we have empirical evidence for what you have described, but we are lacking the insights into the concerns raised initially, meaning that we currently do not know the linkage of dataseet properties to the scaling curves. This encompass type and scale of the features. Also with the discussion of the prior, which indeed is a crucial element of the tfm performance capabilities, it remains an open issue at what scale we can expect a prior to cover also real-world problems.

Q3: Thanks for clarifying and providing the additional experiments that random sampling apparently outperforms a sampling heuristic based on chronological subsampling starting with the most recent observaitons.

Q4: How did you check that the historical periods are less representative of the evaluations? So with respect to what? Like an inherent assumption based on seasonal effects, i.e., domain-specific assumptions? Is this the key criteria? So, then we cover it from the distributional representativeness point of view but not shifts with respect to any feature-target relationships (e.g. a feature might become more/less important over time; other feature compositions provide more predictive strengths with scaling data, etc.).

Minor: Acknowledged

—

Thanks to the authors for elaborating on those points.

Thanks for the follow-up!

Q1:

For TFMs, we vary the number of examples in context while keeping the pretrained weights fixed. Batch size and optimization during TFM pretraining are separate questions worth studying in the future.

Q2:

We agree that how feature types, scales and dimensionality affect the scaling curves remains an open question (we state this in the section 4.2 already). Our main finding is that gains from additional training data can fail to transfer under distribution shift, and that the choice of validation distribution matters. For future TFM pretraining, we believe it would be useful to vary both context size and the degree of shift between context and query examples, and test whether this leads to more consistent gains from larger contexts on real tasks.

Q4:

The windows were chosen based on domain knowledge, not a quantitative measure of representativeness / shift. We narrowed seasonal coverage on Weather, kept the pandemic-era period on Aeolus Flight Delay, and excluded more recent observations on Cooking Time, while keeping validation and test sets fixed. The experiment shows that which periods are represented in training matters for OOD performance as data grows (on each task this may relate to a different kind of shift, e.g. weather is more directly about representing time periods, while cooking and flight delay could be different feature target dependencies). This experiment does not isolate changes in the feature–target relationship, but demonstrates that the onset of performance degradation comes earlier on such training subsets, this was the core motivation behind this result -- to clearly show the same OOD degradation effect at smaller scale.

We've updated the wording in Section 5.3 to be more precise in this and mention the domain knowledge based subset creation.

Reviewer_agBnOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a controlled empirical study of training-data scaling in Tabular Machine Learning, evaluating the GBDTs, MLP-based models, and Tabular Foundation Models (TFMs) across 8 datasets, which contain up to 10M samples or training examples. The authors tried to investigate the scaling behavior under In-Distribution (ID) and Out-of-Distribution (OOD) evaluation, emphasizing model capacity, hyperparameter selection, and validation distribution. The main finding is that although additional training data generally improves prediction accuracy or performance in general, ID or OOD performance can diverge at larger scales. The study further suggests that target-aligned validation can reduce this divergence for MLP-based models, while adding validation examples to the context of TFMs does not necessarily prevent OOD degradation.

Critical strengths
  • The paper investigates an important but underexplored regime of large-scale tabular learning. Its controlled within-task scaling protocol, multiple training sizes, 10 seeds, and approximately 95700 MLP training runs represent a good amount of experimental effort.
  • Evaluating natural temporal and query-based shifts alongside ID controls is a meaningful contribution compared with conventional IID-only tabular evaluations.
  • The finding that ID-aligned model selection can improve ID performance while deteriorating OOD performance at increasing training sizes is potentially valuable, particularly for practical model deployment.
  • The authors provide detailed model configurations, preprocessing descriptions, hyperparameter search spaces, and supplementary experimental results.
Critical weaknesses
  • Limited conceptual and theoretical novelty. While reading the paper, it feels like the paper has narrow technical novelty with zero theoretical novelty. The paper primarily provides some empirical observations about scaling, model selection, and distribution shift. Although a controlled empirical study does not necessarily require theoretical guarantees, the work provides limited mechanistic explanation for why ID and OOD performance diverge or when this phenomenon should be expected. The power-law fitting analysis is descriptive rather than a theoretical justification.

  • Research motivations and narratives felt disconnected. The story building of the paper does not look coherent; it feels like some unconnected dots. The paper states that prior work motivates a deliberate study of data scaling in tabular ML but never gives any justification. The paper wants to tell a story about why MLPs or parametric models are still better than TFMs in some aspects. The motivation moves between data scaling, architectural improvements, model capacity, validation distributions, and differences between MLPs and TFMs. Although these topics are relevant, their relationship to a clearly formulated central hypothesis is not sufficiently established. The paper would benefit from a more coherent motivation explaining the precise research gap and how each experiment addresses it.

  • Incomplete consideration for scalability of TFMs. The paper states that TFMs can only take up to 1M samples. This claim is too broad without proper justification, as more recent works suggest that TFMs can be made compatible end-to-end with sample compression techniques. Please refer to the TACO paper from ICML 2026 Spotlight [1]. TACO demonstrates learned context compression and scalability to several million samples. Its omission weakens the discussion of foundation model scalability. While this does not directly invalidate the reported observations, the conclusions should be qualified or extended to account for scalable context/sample-compression strategies. I quote directly from the paper, “However, we see on all three datasets that using all the test-aligned validation examples in ICL does not save tabular foundation models from the “accuracy on the wrong line” like behaviour and, with the amount of in-distribution data increasing, out-of-distribution performance starts to degrade. MLP, which uses validation for model selection, is generally more stable, and even improves as training size grows at this scale, and this does not really depend on the validation set size." While TACO has shown how efficiently TFMs can handle millions of samples, the existence of TACO and its omission in the paper weakens this argument that TFMs struggle at higher sample sizes compared to MLPs or parametric models.

[1]. Zabërgja, Guri, et al. "End-to-End Compression for Tabular Foundation Models." ICML 2026 Spotlight.

  • Experimental protocols and evaluation fairness. Most of the non-TFMs were tuned, specifically dataset-specific hyperparameter tuning, except TabPack of Gorishniy et al., which is tuning-free. However, the paper does not mention which tuning method was used. The paper follows a lot of established practices of Gorishniy et al., and they usually use Optuna-based hyperparameter tuning [2]. Does the paper also use Optuna for their models? Also, the authors mentioned that the TFMs did not require any tuning, which is acceptable in some sense, as most of the TFMs recommend no tuning. But can the authors provide any reference which classifies tabular models based on their hyperparameter tuning requirements? Moreover, parametric models are examined up to approximately 10M training examples, while TFMs are evaluated only up to approximately 1M. Also, they are not compared head-to-head. Consequently, broader conclusions contrasting these paradigms are not supported by a fully controlled, budget-matched comparison.

[2]. Akiba, Takuya, et al. "Optuna: A Next-Generation Hyperparameter Optimization Framework." Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2019.

  • Feature dimensionality restriction. Although the study investigates large sample sizes, all 8 datasets contain fewer than 1000 features. What will be the case when you have a higher feature count with a higher sample count? For example, if you take ResNet-50 embeddings of large image benchmarks such as [3], [4], you will have 2048 features for each image sample. It therefore does not establish whether the findings generalize to high-dimensional large-scale tabular datasets, such as those involving thousands of learned visual embeddings or other high-dimensional representations. The evaluation also excludes multiclass classification, containing only regression and binary classification tasks.

[3]. https://huggingface.co/datasets/imageomics/TreeOfLife-10M

[4]. Guo, Yandong, et al. "MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition." ECCV 2016.

  • Limited justification for generalizing the scaling observations. Several conclusions are drawn from a relatively small set of 8 heterogeneous tasks. The dataset-shared power-law analysis relies primarily on in-sample fitting quality, without sufficient uncertainty quantification or validation of the fitted scaling exponents. This limits the strength of broader claims about shared scaling behavior.

  • Presentation and reporting inconsistencies. Several methodological details could be clearer, particularly the implementation of the TPE-based hyperparameter optimization and computational budgets. The acronym DL should be defined at first use. Some references lack consistent bibliographic metadata, including publication venues where applicable (see L531-L532, L537-L538, L540-L542, L594-L595, L597, L618-L626). The citation of Google's TabFM should also be checked against its official authorship and citation information. Please also see the table formatting in Page 17, where it looks like the table is not complete and a portion of it might be out of page length.

  • Missing computational resource reporting. Despite conducting approximately 95700 MLP training runs and experiments involving up to 10M samples, the authors do not report GPU specifications, computational budgets, training times, or hardware configurations. This omission makes it difficult to assess the computational feasibility, efficiency, and reproducibility of the proposed experimental study.

Questions for authors

Please see the questions raised in Critical Weaknesses part. Also take notes on the following questions:

  1. What is the main scientific novelty beyond existing scaling and distribution-shift studies? Can the authors explain the underlying mechanism of ID-OOD divergence? Apart from some empirical control studies, is there any theoretical justification?

  2. How do the authors justify the 1M-sample limit for TFMs given TACO [1], which demonstrates scalability to several million samples through context compression? The paper loses its ground in its central claims regarding the comparative performance and scalability of TFMs against MLPs when considering against TACO [1].

  3. Why are high-dimensional datasets (2K-10K+ features) and multiclass classification excluded? How generalizable are the findings?

  4. Which TPE optimization framework was used (e.g., Optuna [2]), and how were tuning budgets and early stopping controlled?

  5. Can the authors provide confidence intervals for the scaling-law exponents and assess whether ID-OOD divergence is statistically consistent across datasets?

  6. Is adding validation examples to a TFM's context a fair counterpart to explicit MLP model selection? Would alternative validation-guided strategies change the conclusions?

5 comments

(1/2) We thank the reviewer for their careful reading and helpful suggestions. Below, we address the concerns and present additional experiments, including the suggested evaluation of TACO. We've also updated the manuscript with additions highlighted in blue.

Limited conceptual and theoretical novelty

Q1: What is the main scientific novelty beyond existing scaling and distribution-shift studies? Can the authors explain the underlying mechanism of ID-OOD divergence? Apart from some empirical control studies, is there any theoretical justification?

Our contribution is empirical. We show that the relationship between ID and OOD performance can reverse as training data grows on real tabular tasks with natural distribution shifts. Additional data continues to improve ID performance while worsening OOD performance, and target-aligned model selection mitigates this effect. The pattern becomes particularly visible in the multi-million-example regime, beyond the scales typically studied before in tabular ML.

The analysis connects three observations. First, ID and OOD evaluation increasingly disagree on the benefits of model capacity. Second, this disagreement matters for validation-based selection: optimizing for ID performance can lead to worse OOD performance as data grows. Third, restricting training to less representative historical periods makes related performance decrease visible at smaller scale.

As discussed in our related work, Accuracy on the Wrong Line provides a theoretical account of how label noise and nuisance features can produce negative ID–OOD relationships. Our study examines related behavior on real tabular tasks and its consequences for model selection. We do not claim a new theoretical result or present the descriptive power-law fits as a theoretical explanation (it was used as a tool for quantifying the performance improvements in parametric models)

Research motivations and narratives felt disconnected. The story building of the paper does not look coherent; it feels like some unconnected dots. The paper states that prior work motivates a deliberate study of data scaling in tabular ML but never gives any justification. The paper wants to tell a story about why MLPs or parametric models are still better than TFMs in some aspects

Q6: Is adding validation examples to a TFM's context a fair counterpart to explicit MLP model selection? Would alternative validation-guided strategies change the conclusions?

Our goal is to understand how gains from additional tabular training data depend on distribution shift and model selection, not to establish the superiority of any paradigm.

The performance evaluation just demonstrate trends that motivate the further analysis, which are the core contribution. Additional data generally helps, but some foundation models perform worse at larger context sizes. The subsequent experiments examine whether similar behavior occurs in parametric models and find that it does occur when In-domain data is used for validation.

Validation-guided strategies for foundation models are possible. The simplest example is context selection with a validation signal -- this could improve results in Figure 6 in section 5.3 (new numbering), another example of foundation models providing validation-guidance is the proprietary TabPFNv3.5 thinking API -- which has the time_col/group_col in its signature, and we believe exploring this direction as we've outlined in conclusion: "This raises questions we find worth pursuing in future work. How should target-aligned data be split between fitting and model selection as in-domain data grows? Can foundation models be given an effective mechanism to treat validation examples differently from training examples?".

We’ve revised the closing discussion in Section 4.2 and the analysis introduction to make this progression explicit. The new “Performance Across Paradigms” section also shows the strong data efficiency of foundation models, which generally perform better at smaller training sizes.

Incomplete consideration for scalability of TFMs. The paper states that TFMs can only take up to 1M samples. This claim is too broad without proper justification, as more recent works suggest that TFMs can be made compatible end-to-end with sample compression techniques. Please refer to the TACO paper from ICML 2026 Spotlight

(2/2)

Q2: How do the authors justify the 1M-sample limit for TFMs given TACO

Thank you for pointing us to TACO. The roughly 1M range of the original evaluation should not have been presented as a general limit for foundation models.

We evaluated the released TACO classifier on Aeolus Flight Delay and Bank Fraud, using training sets up to 10M and 10.97M examples. We considered both a single-member configuration and an eight-member preprocessing ensemble. Both process all selected training examples through context compression.

TACO demonstrates that foundation models can process multi-million-example training sets. The ensemble configuration improves over the single-member configuration, although both show little further OOD improvement beyond 1M in these experiments. This distinguishes computational scalability from continued predictive gains; it does not establish a limitation of context compression in general.

We’ve added the results to “Foundation Model Scalability Beyond One Million Examples” and summarize them in the main text. We've also removed the outdated general-limit statement from the limitations.

Q3: Why are high-dimensional datasets (2K-10K+ features) and multiclass classification excluded? How generalizable are the findings?

Our dataset collection prioritizes real tabular tasks with multi-million-example training pools and meaningful temporal or query-based evaluation splits. Such datasets remain scarce, and the resulting collection covers regression and binary classification with fewer than 1,000 features.

The suggested image-embedding tasks would provide a useful extension, but they also introduce learned representations and a different task setting. We have not evaluated them and do not claim that our findings cover those settings.

We’ve made this scope explicit in the limitations and qualified the analysis conclusions accordingly. The experiments establish that the observed effects occur across several real tabular tasks, rather than their prevalence across all tabular problems.

Q4: Which TPE optimization framework was used (e.g., Optuna [2]), and how were tuning budgets and early stopping controlled?

We use Optuna’s TPESampler with 100 trials per method, dataset and training size, without intermediate-metric pruning or a common wall-clock limit. Configurations and checkpoints are selected using validation data, not test scores. TabPack instead samples 64 configurations and performs ensemble selection within a single pack.

MLP and TabM training uses validation-based early stopping with patience parameter 16. TabPack uses patience 16 for individual members and 32 for ensemble selection. XGBoost and LightGBM use at most 10,000 trees and stop after 300 rounds without validation improvement.

For the main foundation-model evaluation, we use the provided default configurations described in their respective model reports. This is an evaluation choice, not a claim that foundation models cannot benefit from tuning.

We’ve expanded these details in “Preprocessing and hyperparameter tuning” and identified Optuna in the main setup.

Presentation and reporting inconsistencies.

Thanks! we've updated the manuscript with fixes.

Missing computational resource reporting.

We've reported the computational resources in the source code. We've added a note with a summary of resources used in the Reproducibility statement section in the paper.

Q5: Can the authors provide confidence intervals for the scaling-law exponents and assess whether ID–OOD divergence is statistically consistent across datasets?

We’ve added 95% confidence intervals using 5,000 seed-block bootstrap replicates, refitting the individual and shared exponents in each replicate. On seven of eight datasets, the mean in-sample R² loss from sharing an exponent is below 0.01, including its upper confidence bound; Jane Street is the exception. This supports similar observed scaling trends without establishing that the exponents are equal.

We also test the joint claim that increasing training size improves ID performance and worsens OOD performance. After Holm correction across 16 dataset–model pairs, this effect is significant for MLP with Muon on Cooking Time, Weather, Jane Street, Bank Fraud and Aeolus Flight Delay, and for LightGBM on Cooking Time and Jane Street. The effect therefore exceeds variation across training runs in several settings, but is not universal.

The procedures and results are reported in “Shared scaling exponents across methods” and “Significance of ID–OOD divergence”. The intervals describe seed variation conditional on the fixed datasets and experimental setup.

Reviewer_agBnOpenReview ↗

Author Response Acknowledgement

Thank you for the additional experiments and for acknowledging that the original 10M-sample limit was incorrect for TFMs. The revised observation limited further OOD improvement for two TACO configurations on two datasets is substantially narrower than a general scalability limitation. Initially the paper’s narrative was that TFMs can’t handle 10M samples but with the acknowledgement of TACO, the narrative changed.

The comparison combines differences in model families with differences in adaptation protocols: MLPs receive validation-guided tuning and early stopping, whereas the evaluated TFMs primarily receive validation examples as additional context. Validation-guided TFM adaptation already exists in [1] and thus raises a valid novelty question. They explicitly use validation-based learning-rate selection and early stopping for TabPFNv2. This does not establish that adaptation would resolve your observed degradation, but it means the current comparison cannot isolate a model-family limitation.

As I initially mentioned in my review about “Limited conceptual and theoretical novelty”. The updated manuscript offers limited explanation of why the observed ID-OOD divergence occurs and when it should be expected. The reference to Sanyal et al. does not fully resolve this concern. Their theoretical results establish sufficient conditions for such behavior under a particular model of distribution shift and learned-model properties. Observing similar ID-OOD curves does not establish that those conditions, or the proposed mechanism, hold in your experiments. Inferring the mechanism from the observed pattern would reverse the direction of the theoretical implication. Could the authors provide evidence connecting their observations to the cited mechanism for example, whether increasing training size increases reliance on nuisance or spurious features or explicitly present this connection as a hypothesis? Likewise, the additional significance tests establish that the trends exceed seed-level variation, but do not identify their cause.

On image derived tabular representation part, MulTaBench [2] already establishes image and text-derived representations as a relevant setting for evaluating tabular learners, including TFMs. Freezing the encoder and holding the feature representation fixed across training sizes would preserve the proposed within-task scaling comparison.

  • [1]. Rubachev, Ivan, et al. "On Finetuning Tabular Foundation Models." arXiv preprint arXiv:2506.08982 (2025).
  • [2] Arazi et al. “MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Image.” NeurIPS 2026.
Reviewer_z1KqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a strictly controlled study on training-data scaling in tabular machine learning across eight datasets. Holding features and evaluation sets fixed, the authors combine natural distribution shifts with in-distribution controls. GBDTs and MLPs exhibit similar scaling trends, with architectural enhancements remaining beneficial across all scales. Newer tabular foundation models leverage larger context sizes more consistently. Crucially, as training data grows, models selected via in-distribution validation improve in-distribution performance but degrade out-of-distribution accuracy, an issue target-aligned validation effectively mitigates. Adding validation contexts helps foundation models but cannot prevent degradation under severe distribution shifts. Ultimately, this controlled study underscores the vital role of model selection and target-aligned data as tabular learning scales.

Critical strengths
  1. The paper evaluates MLP-based models, tabular foundation models, and gradient-boosted trees such as LightGBM. These cover three major approaches to tabular machine learning and provide a representative basis for studying data-scaling behavior across modeling paradigms.
  2. The evaluation includes both in-distribution (ID) and out-of-distribution (OOD) settings across eight datasets. Compared with evaluations based solely on random splits, this design provides a more informative assessment of model behavior under practical distribution shifts.
  3. The five main experiments provide a coherent analysis of training-set size, model capacity, validation distribution, and the use of validation data. The experimental design is well aligned with the paper’s objectives and highlights interactions between data scaling and model selection.
Critical weaknesses
  1. The foundation-model evaluation is limited to the TabPFN and TabICL families of 4 models only, although the Related Work chapter discusses other tabular foundation models. The two families also share architectural ideas, so comparing multiple versions may not adequately capture differences across architectures and pretraining approaches. Including additional models with distinct designs would help establish the scope of the conclusions.
  2. The eight tasks provide useful evidence for a controlled study, but the dataset and model coverage remains limited relative to the diversity of tabular prediction problems. In particular, the analyses of model capacity, validation distribution, and validation-data usage each focus on a small set of models. Even within the scope of a controlled study, additional cross-task or cross-model validation, or more narrowly stated conclusions, would strengthen the paper.
  3. The main results for parametric models and foundation models are presented separately in Figures 1 and 2. Although their maximum supported dataset sizes differ, they share a substantial evaluation range below one million examples. Except for Figure 5, the paper lacks direct comparisons within this common range under clearly controlled data availability and validation protocols. This limits the assessment of how scaling gains and performance gaps vary across paradigms.
  4. The incremental contribution over prior work needs to be more clearly established. Apart from the effect of validation distribution and its interaction with scale, several findings are consistent with existing observations about data scaling. TabPFN-3 already reports within-task training-data scaling experiments, while TabPFN-3.5 and TabICL discuss differences in the use of validation information. The paper should more clearly distinguish replication of known phenomena, broader experimental coverage, and new empirical findings.
  5. In Figure 5, foundation models use validation examples as additional context, whereas the MLP uses validation data for hyperparameter tuning and early stopping. These represent substantially different uses of validation information. For a pretrained in-context learning model, appending these examples amounts to expanding the context rather than using independent validation feedback for model selection. Therefore, matching the number of validation examples available to both model families does not constitute a controlled comparison of validation mechanisms.
Questions for authors
  1. Several curves exhibit non-monotonic behavior or noticeable fluctuations, including Jane Street in Figure 2, some capacity-scaling curves in Figure 3, and Aeolus Flight Delay and Cooking Time in Figure 4. Could the authors further investigate these behaviors? In particular, Figures 2 and 4 vary training-set size, whereas Figure 3 varies parameter count. Are these degradations associated with context-length generalization, distribution shift, overfitting, or optimization issues?
  2. What is the main motivation for evaluating both TabPFN v3 and v3.5, rather than the solely latest one? Comparing versions can help characterize changes in scaling behavior, but given the limited coverage of foundation-model families, could the authors explain the choice to prioritize multiple versions over models with different architectures or pretraining approaches?
3 comments

(1/2)

We thank the reviewer for their time and effort. We would like to address the concerns raised in the review. We've also updated the manuscript with new additions highlighted in blue (we mention the relevant additions in the response bellow too).

The foundation-model evaluation is limited to the TabPFN and TabICL families of 4 models only.

Q2: What is the main motivation for evaluating both TabPFN v3 and v3.5, rather than the solely latest one?

We believe that covering as broad of a spectrum of tabular foundation models is important (this includes variations that differ only in pretraining mixtures and model size too). At the time of the study we've selected the most capable and efficient (in terms of context-handling) models. We agree that adding architectural diversity is important too.

In the rebuttal experiments we've added a series of recently released Kumo Tabular foundation models and a scalable tabular foundation model in TACO (that utilizes context compression, thus has a widely different encoder-decoder like architecture compared to regular TFMs).

The additional results show encouraging progress. Kumo improves performance on many tasks, and TACO processes around 10M training examples. At the same time, Kumo also shows OOD performance declines at larger context sizes on some tasks, while TACO’s performance largely plateaus in the evaluated configurations, motivating the analysis in section 5.

We’ve added Kumo to the foundation-model evaluation and TACO to the appendix "Foundation Model Scalability Beyond One Million Examples".

Even within the scope of a controlled study, additional cross-task or cross-model validation, or more narrowly stated conclusions, would strengthen the paper.

We have clarified the scope of the conclusions and limitations. In particular, the analyses demonstrate that these effects can occur, rather than that they hold uniformly across all tabular models and tasks.

Wording changes in section 5, Conclusion and Limitation sections

The main results for parametric models and foundation models are presented separately in Figures 1 and 2... ...limits the assessment of how scaling gains and performance gaps vary across paradigms.

Our original protocol keeps validation sets fixed to study training-data scaling. Under that protocol, parametric models use validation for model selection, while foundation models receive only training examples in context. This difference, together with potentially large validation sets relative to small training subsets, motivated presenting the paradigms separately.

We agree that a direct comparison is useful. We now compare TabPack, TabPack with cosine embeddings, TabPFN v3.5 and Kumo large under matched task-specific data availability. Validation sizes decrease proportionally below 1M training examples. Foundation models receive training plus validation as context, while TabPack variants select on validation and then refit on training plus validation.

Foundation models generally perform better at smaller sizes, sometimes matching TabPack variants with over an order of magnitude fewer training examples. The gaps narrow and sometimes reverse as training size grows.

We’ve added the results to “Performance Across Paradigms” in section 4, and the protocol details to the corresponding appendix.

The incremental contribution over prior work needs to be more clearly established

The benefits of additional training data and the usefulness of validation data are established and expected findings. However, our works is the first large scale study across modern methods and natural distribution shifts at multi-million-example scales. And the core novel empirical contribution are the results demonstrating the interaction between scale, distribution shift and model selection. On real tabular tasks, we show that additional data can improve ID performance while worsening OOD performance, and the validation distribution determines whether this behavior appears (in both TFMs and parametric models). The capacity sweeps and restricted-training-pool experiments from section 5.3 examine this interaction from complementary directions.

We therefore distinguish confirmation of known effects (improvements from scale and adding validation to context), from the analysis of when those effects fail.

We’ve revised the closing discussion in Section 4.2 and the analysis introduction to make this connection clearer.

(2/2)

...matching the number of validation examples available to both model families does not constitute a controlled comparison of validation mechanisms.

This is actually an important message we would like the readers to takeaway after reading the section (and the paper).

Two paradigms currently differ in their use of explicit validation data. Only recently foundation models started to explicitly utilized validation-like structure (e.g. the very recent, proprietary release of TabPFN thinking, and its API asking for a time_col, or a group_col hints some validation-based selection). We believe that highlighting this "degree of freedom" and difference is important for future research of both more performant tabular foundation models that better adapt to OOD scenarios and improving the non-foundation models too. As we already mention in the conclusion: "This raises questions we find worth pursuing in future work. How should target-aligned data be split between fitting and model selection as in-domain data grows? Can foundation models be given an effective mechanism to treat validation examples differently from training examples?".

Q1 Several curves exhibit non-monotonic behavior or noticeable fluctuations. Are these degradations associated with context-length generalization, distribution shift, overfitting, or optimization issues?

Thanks for the question! It helps clarify the message we've outlined in the response above.

We distinguish the systematic OOD declines from smaller fluctuations and from the capacity-scaling curves. For training-data scaling, our explanation centers on the interaction between distribution shift and scale. Section 5.2 shows that parametric models can also exhibit declining OOD performance when selection favors ID accuracy (on the same datasets where we see non-monotonic behaviour for TFMs, look at Figure 11 in the Appendix for JaneStreet and cooking time results for tabpack for an additional example of these). Section 5.3 further restricts training to historical periods that are less representative of evaluation conditions; related declines then appear at sub-million training sizes. This connects our observations to Accuracy on the Wrong Line, which shows how additional data can exacerbate negative ID–OOD performance relationships.

The capacity experiment varies a different quantity: MLP width at fixed training sizes. Learning rate, weight decay and checkpoint are selected separately for each width, so these curves also reflect changes in the selected configuration (e.g. large enough models on small enough datasets overfit, and sometimes even exhibit a double-descent like curve shapes).

We’ve clarified this explanatory thread in the analysis introduction

Dear reviewer! The rebuttal discussion period is coming to an end. Could you please clarify if our additions and answers address your concerns?

Thanks again for your time and feedback

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response (summary of individual responses)

Dear Area Chair and Reviewers,

We thank you for the careful reviews and constructive suggestions. We appreciate the recognition of the study's "coherent analysis" of data size, capacity and validation (R1, z1Kq), its focus on an "important but underexplored regime" and findings that are "potentially valuable, particularly for practical model deployment" (R2, agBn), and its "Thorough empirical analysis of IID and OOD performance across different model families" (R3, 3Tax).

Below is a short overview of the main additions and clarifications in the revised manuscript, with pointers to the corresponding responses.

  1. Broader foundation-model coverage and scalability: added Kumo Tabular and TACO, including context-compression experiments on training sets of around 10M examples. Addressed in the response to R1's and R2's.

  2. Direct comparison across paradigms: added "Performance Across Paradigms", comparing TabPFN v3.5, Kumo large and two TabPack variants under matched task-specific data availability, with protocol details in the appendix. Addressed in the responses to R1 and R2.

  3. Clearer interpretation and scope: revised the transition from the foundation-model results to the analysis, clarified the connection between distribution shift, scale and model selection, and qualified the conclusions and limitations. Addressed in R1's novelty and non-monotonicity responses, R2's novelty and motivation responses, and R3's Q2/Q4 response.

  4. Statistical analysis: added confidence intervals for scaling exponents and fit quality, and significance tests of joint ID improvement and OOD deterioration across datasets and models. Addressed in R2's Q5 response.

  5. Sensitivity checks: added a batch-size comparison on all eight datasets and a comparison of random versus latest-observation training subsets on the seven temporal datasets. Addressed in R3's Q1 and Q3 responses.

  6. Experimental reporting: expanded tuning and early-stopping details, added runtime tables and a compute summary, and corrected the reported bibliographic and presentation issues. Addressed in the reporting responses to R2 and R3.

The revised manuscript uses blue highlighting to identify changes. We hope these additions and clarifications address the concerns raised, and we are happy to discuss any remaining questions.

Thank you again for your time and feedback!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.