- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
In this paper, the authors provide a thorough empirical investigation of scaling training data in the domain of tabular machine learning. They take GBDTs, MLP-based models as well as tabular foundation models into account. The experiments are framed to investigate the the effects of in-distribution and out-of-distribution generalization as the amount of training data grows. Their empirical study highlights the effect of iid performance and how it translates to generalization under some distribution shifts. Whereas GBDT and MLP-based models show better generalization with increasing data samples considered, we also see the accuracy on the wrong line effect. TFMs on the other hand are more prone towards more variation in the performance when scaling input data.
Critical strengths
S1: Thorough empirical analysis of IID and OOD performance across different model families
The authors provide a thorough empirical study on the performance of models and their iid and ood performance. We observe diverse behavior of parametric models and GBDTs compared to TFMs. For the first, we observe the dependency of the model selection to the iid validation against ood validation, where for the latter the validation samples considered in their context does show opposite behavior.
S2: Scaling analysis for TabPack
That component was missing in the analysis of the TabPack paper and also the diversity issues of the models considered in the ensemble. A take-away that needs for inspection for the community is to see how the improvements in IID performance can reliably be translated under some distributions shifts.
Critical weaknesses
W1: Some hyperparameters and setting variations are left out from the analysis
The empirical study already provides a broad inspection of hyperparameters that are validated and compared against each other. We see a thorough scaling of the models’ capacity, adapted learning rates and weight decays. What I’d be curious to see, that has also been figured out for foundation models - also from other domains - that they not only subdivide the learning procedure itself into different stages, but also be curious about the learning rate schedulers and the batch size used for training. here, we see that the e.g. the batch size is fixed across all different settings. My question would be if the authors conducted an initial analysis or runs where these choices are justified with.
W2: Meta feature analysis of the datasets
From the different benchmark datasets used in the evaluation, we see in figure 1 a rather homogenoeus behavior for parametric models, but we see more deterioations w.r.t. the performance for TFM in figure 2. The insights provided in Section 4.2. are rather very general with stating just “differ substantially” without inspecting this behavior in more detail. To answer this question, I’d like to ask the authors if they did for example a more in-depth analysis in the meta-features the datasets potentially share, i.e., do datasets like cooking time, weather and bank fraud potentially share some characteristic that lead to a similar behavior, whereas for datasets like delivery eta and istella we see that tabpfn v3.5 fast and tabicl diverge from the performance of tabpfn v3, and in jane street potentially we have a meta feature constellation not shared by any of the other datasets leading to this divergent behavior of tabicl v2 (maybe shared a bit with aelous flight delay)
W3: Scaling criteria on the iid/ood performance not fully clear
The empirical analysis is built on the idea to scale the training data starting from thounds and scaling to millions of data points. What is not fully clear to me, is how the scaling itself is done. Let’s consider for example the aeolus flight delay dataset, where it is stated that the samples from 2016-2022 were leveraged for the training phase. But which points were now selected from the training data. For simplicity, assume we take 10,20,30%,… from the data. Is the time ordering considered meaning that you first take the 10% most recent data samples and expand it backwards in time, do you start from the beginning of the considered time period (i.e. starting with the first 10% of samples starting from 2016), or do you considered 10% from more or less randomly selected short time intervals spanning the whole time period and expanding the intervals step-wise till you reach 100%? Maybe the assumption is to take the most recent once and extend it backward as the assumption is this distribution rather fits with the validation/test set. Maybe the authors can clarify this setup (potentially I missed it).
W4: Limited explanation of the diverging performance
Speaking of W2 and W3, I’d like to stress the point of explanation a bit more, as in the current version of the manuscript, one could also inspect if the models are over-confident w.r.t. to some spurious features, or even - due to the time-dependency W3 - do we also observe some changes or drifts in the feature-target relationships? Maybe the authors gained some insights already with the conducted experiments but due to no pattern that arised w.r.t. theh deteriotations, they were potentially not reported. However, I’d be curious also to see a bit more of a targeted discussion regarding this point.
Minor:
- 033: “its”
- 323: post-hoc
- 377: between
- 310: missing white space between “in (Gorishniy et al. […]”
Formatting issues:
- Table below Figure 8 is cropped due to ending of the page…not readable
Questions for authors
- Q1: Did the authors conduct experiments w.r.t. some hyperparameters and design choices that were not further changed and assumed to be fixed across all experiments (batch size, learning rate schedulers, learning stages for increased number of training samples)?
- Q2: Did the authors observe or inspect and meta feature characteristics that might yield an explanation of the deterioration across some benchmarks that impacts the models’ performance?
- Q3: Can the authors clarify the scaling choice in more detail?
- Q4: Can the authors provide a more in-depth investigation on the effects stemming potentially also from some shifts in the feature-target relationship and how it might impact a model’s performance.
4 comments
(2/2)
Scaling criteria on the iid/ood performance not fully clear
Q3: Can the authors clarify the scaling choice in more detail?
Our main experiments use random subsets of a fixed training pool. We generate one random permutation of the training rows and take its first entries for training size . Each smaller training set is therefore contained in every larger one, and all methods use the same subset at a given size. Validation and test sets remain fixed; ID holdouts are reserved before training subsets are constructed.
We've now also evaluated the alternative suggested by the reviewer: selecting the latest training observations and expanding backward in time. This comparison uses TabPack with Muon on the seven temporal datasets, with ten evaluation seeds, the same fixed temporal validation sets, and no refitting. Istella is excluded because its split is query-based.
We've added the results to Appendix "Training Subsampling Strategies".
Selecting the latest observations generally performs worse than random sampling on these tasks. Cooking Time is a notable exception: recent-only training improves performance at intermediate training sizes. This supports the importance of temporal representativeness on this dataset and connects to the experiment in Section 5.3, where we restrict training to the earliest 25% of observations. Adding more examples from that fixed early period does not restore the missing recent coverage, and TabPFN v3.5 exhibits increasing OOD deterioration.
More generally, proximity in time does not consistently make a training subset more useful: recent windows can also lose relevant historical or seasonal coverage. Unlike random sampling from a fixed pool, expanding a recent window changes both sample size and the periods represented. This motivates our main protocol, which separates the effect of training-set size from changes in temporal coverage.
Minor + Formatting issues
Thank you for identifying the issues. We've fixed them in the updated manuscript.
Q1: Thank you very much for providing the new insights into the batch sizes. Even though, we see now that the batch size does not have a major impact for TabPacks performance, I’m still curious to inspect the raised question a bit more in-depth if it holds across different model architectures. Recent studies have shown the impact for LLMs, and in future work we should also shine a light on the training dynamics in TFMs.
Q2: Thanks for providing the additional discussion. So, we have empirical evidence for what you have described, but we are lacking the insights into the concerns raised initially, meaning that we currently do not know the linkage of dataseet properties to the scaling curves. This encompass type and scale of the features. Also with the discussion of the prior, which indeed is a crucial element of the tfm performance capabilities, it remains an open issue at what scale we can expect a prior to cover also real-world problems.
Q3: Thanks for clarifying and providing the additional experiments that random sampling apparently outperforms a sampling heuristic based on chronological subsampling starting with the most recent observaitons.
Q4: How did you check that the historical periods are less representative of the evaluations? So with respect to what? Like an inherent assumption based on seasonal effects, i.e., domain-specific assumptions? Is this the key criteria? So, then we cover it from the distributional representativeness point of view but not shifts with respect to any feature-target relationships (e.g. a feature might become more/less important over time; other feature compositions provide more predictive strengths with scaling data, etc.).
Minor: Acknowledged
—
Thanks to the authors for elaborating on those points.
Thanks for the follow-up!
Q1:
For TFMs, we vary the number of examples in context while keeping the pretrained weights fixed. Batch size and optimization during TFM pretraining are separate questions worth studying in the future.
Q2:
We agree that how feature types, scales and dimensionality affect the scaling curves remains an open question (we state this in the section 4.2 already). Our main finding is that gains from additional training data can fail to transfer under distribution shift, and that the choice of validation distribution matters. For future TFM pretraining, we believe it would be useful to vary both context size and the degree of shift between context and query examples, and test whether this leads to more consistent gains from larger contexts on real tasks.
Q4:
The windows were chosen based on domain knowledge, not a quantitative measure of representativeness / shift. We narrowed seasonal coverage on Weather, kept the pandemic-era period on Aeolus Flight Delay, and excluded more recent observations on Cooking Time, while keeping validation and test sets fixed. The experiment shows that which periods are represented in training matters for OOD performance as data grows (on each task this may relate to a different kind of shift, e.g. weather is more directly about representing time periods, while cooking and flight delay could be different feature target dependencies). This experiment does not isolate changes in the feature–target relationship, but demonstrates that the onset of performance degradation comes earlier on such training subsets, this was the core motivation behind this result -- to clearly show the same OOD degradation effect at smaller scale.
We've updated the wording in Section 5.3 to be more precise in this and mention the domain knowledge based subset creation.
(1/2)
We thank the reviewer for the careful reading and constructive suggestions! We address questions and weaknesses below, we've also uploaded the updated manuscript with changes highlighted in color.
We choose batch size based on training-set size, using a grid similar to one used in (TabArena). The same rule applies throughout the main experiments, except for the model-capacity sweeps, which use a fixed batch size.
To assess sensitivity to this choice, we additionally evaluated TabPack with Muon at batch sizes 1024, 2048, 4096, and 8192 on all eight datasets at the largest training sizes. We repeated configuration and ensemble selection for each batch size while keeping the remaining protocol unchanged. The differences were generally small and dataset-dependent, with no batch size consistently performing best. Our default of 4096 was within 0.5% of the best mean test score on every dataset. We’ve added the experiment and full results to the appendix "Batch Size Parameter".
The learning rate schedule are at odds with the early-stopping which is used for every DL model in the parametric models section, so we chose not to study it. Furthermore, we already study the imporvements in optimization by comparing AdamW and Muon. The general observation that improvements transfer at this dataset scale makes us believe that scheduling could help too if the early-stopping issue is dealt with, but we believe it is out of scope for the current study and the optimizer experiments are sufficient.
Thank you for these questions. Our explanation centers on the interaction between distribution shift and training-data scale. Sections 5.2 and 5.3 were actually motivated by the non-monotonic foundation-model performance on Cooking Time and Jane Street in the begging of the project, with the additional Kumo results showing related behavior on Aeolus Flight Delay.
As we've discussed in related work, Accuracy on the Wrong Line links negative ID–OOD performance correlations to label noise and nuisance features, and shows that additional training data can exacerbate this behavior (Sanyal et al., 2025). This together with the observation of degradations of TFM performance motivates the analysis in sections 5.2 and 5.3. We first see that parametric models also degrade on the same exact datasets when the methods are optimized for ID accuracy (see Figure 11 in the updated manuscript in appendix for Cooking Time, Jane Street and Aeolus Flight delay results). Section 5.3 probes the role of distribution shift further. We restrict training to historical periods that are less representative of the evaluation conditions, while keeping validation and test sets unchanged. These restrictions reduce seasonal coverage on Weather, isolate the pandemic period on Aeolus Flight Delay, and exclude recent observations on Cooking Time; Appendix B describes the construction. Under these restrictions, deterioration appears at sub-million training sizes. Together, these experiments support distribution shift as a central factor in the observed deterioration, with its impact becoming more pronounced as training data grows.
Separately, we believe synthetic-prior coverage helps explain some dataset-specific performance differences across TFMs. Maps Routing has substantially more features than the other tasks, while Istella is a regression task derived from pointwise learning-to-rank data with only five target values. These distinctive properties help explain why some pretrained models perform poorly on these tasks while TabPFN v3.5 and Kumo perform substantially better.
We’ve revised the closing discussion in Section 4.2 and the introduction to the analysis section to make the connection between foundation-model performance declines, distribution shift and model selection explicit.