- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to allocate finite pretraining compute across a synthetic task distribution to train tabular foundation models. The authors introduce a predictor-dependent score measuring the expected reduction in joint predictive log loss after a model observes a context dataset from a task. They call it task expected predictive information gain (TEPIG). High TEPIG indicates that a task contains predictive structure that the chosen proxy model can extract. Low TEPIG reflects either little structure or structure that the proxy can't exploit. The paper uses TEPIG to alter the pretraining prior, so tasks with higher TEPIG are sampled more during training. TEPIG is estimated with both stronger predictors and cheap proxy models. The method is evaluated first in a Gaussian-process setting, then on an SCM prior, and finally on a reduced-budget TabICLv2 regression prior. Across these settings, high-TEPIG tilting broadly accelerates learning, with the strongest effects occurring early in training and on high-TEPIG regions of the task distribution.
Critical strengths
-
TEPIG is well-motivated and practically useful. I like the distinction between learnable structure and irreducible difficulty. The paper notes that high predictive loss alone doesn't tell us whether a task warrants more training effort, since high loss may reflect noise. The score is explicitly model-dependent. I found this distinction useful, and the controlled examples clearly show that TEPIG is high when a predictor can extract structure from additional observations and low when the task is either already easy, too noisy, or too complex for that predictor.
-
The empirical results are broad, realistic, and careful. The authors evaluate TEPIG in a Gaussian-process setting where the true posterior predictive is available, then on a more complex SCM prior, and finally on a TabICLv2 regression prior. They also test whether cheap models such as a KNN can guide the reweighting, evaluate tilted models on the original prior, include a low-TEPIG control, validate their approximations, and report uncertainty consistently. High-TEPIG tilting consistently accelerates learning, which is a pretty strong result. This breadth and care make the empirical case pretty convincing.
Critical weaknesses
-
The paper’s compute-efficiency claim is not evaluated using total compute. The experiments compare models at matched optimizer steps, but TEPIG training requires generating tasks, scoring them with a proxy, and rejecting a lot of them before obtaining accepted training tasks. In the SCM experiments, some acceptance rates are around 12–15%, which is low. The appendix notes that matched-step comparisons don't account for the cost of model computation or of scoring and rejecting candidate tasks. Consequently, the paper shows TEPIG can reduce the number of target-model training updates, but it does not yet establish that it reduces total wall-clock time, FLOPs, or GPU cost. This is important because improved compute allocation is one of the paper’s main motivations.
-
The experiments don't fully isolate whether TEPIG itself is the important signal, as opposed to a simpler task-difficulty or curriculum heuristic. The paper includes an untilted baseline and a useful low-TEPIG control, but I don't see comparisons against simpler alternatives like raw proxy loss, endpoint improvement, a simple measure of task difficulty, or a random reweighting of comparable strength. This seems important. The current evidence supports the broader conclusion that proxy-guided task reweighting can help, but does not yet establish that TEPIG is particularly important.
Questions for authors
-
Referring to weaknesses, some comparison of compute cost as opposed to optimizer steps would be very useful.
-
Pick one simple hardness-aware reweighting, like endpoint improvement or raw proxy loss, and show that TEPIG beats it out in a meaningful way.
-
How many independent target-model training runs were used for the SCM and TabICLv2-prior experiments? The downstream gains are pretty modest, and a measure of training-run variability would be useful.
Good answers to 1 and 2 would raise my score.
4 comments
Response to Reviewer e4Uj (2/2)
- Pick one simple hardness-aware reweighting, like endpoint improvement or raw proxy loss, and show that TEPIG beats it out in a meaningful way.
Following your great suggestion, we now have introduced two new baselines.
- Tilting based on training loss (for the same proxy models used for TEPIG-tilting).
- The Extra Trees (ET) filter used for TabICL v2 [1].
We evaluate those alternative methods in the following setups:
- On the new GP setup (Appendix M.4, L2754) mentioned above. Here, we find that loss-based tilting leads to a higher final loss and slower convergence than untilted training. The ET filter proposed for TabICL v2 does not lead to a significantly lower final loss. It reduces the number of training steps needed to reach the untilted loss by 13.7%, compared to an almost twice as large reduction for TEPIG-based tilting.
- On the SCM prior (Appendix P.4, L3605), we also find that loss-based tilting substantially hurts performance and convergence speed. The ET filter from TabICL v2 improves in-prior performance early in training, then falls below untilted training, and ends with only a small improvement.
- In a new prior with a step-function component (Appendix O, L3374), the ET filter from TabICL v2 rejects almost no datasets and therefore has no effect, while TEPIG-based tilting reduces the number of training steps needed by 41% compared to the untilted prior.
- On the TabICL v2 regression prior (Appendix Q.6, L4364) we now explicitly compare TEPIG-based tilting and the ET filter and find that TEPIG-based tilting yields a consistently lower in-prior loss throughout training, with the largest gains early on. It also improves real-world performance (Appendix Q.8, L4474) on 26 out of 39 additional datasets from two established tabular benchmarks.
- In new experiments on the TabICL v2 classification prior (Appendix Q.4, L4212) we also find TEPIG-based tilting to perform better than the ET filter.
Standard curriculum learning methods [2] such as [3] are not directly applicable because they select examples from a fixed, finite training set using each example's training history under the model being trained (in [3], its residual and how fast it changes). In TFM pretraining, datasets are freshly sampled from the prior, so no such per-example history exists. Loss-based tilting is the closest applicable analogue, since it prioritises datasets by a difficulty score. We now also discuss curriculum learning in the updated manuscript (Section 3, L177).
- How many independent target-model training runs were used for the SCM and TabICLv2-prior experiments?
Due to the high computational cost (around 128 GPU (A100) hours for our TabICL v2 experiment and 40 GPU hours for each SCM run), we have only conducted one run per tilting method so far for those priors, while running larger numbers of seeds for the GP-experiments.
However, in response to your question we ran 5 additional KNN tilting and 2 additional TabICL v2 tilting runs on the SCM prior and 2 more runs on the TabICL v2 prior. Our new results on the SCM prior (Appendix P.5, L3726) as well as our new results for TabICL v2 (Appendix Q.5, L4320) show the significance of the improvement across seeds.
Please let us know if you have any further questions or remaining concerns; we are happy to address them. If not, we would be grateful if you would consider raising your score.
References
[1] Jingang Qu et al. TabICLv2: A Better, Faster, Scalable, and Open Tabular Foundation Model. ICML 2026
[2] Yoshua Bengio et al. Curriculum Learning. ICML 2009
[3] Tianyi Zhou et al. Curriculum Learning by Optimizing Learning Dynamics. AISTATS 2021
Thank you for your response
Thank you for the detailed response. The compute accounting in Appendix K resolves my first concern, and the loss-based tilting and ET-filter comparisons substantially address my second. I am tempted to increase my score. However, after reading the other reviews and the discussion, I share concerns about the modest real-world improvements.
On balance, I would ideally like to increase my score to a 3.5, but the ICLR reviewing system doesn't allow for that level of granularity. I am keeping my score at 3, but I view the paper more favorably after the revision, and I ask that the AC reviewing this paper consider my review in that light. This paper is a well-substantiated delta over what came before that opens up avenues for future research.
Response to Reviewer e4Uj's follow-up
Thank you for your reply!
We are very happy to hear that the additional compute accounting resolved your first concern and that the new baseline comparisons substantially addressed your second!
We also appreciate your more favorable assessment and your clarification that you would raise your score to 3.5 if this option were available.
Regarding the real-world improvements, we see our central contribution as proposing a measure of learnable predictive structure and showing evidence that it can guide more efficient allocation of training steps for a given prior. How much these improvements translate to real-world datasets depends on how well the chosen prior describes those datasets. We discuss this point in more detail in our response to Reviewer UPNv (part 2/2, in our answer to “with no significant difference on TabArena”).
We also believe that our extended real-world evaluation (Appendix Q.8, L4474) provides encouraging evidence on real-world data where TEPIG-based tilting improves over the original filter designed for TabICL v2 on 26 of 39 additional datasets.
Moreover, we think that our findings, both on prior analysis (Section 5, L297) and on more effective training-step allocation, motivate future work on the role of TEPIG in prior design, especially regarding the question of how well a synthetic prior matches real-world data.
Thank you again for the constructive discussion, and for recognizing the avenues for future research that this work opens up.
Response to Reviewer e4Uj (1/2)
Thank you for your review!
We are very happy to hear that you think that "TEPIG is well-motivated and practically useful." and "The empirical results are broad, realistic, and careful. "
We address your questions below and have also updated the manuscript with the content in the linked documents. Section, figure, table and line numbers (L) refer to the updated manuscript.
To keep our answers as concise as possible, we do not answer the weaknesses you mention again since they are already addressed by your questions.
Questions
In terms of overall wall-clock training time, we generally train in the same amount of time with and without tilting. This is because the rejection sampling and data-generation can be done in parallel to training by running multiple CPU cores at once that generate data and compute the TEPIG scores.
In terms of overall FLOPs, model training costs roughly 2.3⋅1012 FLOPs per training dataset on the SCM prior and 3.6⋅1011 FLOPs per training dataset on the TabICL v2 prior. Training-data generation with the strongest KNN-based tilting (τ=20 and τ=2.5, corresponding to low acceptance rates) costs roughly 6⋅109 (SCM prior) and 1.2⋅1010 (TabICL v2 prior) operations per training dataset on average, i.e. 0.25% and 3% of training compute. Without tilting, data generation accounts for 0.003% and 0.13% of training compute. Any savings in the number of training steps therefore directly translate to savings in GPU compute and wall-clock time and almost directly translate to savings in FLOPs since the CPU compute fraction is small overall.
The time of computing TEPIG with the proposed cheap KNN tilting is of the same order as data generation itself. On one CPU core (AMD EPYC 7662 from 2020), scoring one dataset takes about 0.22 s on the SCM prior (512 context and 512 query points) and 0.75 s on the TabICL v2 prior (1024 context and 512 query points). Generating one dataset takes about 0.18 s with the SCM prior and 0.45 s with the TabICL v2 prior.
We detail our calculations and results here (Appendix K, L1971).