acceptodds
Under review as a conference paper at ICLR 2027

Measuring Structure in Tabular Foundation Model Priors

Abstract

Tabular Foundation Models (TFMs) are trained to predict labels given in-context datasets and queries sampled from tasks (data-generating processes). These tasks are sampled from diverse but synthetic priors. The prior determines how often different tasks appear during pretraining, hence the finite computation spent training on them. However, it is unclear from the various choices in designing a prior, how much compute to allocate to each task. Motivated by the notion of epiplexity, we characterise the learnable predictive structure accessible to a model to help better allocate compute across tasks. To do this, we introduce task expected predictive information gain (TEPIG), a model-dependent measure of the expected reduction in joint predictive log loss obtained by observing training datapoints from a task. We illustrate its behaviour in controlled examples and test the hypothesis that TEPIG, when measured with cheaper proxy models, can guide the allocation of pretraining compute for TFMs. In a tractable Gaussian process setting, TEPIG-guided sampling improves predictive accuracy and convergence when evaluated under the original prior. On more complex priors, including a state-of-the-art prior used for open tabular foundation models, it allows for training in fewer steps. These findings demonstrate TEPIG’s practical value for measuring model-dependent structure in tabular foundation model priors and suggest its potential as a tool for more efficient compute allocation for TFM training.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Tabular Foundation Models, TabPFN, Epiplexity, Priors, Information Gain

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.60 over 5 reviews, above 70% of the others.
Reviewer_e4UjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to allocate finite pretraining compute across a synthetic task distribution to train tabular foundation models. The authors introduce a predictor-dependent score measuring the expected reduction in joint predictive log loss after a model observes a context dataset from a task. They call it task expected predictive information gain (TEPIG). High TEPIG indicates that a task contains predictive structure that the chosen proxy model can extract. Low TEPIG reflects either little structure or structure that the proxy can't exploit. The paper uses TEPIG to alter the pretraining prior, so tasks with higher TEPIG are sampled more during training. TEPIG is estimated with both stronger predictors and cheap proxy models. The method is evaluated first in a Gaussian-process setting, then on an SCM prior, and finally on a reduced-budget TabICLv2 regression prior. Across these settings, high-TEPIG tilting broadly accelerates learning, with the strongest effects occurring early in training and on high-TEPIG regions of the task distribution.

Critical strengths
  1. TEPIG is well-motivated and practically useful. I like the distinction between learnable structure and irreducible difficulty. The paper notes that high predictive loss alone doesn't tell us whether a task warrants more training effort, since high loss may reflect noise. The score is explicitly model-dependent. I found this distinction useful, and the controlled examples clearly show that TEPIG is high when a predictor can extract structure from additional observations and low when the task is either already easy, too noisy, or too complex for that predictor.

  2. The empirical results are broad, realistic, and careful. The authors evaluate TEPIG in a Gaussian-process setting where the true posterior predictive is available, then on a more complex SCM prior, and finally on a TabICLv2 regression prior. They also test whether cheap models such as a KNN can guide the reweighting, evaluate tilted models on the original prior, include a low-TEPIG control, validate their approximations, and report uncertainty consistently. High-TEPIG tilting consistently accelerates learning, which is a pretty strong result. This breadth and care make the empirical case pretty convincing.

Critical weaknesses
  1. The paper’s compute-efficiency claim is not evaluated using total compute. The experiments compare models at matched optimizer steps, but TEPIG training requires generating tasks, scoring them with a proxy, and rejecting a lot of them before obtaining accepted training tasks. In the SCM experiments, some acceptance rates are around 12–15%, which is low. The appendix notes that matched-step comparisons don't account for the cost of model computation or of scoring and rejecting candidate tasks. Consequently, the paper shows TEPIG can reduce the number of target-model training updates, but it does not yet establish that it reduces total wall-clock time, FLOPs, or GPU cost. This is important because improved compute allocation is one of the paper’s main motivations.

  2. The experiments don't fully isolate whether TEPIG itself is the important signal, as opposed to a simpler task-difficulty or curriculum heuristic. The paper includes an untilted baseline and a useful low-TEPIG control, but I don't see comparisons against simpler alternatives like raw proxy loss, endpoint improvement, a simple measure of task difficulty, or a random reweighting of comparable strength. This seems important. The current evidence supports the broader conclusion that proxy-guided task reweighting can help, but does not yet establish that TEPIG is particularly important.

Questions for authors
  1. Referring to weaknesses, some comparison of compute cost as opposed to optimizer steps would be very useful.

  2. Pick one simple hardness-aware reweighting, like endpoint improvement or raw proxy loss, and show that TEPIG beats it out in a meaningful way.

  3. How many independent target-model training runs were used for the SCM and TabICLv2-prior experiments? The downstream gains are pretty modest, and a measure of training-run variability would be useful.

Good answers to 1 and 2 would raise my score.

4 comments
AuthorsOpenReview ↗

Response to Reviewer e4Uj (1/2)

Thank you for your review!

We are very happy to hear that you think that "TEPIG is well-motivated and practically useful." and "The empirical results are broad, realistic, and careful. "

We address your questions below and have also updated the manuscript with the content in the linked documents. Section, figure, table and line numbers (L) refer to the updated manuscript.

To keep our answers as concise as possible, we do not answer the weaknesses you mention again since they are already addressed by your questions.

Questions

  1. Referring to weaknesses, some comparison of compute cost as opposed to optimizer steps would be very useful.

In terms of overall wall-clock training time, we generally train in the same amount of time with and without tilting. This is because the rejection sampling and data-generation can be done in parallel to training by running multiple CPU cores at once that generate data and compute the TEPIG scores.

In terms of overall FLOPs, model training costs roughly 2.3⋅10122.3·10^{12} FLOPs per training dataset on the SCM prior and 3.6⋅10113.6·10^{11} FLOPs per training dataset on the TabICL v2 prior. Training-data generation with the strongest KNN-based tilting (τ=20\tau = 20 and τ=2.5\tau = 2.5, corresponding to low acceptance rates) costs roughly 6⋅1096·10^9 (SCM prior) and 1.2⋅10101.2·10^{10} (TabICL v2 prior) operations per training dataset on average, i.e. 0.25% and 3% of training compute. Without tilting, data generation accounts for 0.003% and 0.13% of training compute. Any savings in the number of training steps therefore directly translate to savings in GPU compute and wall-clock time and almost directly translate to savings in FLOPs since the CPU compute fraction is small overall.

The time of computing TEPIG with the proposed cheap KNN tilting is of the same order as data generation itself. On one CPU core (AMD EPYC 7662 from 2020), scoring one dataset takes about 0.22 s on the SCM prior (512 context and 512 query points) and 0.75 s on the TabICL v2 prior (1024 context and 512 query points). Generating one dataset takes about 0.18 s with the SCM prior and 0.45 s with the TabICL v2 prior.

We detail our calculations and results here (Appendix K, L1971).

AuthorsOpenReview ↗

Response to Reviewer e4Uj (2/2)

  1. Pick one simple hardness-aware reweighting, like endpoint improvement or raw proxy loss, and show that TEPIG beats it out in a meaningful way.

Following your great suggestion, we now have introduced two new baselines.

  • Tilting based on training loss (for the same proxy models used for TEPIG-tilting).
  • The Extra Trees (ET) filter used for TabICL v2 [1].

We evaluate those alternative methods in the following setups:

  • On the new GP setup (Appendix M.4, L2754) mentioned above. Here, we find that loss-based tilting leads to a higher final loss and slower convergence than untilted training. The ET filter proposed for TabICL v2 does not lead to a significantly lower final loss. It reduces the number of training steps needed to reach the untilted loss by 13.7%, compared to an almost twice as large reduction for TEPIG-based tilting.
  • On the SCM prior (Appendix P.4, L3605), we also find that loss-based tilting substantially hurts performance and convergence speed. The ET filter from TabICL v2 improves in-prior performance early in training, then falls below untilted training, and ends with only a small improvement.
  • In a new prior with a step-function component (Appendix O, L3374), the ET filter from TabICL v2 rejects almost no datasets and therefore has no effect, while TEPIG-based tilting reduces the number of training steps needed by 41% compared to the untilted prior.
  • On the TabICL v2 regression prior (Appendix Q.6, L4364) we now explicitly compare TEPIG-based tilting and the ET filter and find that TEPIG-based tilting yields a consistently lower in-prior loss throughout training, with the largest gains early on. It also improves real-world performance (Appendix Q.8, L4474) on 26 out of 39 additional datasets from two established tabular benchmarks.
  • In new experiments on the TabICL v2 classification prior (Appendix Q.4, L4212) we also find TEPIG-based tilting to perform better than the ET filter.

Standard curriculum learning methods [2] such as [3] are not directly applicable because they select examples from a fixed, finite training set using each example's training history under the model being trained (in [3], its residual and how fast it changes). In TFM pretraining, datasets are freshly sampled from the prior, so no such per-example history exists. Loss-based tilting is the closest applicable analogue, since it prioritises datasets by a difficulty score. We now also discuss curriculum learning in the updated manuscript (Section 3, L177).

  1. How many independent target-model training runs were used for the SCM and TabICLv2-prior experiments?

Due to the high computational cost (around 128 GPU (A100) hours for our TabICL v2 experiment and 40 GPU hours for each SCM run), we have only conducted one run per tilting method so far for those priors, while running larger numbers of seeds for the GP-experiments.

However, in response to your question we ran 5 additional KNN tilting and 2 additional TabICL v2 tilting runs on the SCM prior and 2 more runs on the TabICL v2 prior. Our new results on the SCM prior (Appendix P.5, L3726) as well as our new results for TabICL v2 (Appendix Q.5, L4320) show the significance of the improvement across seeds.

Please let us know if you have any further questions or remaining concerns; we are happy to address them. If not, we would be grateful if you would consider raising your score.

References

[1] Jingang Qu et al. TabICLv2: A Better, Faster, Scalable, and Open Tabular Foundation Model. ICML 2026

[2] Yoshua Bengio et al. Curriculum Learning. ICML 2009

[3] Tianyi Zhou et al. Curriculum Learning by Optimizing Learning Dynamics. AISTATS 2021

Reviewer_e4UjOpenReview ↗

Thank you for your response

Thank you for the detailed response. The compute accounting in Appendix K resolves my first concern, and the loss-based tilting and ET-filter comparisons substantially address my second. I am tempted to increase my score. However, after reading the other reviews and the discussion, I share concerns about the modest real-world improvements.

On balance, I would ideally like to increase my score to a 3.5, but the ICLR reviewing system doesn't allow for that level of granularity. I am keeping my score at 3, but I view the paper more favorably after the revision, and I ask that the AC reviewing this paper consider my review in that light. This paper is a well-substantiated delta over what came before that opens up avenues for future research.

AuthorsOpenReview ↗

Response to Reviewer e4Uj's follow-up

Thank you for your reply!

We are very happy to hear that the additional compute accounting resolved your first concern and that the new baseline comparisons substantially addressed your second!

We also appreciate your more favorable assessment and your clarification that you would raise your score to 3.5 if this option were available.

Regarding the real-world improvements, we see our central contribution as proposing a measure of learnable predictive structure and showing evidence that it can guide more efficient allocation of training steps for a given prior. How much these improvements translate to real-world datasets depends on how well the chosen prior describes those datasets. We discuss this point in more detail in our response to Reviewer UPNv (part 2/2, in our answer to “with no significant difference on TabArena”).

We also believe that our extended real-world evaluation (Appendix Q.8, L4474) provides encouraging evidence on real-world data where TEPIG-based tilting improves over the original filter designed for TabICL v2 on 26 of 39 additional datasets.

Moreover, we think that our findings, both on prior analysis (Section 5, L297) and on more effective training-step allocation, motivate future work on the role of TEPIG in prior design, especially regarding the question of how well a synthetic prior matches real-world data.

Thank you again for the constructive discussion, and for recognizing the avenues for future research that this work opens up.

Reviewer_g3eTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TEPIG, a predictor-dependent measure of the predictive benefit obtained from conditioning on task observations, and uses it to guide task-sampling frequencies during tabular foundation model pretraining. The experiments show promising improvements in training efficiency on regression tasks.

Critical strengths

The paper introduces an intuitive, task-level measure of predictive benefit that is applicable when the prior's exact posterior predictive distribution is unavailable. It offers a practical mechanism that leverages cheap proxy models to guide task sampling. Empiricall, reports on regression benchmarks show improvements in training efficiency.

Critical weaknesses

Limited novelty. TEPIG builds on EPIG, and the main contribution of the paper lies in applying a task-level variant to prior reweighting. The authors should clearly position the novelty of TEPIG against existing information-based data selection methods or curriculum learning [1].

Lacks theoretical guarantees. The main claim of the paper is that tasks with higher TEPIG have higher returns on pretraining compute, which lacks theoretical justification on why this is the case.

Predictive-convergence assumption. In line 260-262, the authors assume that the predict loss has approximately converged by context size nn. This requires more empirical validation, including showing the accuracy across context sizes and tasks.

Experiments. The experiments focus on regression priors and provide limited evidence for larger scale training, other kinds of tasks and the computational costs of storing and rejecting tasks. Moreover, the experiments lack comparisons with relevant baselines such as EPIG or curriculum learning [1].

[1] Curriculum Learning by Optimizing Learning Dynamics. Tianyi Zhou, Shengjie Wang, Jeff Bilmes Proceedings of The 24th International Conference on Artificial Intelligence and Statistics

Questions for authors

See Weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer g3eT (1/3)

Thank you for your review! We would like to thank you for pointing out that "The paper introduces an intuitive, task-level measure of predictive benefit that is applicable when the prior's exact posterior predictive distribution is unavailable."

We address your questions below and have also updated the manuscript with the content in the linked documents. Section, figure, table and line numbers (L) refer to the updated manuscript.

Weaknesses

Limited novelty. [...]

In this paper, we tackle the problem of choosing where to place prior mass on complicated mixtures of synthetic data-generating processes (DGPs) from which datasets are sampled for TFM training. Specifically, the question we answer is: How should we change the prior mass over this mixture of DGPs such that TFM training is efficient (lower loss with fewer datasets seen)? Intuitively, our starting insight is that TFM training will most benefit if we show it fewer of "simple" predictive tasks so that it can spend more time learning "harder" predictive tasks.

Our technical contribution is to formalise this intuition building on ideas from information theory and active learning (specifically EPIG [2] and V-information [4] / Epiplexity [3]). Consequently, we introduce TEPIG - the expected reduction in a predictor qq’s joint predictive log loss after observing a context dataset from a given task. Whereas EPIG is used to select informative observations that reduce predictive uncertainty for a certain task, we use TEPIG to score individual DGPs relative to a proxy model, and use these scores to determine how frequently tasks are sampled during TFM pretraining.

We then show that tilting a prior towards high TEPIG can allow for training a TFM that performs better on the original prior than a model trained on the original prior. We show this across three priors (sections 6.1, 6.2, 6.3). Specifically, the empirical result that tilted models perform better on high TEPIG datasets, without a loss of performance on low TEPIG datasets are evidence that TEPIG is a useful measure of dataset difficulty (Figure 5 (c), Figure 6 (b)).

As far as we are aware, this connection between predictive information and prior mass over datasets for TFM training is novel, as well as the empirical results.

In Sections 2.2 (L131) and 3 (L143) and Appendix A (L810), we discuss the relation of TEPIG to the related concepts of EPIG [2], epiplexity [3] and V-information [4].

Lacks theoretical guarantees. [...]

We believe that our explanations in Sections 1 and 4, together with our theoretical result in Theorem 1 (Appendix G, L1483), which relates TEPIG to the information gain with respect to a posterior over parameters, provide an interesting theoretical justification of TEPIG.

We furthermore find in a new experiment (Appendix H, L1613; summarised in Section 4, L244) that TEPIG, measured with cheap proxy models, is strongly correlated with the amortisation gap of trained models (Spearman correlations between 0.69 and 0.95 across nine model configurations and three proxies). This provides an explanation for why TEPIG-based tilting improves training speed. The amortisation gap of a task is high if (a) there exists a good solution to the task in the form of the posterior predictive distribution and (b) the model being trained still deviates from it. Tilting towards higher TEPIG therefore allocates more training to the tasks on which the model has the most left to learn.

A theoretical analysis that connects the learning dynamics of a TFM to TEPIG would be an interesting direction for future work.

Predictive-convergence assumption. [...]

We emphasise that the predictive convergence assumption is an approximation made for estimation TEPIG faster. We now provide detailed empirical results (Appendix E, L1080; referenced in Section 4.1, L289) on this assumption. In general we find that assuming predictive convergence decreases performance consistently albeit only relatively slightly when leaving all other TEPIG-estimation hyperparameters the same.

Importantly, however, for a fixed time budget using TEPIG_C (TEPIG assuming predictive convergence) rather than full TEPIG mostly leads to a better approximation compared to reducing the number of Monte-Carlo samples to estimate the expectation in TEPIG. This means that TEPIG_C Pareto-dominates TEPIG in terms of a speed-approximation trade-off, especially for a small number of Monte-Carlo samples.

AuthorsOpenReview ↗

Response to Reviewer g3eT (2/3)

The experiments focus on regression priors [...]

We now also provide results (Appendix Q.4, L4212) on the TabICL v2 classification prior, where we find that TEPIG-based tilting leads to consistent improvements compared to the Extra-Trees filter proposed for TabICL v2 [5].

[...] limited evidence for larger scale training [...]

For example the computational cost of training the default TabICL v2 model is around 50 GPU (A100) days, which is unfortunately completely infeasible for our academic budget. Instead, we went for a combination of a carefully controlled interpretable Gaussian-Process setup, a smaller-scale (one training run takes around 40 A100 hours) yet realistic prior setup proposed by [6] together with training on the real TabICL v2 prior, albeit for a small number of steps and a smaller model.

[...] and the computational costs of storing and rejecting tasks.

In terms of overall wall-clock training time, we generally train in the same amount of time with and without tilting. This is because the rejection sampling and data-generation can be done in parallel to training by running multiple CPU cores at once that generate data and compute the TEPIG scores.

In terms of overall FLOPs, model training costs roughly 2.3⋅10122.3·10^{12} FLOPs per training dataset on the SCM prior and 3.6⋅10113.6·10^{11} FLOPs per training dataset on the TabICL v2 prior. Training-data generation with the strongest KNN-based tilting (τ=20\tau = 20 and τ=2.5\tau = 2.5) costs roughly 6⋅1096·10^9 (SCM prior) and 1.2⋅10101.2·10^{10} (TabICL v2 prior) operations per training dataset on average, i.e. 0.25% and 3% of training compute. Without tilting, data generation accounts for 0.003% and 0.13% of training compute.

The time of computing TEPIG with the proposed cheap KNN tilting is of the same order as data generation itself. On one CPU core (AMD EPYC 7662 from 2020), scoring one dataset takes about 0.22 s on the SCM prior (512 context and 512 query points) and 0.75 s on the TabICL v2 prior (1024 context and 512 query points). Generating one dataset takes about 0.18 s with the SCM prior and 0.45 s with the TabICL v2 prior.

We use up to 50 GB of storage for the SCM prior and 120 GB of storage for the TabICL v2 prior, but did not specifically optimise for reducing the amount of storage (Appendix K, L2034).

We detail our calculations and results here (Appendix K, L1971).

AuthorsOpenReview ↗

Response to Reviewer g3eT (3/3)

Moreover, the experiments lack comparisons with relevant baselines such as EPIG or curriculum learning [1].

We would like to point out that in the realistic case of TFM priors, the exact posterior predictive is not accessible (it is in fact the object one tries to learn), and EPIG-based methods are therefore not applicable. Furthermore, EPIG is proposed for datapoint selection in active learning (choosing data points to reduce predictive uncertainty) and is therefore not obviously applicable for adapting the prior over datasets in TFM pretraining.

Curriculum learning methods [7] such as [1] decide during training which examples of a fixed, finite training set to use next. [1] scores each example by its current residual and by how fast this residual changes, which requires tracking every example over repeated visits by the model being trained. In TFM pretraining, the training data is an effectively unlimited stream of freshly sampled synthetic datasets, so there is no per-example training history to build on, and [1] is not directly applicable. We now also discuss this in the updated manuscript (Section 3, L177).

Furthermore, we have extended our evaluations to the following baselines:

  • Tilting based on training loss (for the same proxy models used for TEPIG-tilting).
  • The Extra Trees (ET) filter used for TabICL v2 [5].

We evaluate those alternative methods in the following setups:

  • On the new GP setup (Appendix M.4, L2754) mentioned above. Here, we find that loss-based tilting leads to a higher final loss and slower convergence than untilted training. The ET filter proposed for TabICL v2 does not lead to a significantly lower final loss. It reduces the number of training steps needed to reach the untilted loss by 13.7%, compared to an almost twice as large reduction for TEPIG-based tilting.
  • On the SCM prior (Appendix P.4, L3605), we also find that loss-based tilting substantially hurts performance and convergence speed. The ET filter from TabICL v2 improves in-prior performance early in training, then falls below untilted training, and ends with only a small improvement.
  • In a new prior with a step-function component (Appendix O, L3374), the ET filter from TabICL v2 rejects almost no datasets and therefore has no effect, while TEPIG-based tilting reduces the number of training steps needed by 41% compared to the untilted prior.
  • On the TabICL v2 regression prior (Appendix Q.6, L4364) we now explicitly compare TEPIG-based tilting and the ET filter and find that TEPIG-based tilting yields a consistently lower in-prior loss throughout training, with the largest gains early on. It also improves real-world performance (Appendix Q.8, L4474) on 26 out of 39 additional datasets from two established tabular benchmarks.
  • In new experiments on the TabICL v2 classification prior (Appendix Q.4, L4212) we also find TEPIG-based tilting to perform better than the ET filter.

Please let us know if you have any further questions or remaining concerns; we are happy to address them. If not, we would be grateful if you would consider raising your score.

References

[1] Tianyi Zhou et al. Curriculum Learning by Optimizing Learning Dynamics. AISTATS 2021

[2] Freddie Bickford Smith et al. Prediction-Oriented Bayesian Active Learning. AISTATS 2023

[3] Marc Finzi et al. From Entropy to Epiplexity: Rethinking Information for Computationally Bounded Intelligence. arXiv 2026

[4] Yilun Xu et al. A Theory of Usable Information under Computational Constraints. ICLR 2020

[5] Jingang Qu et al. TabICLv2: A Better, Faster, Scalable, and Open Tabular Foundation Model. ICML 2026

[6] Arik Reuter et al. Use What You Know: Causal Foundation Models with Partial Graphs. ICML 2026

[7] Yoshua Bengio et al. Curriculum Learning. ICML 2009

AuthorsOpenReview ↗

The Rebuttal Period Will Be Over Soon (Less Than 48 Hours Left)

Thank you again for your review!

We address your comments and questions in our answers above. In particular, our new study addresses the predictive-convergence assumption and we also find benefits of TEPIG-based tilting for classification.

If you have further questions, we are very happy to answer them.

If everything is addressed, we would greatly appreciate you increasing your score.

Reviewer_HeZ2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Instead of investing compute equally on synthetic tasks drawn randomly from a prior, the authors suggest investing more resources into tasks that predict a higher log loss reduction when conditioning/training them. The authors show higher convergence and better final results when prior training is tilted towards tasks that offer a higher expectancy in loss reduction.

Critical strengths
  • Training on the tilted prior seems to offer faster convergence and better final results.
Critical weaknesses
  • The paper can be written better in my opinion. Even after reading the submission I am not entirely sure on how exactly the authors implement TEPIG. The number of tasks that can be sampled on a prior is infinite, how do the authors determine what TEPIG scores are high and what are low? Additionally, it seems like the TEPIG score calculation for a given task is iterative, what is the overhead of running the said procedure in time?

  • Not sure why the authors decided to move away from the default TabICLv2 architecture and go for a smaller version. I would have preferred it if the authors used the original version, followed the same procedure in training and showed the TabArena performance over time from training on the original prior and from training on the titled prior. Maybe evaluating both checkpoints every xx iterations.

Questions for authors
  • It would be great to observe the performance of the original TabICLv2 architecture, with the original prior training and with the titled prior training in terms of time.

  • Could the authors describe based on my comments how exactly TEPIG is calculated and what is the overhead that it ads?

  • In Table 6, did the authors run on all TabArena datasets? Could the authors provide the median and ranks of each method, the mean is not really meaningful because a method might win in a couple of datasets with a higher margin and lose overall in the majority. I count 6 wins, 1 tie from 11 datasets for the untilted prior.

  • It looks as if the gain from TEPIG drops as the number of training steps increases, the authors offer results for up to 20k steps, while the original TabICLv2 work was trained for 550k steps. How would the tilted prior perform with more steps?

  • How does one determine τ\tau, is it task dependent?

9 comments
AuthorsOpenReview ↗

Response to Reviewer HeZ2 (1/2)

Thank you for your review!

We are happy that you write that "Training on the tilted prior seems to offer faster convergence and better final results".

We address your questions below and have also updated the manuscript with the content in the linked documents. Section, figure, table and line numbers (L) refer to the updated manuscript.

Weaknesses

[...] I am not entirely sure on how exactly the authors implement TEPIG.

We now provide a new detailed description of the concrete algorithm used to compute TEPIG with any proxy model. We have added it as Appendix D of the updated manuscript (L1001) and reference it from Section 4.1 (L292).

The number of tasks that can be sampled on a prior is infinite, how do the authors determine what TEPIG scores are high and what are low?

For tilting the prior based on TEPIG, we compute the ranks on the training-datasets we sample. In practice, we keep a buffer of 5,000 tasks for the SCM prior and 10,000 tasks for the TabICL v2 prior which we use for ranking. Because the tilting is rank-based, only the relative TEPIG scores of different datasets matter. We now describe this in the updated version of the manuscript (Section 6, L344).

Additionally, it seems like the TEPIG score calculation for a given task is iterative, what is the overhead of running the said procedure in time?

In terms of overall wall-clock training time, we generally train in the same amount of time with and without tilting. This is because the rejection sampling and data-generation can be done in parallel to training by running multiple CPU cores at once that generate data and compute the TEPIG scores.

In terms of overall FLOPs, model training costs roughly 2.3⋅10122.3·10^{12} FLOPs per training dataset on the SCM prior and 3.6⋅10113.6·10^{11} FLOPs per training dataset on the TabICL v2 prior. Training-data generation with the strongest KNN-based tilting (τ=20\tau = 20 and τ=2.5\tau = 2.5) costs roughly 6⋅1096·10^9 (SCM prior) and 1.2⋅10101.2·10^{10} (TabICL v2 prior) operations per training dataset on average, i.e. 0.25% and 3% of training compute. Without tilting, data generation accounts for 0.003% and 0.13% of training compute.

The time of computing TEPIG with the proposed cheap KNN tilting is of the same order as data generation itself. On one CPU core (AMD EPYC 7662 from 2020), scoring one dataset takes about 0.22 s on the SCM prior (512 context and 512 query points) and 0.75 s on the TabICL v2 prior (1024 context and 512 query points). Generating one dataset takes about 0.18 s with the SCM prior and 0.45 s with the TabICL v2 prior.

We detail our calculations and results here (Appendix K, L1971).

Not sure why the authors decided to move away from the default TabICLv2 architecture and go for a smaller version.

The computational cost of training the default TabICL v2 model [1] is around 50 GPU (A100) days, which is unfortunately completely infeasible for our academic budget. Instead, we went for a combination of a carefully controlled interpretable Gaussian-Process setup, a smaller-scale (one training run takes around 40 A100 hours) yet realistic prior setup proposed by [2] together with training on the real TabICL v2 prior, albeit for a small number of steps and a smaller model.

We would also like to point out that we ran training with five more seeds for KNN tilting and two more seeds for TabICL v2 tilting on the SCM prior (Appendix P.5, L3726) and two more runs for our TabICL v2 regression setup (Appendix Q.5, L4320), confirming our initial findings.

Furthermore, we now also ran a new experiment (Appendix Q.4, L4212) on the TabICL v2 classification prior providing further evidence that TEPIG-based tilting helps in the context of TabICL v2.

AuthorsOpenReview ↗

Response to Reviewer HeZ2 (2/2)

Questions

In Table 6, did the authors run on all TabArena datasets?

TabArena [3] contains 13 regression datasets, of which we use 11: one has fewer than the 1,024 data points we train on, and one has more than the 100 features supported by our pretraining setup (Appendix Q.3, L4124). Based on your comment, we now also report ranks and win rates (Appendix Q.7, L4419) on these 11 datasets: the model trained with TEPIG-based tilting improves its mean rank to 4.45 from 5.55 without tilting. Additionally, we evaluate on 39 new real-world datasets (Appendix Q.8, L4474) from two further benchmarks, where TEPIG-based tilting improves over the standard ET filter on 26 datasets (16 significantly).

Could the authors describe based on my comments how exactly TEPIG is calculated [...]?

TEPIG can be computed from the learning curve of a (tabular) machine learning model (e.g. KNN or TabICL v2). The learning curve results from fitting the model with more and more training-datapoints while evaluating its test loss.

For example, in our SCM experiment with KNN, we compute it as follows:

  1. We draw a dataset with 512 training rows and 512 test rows from the prior.
  2. We fit the proxy on the first kk training rows and record its loss ℓk\ell_k on unseen rows, for 26 values of kk between 1 and 512.
  3. TEPIG is the area between this learning curve and its final value ℓ512\ell_{512}.

A detailed description including pseudocode can be found here (Algorithms 1 and 2 in Appendix D, L1041 and L1065).

[...] and what is the overhead that it ads?

We kindly refer to our answer to the third weakness you mentioned.

It looks as if the gain from TEPIG drops as the number of training steps increases [...]

We believe this to be the case as the TEPIG-tilted models start learning faster than the untilted models. This can be seen more clearly in Figure 42 (L3860) for the SCM prior, and Figure 47 (a) (L4207) for the TabICL prior.

How does one determine τ\tau, is it task dependent?

In general, we see tau as a hyperparameter that controls the shift towards more (or less) learnability across a prior. τ\tau is not task-dependent and fixed once per prior. A contribution of our work is to make this learnability dimension of the prior (a) visible (as shown in Figure 3 in our manuscript) and (b) controllable through tilting. What exactly the ideal τ\tau is can depend on the prior, and the model to be trained, but throughout our experiments, we find that maximal tilting strength in terms of the Kullback-Leibler divergence between untilted and tilted TEPIG distribution that keeps the acceptance rate of tasks under the rejection sampling scheme above 10 percent to work the best (Table 4 in Appendix J.1, L1946).

Please let us know if you have any further questions or remaining concerns; we are happy to address them. If not, we would be grateful if you would consider raising your score.

References

[1] Jingang Qu et al. TabICLv2: A Better, Faster, Scalable, and Open Tabular Foundation Model. ICML 2026

[2] Arik Reuter et al. Use What You Know: Causal Foundation Models with Partial Graphs. ICML 2026

[3] Nick Erickson et al. TabArena: A Living Benchmark for Machine Learning on Tabular Data. NeurIPS 2025

AuthorsOpenReview ↗

The Rebuttal Period Will Be Over Soon (Less Than 48 Hours Left)

Thank you again for your review!

We address your comments and questions in our answers above.

In particular, our new description addresses how TEPIG is implemented, and our new cost analysis addresses its overhead.

If you have further questions, we are very happy to answer them.

If everything is addressed, we would greatly appreciate you increasing your score.

Reviewer_E561OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper propose TEPIG (Task Expected Predictive Information Gain) to estimates the predictive benefit of allocating compute to different tasks, to improve the training of TFM.

Critical strengths
  1. The motivation is very clear: the paper motivates with an example that some task has high variance and some task has high complexity, how to allocate training resources on them is the problem,

  2. The paper evaluate the method on not only synthetic dataset but also real-world dataset.

  3. While the definition of TEPIG is clear, Formula (4), the model q is involved. q is changing during model training, so how to estimate TEPIG is hard. The paper explore proxy models for estimating TEPIG, which is great.

Critical weaknesses
  1. Fig1 mentioned PPD which is not defined.

  2. The contribution seems to be weak presented. The paper does not claim how much performance gain the method contribute comparing baseline on real-world benchmarks.

Questions for authors

How does the author claim the technique novelty here? Is TEPIG itself new or TEPIG is new in the field of TFM?

3 comments
AuthorsOpenReview ↗

Response to Reviewer E561 (1/2)

Thank you for your review!

We are happy to hear that you assess the motivation of our paper as very clear.

We address your questions below and have also updated the manuscript with the content in the linked documents. Section, figure, table and line numbers (L) refer to the updated manuscript.

Weaknesses

  1. Fig1 mentioned PPD which is not defined.

Thank you for mentioning this. We now clarify that PPD is the posterior predictive distribution in the updated version of the manuscript (caption of Figure 1, L70).

q is changing during model training, so how to estimate TEPIG is hard.

We would like to clarify that qq is not changing during model training. qq is a proxy model chosen to compute TEPIG. The proxy model can be any machine learning model that provides predictions (such as KNNs) or other trained TFMs. We have added an improved description of how we implemented the estimation of TEPIG to our manuscript (Appendix D, L1001; referenced in Section 4.1, L292).

The paper does not claim how much performance gain the method contribute comparing baseline on real-world benchmarks.

We now have introduced two new baselines.

  • Tilting based on training loss (for the same proxy models used for TEPIG-tilting).
  • The Extra Trees (ET) filter used for TabICL v2 [1].

We evaluate those alternative methods in the following setups:

  • On the new GP setup (Appendix M.4, L2754). Here, we find that loss-based tilting leads to a higher final loss and slower convergence than untilted training. The ET filter proposed for TabICL v2 does not lead to a significantly lower final loss. It reduces the number of training steps needed to reach the untilted loss by 13.7%, compared to an almost twice as large reduction for TEPIG-based tilting.
  • On the SCM prior (Appendix P.4, L3605), we also find that loss-based tilting substantially hurts performance and convergence speed. The ET filter from TabICL v2 improves in-prior performance early in training, then falls below untilted training, and ends with only a small improvement.
  • In a new prior with a step-function component (Appendix O, L3374), the ET filter from TabICL v2 rejects almost no datasets and therefore has no effect, while TEPIG-based tilting reduces the number of training steps needed by 41% compared to the untilted prior.
  • On the TabICL v2 regression prior (Appendix Q.6, L4364) we now explicitly compare TEPIG-based tilting and the ET filter and find that TEPIG-based tilting yields a consistently lower in-prior loss throughout training, with the largest gains early on. It also improves real-world performance (Appendix Q.8, L4474) on 26 out of 39 additional datasets from two established tabular benchmarks.
  • In new experiments on the TabICL v2 classification prior (Appendix Q.4, L4212) we also find TEPIG-based tilting to perform better than the ET filter.

We would also like to point out that we already report real-world results for both priors: gains on 12 OpenML datasets for the SCM prior (Figure 5 (b), Table 14 in Appendix P, L3571), and results on 11 TabArena [2] datasets for the TabICL v2 prior (Table 15, L4149, and Figure 46 (b), L4116, in Appendix Q).

Furthermore, we have extended our evaluation by also comparing the performance on 39 additional real-world datasets (Appendix Q.8, L4474) from two further benchmarks (TALENT [3]; Grinsztajn et al. [4]), where we find that TEPIG-based tilting alone improves over the standard ET filter in 26 out of 39 cases. We find 16 of these improvements to be significant at 95%-percentile intervals.

AuthorsOpenReview ↗

Response to Reviewer E561 (2/2)

Questions

How does the author claim the technique novelty here?

In this paper, we tackle the problem of choosing where to place prior mass on complicated mixtures of synthetic data-generating processes (DGPs) from which datasets are sampled for TFM training. Specifically, the question we answer is: How should we change the prior mass over this mixture of DGPs such that TFM training is efficient (lower loss with fewer datasets seen)? Intuitively, our starting insight is that TFM training will most benefit if we show it fewer of "simple" predictive tasks so that it can spend more time learning "harder" predictive tasks.

Our technical contribution is to formalise this intuition building on ideas from information theory and active learning (specifically EPIG [5] and V-information [6] / Epiplexity [7]). Consequently, we introduce TEPIG - the expected reduction in a predictor qq’s joint predictive log loss after observing a context dataset from a given task. Whereas EPIG is used to select informative observations that reduce predictive uncertainty for certain task, we use TEPIG to score individual DGPs relative to a proxy model, and use these scores to determine how frequently tasks are sampled during TFM pretraining.

We then show that tilting a prior towards high TEPIG can allow for training a TFM that performs better on the original prior than a model trained on the original prior. We show this across three priors (sections 6.1, 6.2, 6.3). Specifically, the empirical result that tilted models perform better on high TEPIG datasets, without a loss of performance on low TEPIG datasets are evidence that TEPIG is a useful measure of dataset difficulty (Figure 5 (c), Figure 6 (b)).

As far as we are aware, this connection between predictive information and prior mass over datasets for TFM training is novel, as well as the empirical results.

Please let us know if you have any further questions or remaining concerns; we are happy to address them. If not, we would be grateful if you would consider raising your score.

References

[1] Jingang Qu et al. TabICLv2: A Better, Faster, Scalable, and Open Tabular Foundation Model. ICML 2026

[2] Nick Erickson et al. TabArena: A Living Benchmark for Machine Learning on Tabular Data. NeurIPS 2025

[3] Si-Yang Liu et al. TALENT: A Tabular Analytics and Learning Toolbox. JMLR 2025

[4] Léo Grinsztajn et al. Why Do Tree-Based Models Still Outperform Deep Learning on Typical Tabular Data? NeurIPS 2022

[5] Freddie Bickford Smith et al. Prediction-Oriented Bayesian Active Learning. AISTATS 2023

[6] Yilun Xu et al. A Theory of Usable Information under Computational Constraints. ICLR 2020

[7] Marc Finzi et al. From Entropy to Epiplexity: Rethinking Information for Computationally Bounded Intelligence. arXiv 2026

AuthorsOpenReview ↗

The Rebuttal Period Will Be Over Soon (Less Than 48 Hours Left)

Thank you again for your review!

We address your comments and questions in our answers above.

In particular, our new real-world evaluation addresses the performance gain on real-world benchmarks and we discuss the novelty of TEPIG.

If you have further questions, we are very happy to answer them.

If everything is addressed, we would greatly appreciate you increasing your score.

Reviewer_UPNvOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explains a task expected predictive information gain(TEPIG) which is for a predictor q and one synthetic task, and the expected reduction in q's joint the log loss on nth query point after seeing n context points. The tilting experiments and the prior analysis use an approximation, defined as TEPIG_C, the area of q's learning curve above its value at n (Equation 6). TEPIG is used to describe how TabICL v2 prior choices shift task scores shown in Section 5 and to tilt task sampling during Tabular Foundation Model pretraining by rejection sampling in Appendix G. On a Gaussian process(GP) prior with an exact posterior predictive, tilting reaches the untitled model's final KL in 31 to 43% fewer steps; on the structural causal model prior in the paper cited Reuter et al. 2026, median negative log-likelihood gains of 0.12 to 0.21 remain at 50k steps; on a reduced TabICL v2 prior, gains are small. It is reported that tilting toward low TEPIG hurts.

Critical strengths
  • A relatively simple and cheap per task score with a pattern across the three priors established
  • A transparent reporting of both positive and negative results
  • tabular models are very relevant in certain industries or domain with a large amount of data in that form
Critical weaknesses
  • The experiments don't seem to test whether "TEPIG" really adds anything over a simpler task scores. Tiling seems to just move mass to the shortest length scales as shown in Figure 12b and 14b? The reversed tilt seems to show that direction matters, but it can't separate the TEPIG from other difficulty scores with the numbers as listed. Two related ideas also go undiscussed in the paper for the predictive information of [1], and the TabICL v2 prior's existing filter for datasets on which the ExtraTrees cannot beat a constant baseline [2]. I think this matters because this measure is the paper's main contribution so should seem to need to be addressed.

  • The evidence on realistic priors doesn't seem to be as supportive in this paper as the original abstract says in that on complex prior's "it allows for training in fewer steps". The learning curves show faster early training, but the gains shrink? Like on the TabICL v2 for the prior, the final gain is 0.6·10E-4 median pinball loss on a loss at about 0.07, and with no significant difference on TabArena for Section 6.3. The SCM and TabICL v2 comparisons seem to use one training run for each, and the intervals "do not quantify training-seed variability"?

[1] W. Bialek, I. Nemenman, and N. Tishby, "Predictability, complexity, and learning," Neural Comput., vol. 13, no. 11, pp. 2409–2463, 2001.

[2] J. Qu, D. Holzmüller, G. Varoquaux, and M. Le Morvan, "TabICLv2: A better, faster, scalable, and open tabular foundation model," in Proc. 43rd Int. Conf. Mach. Learn. (ICML), 2026.

Questions for authors
  • How many training runs does each of the SCM and TabICL v2 arm use for this project? Related to this what is there seed to seed sopread here like are there more statistical measures; error bars to report for the untilted baseline's final metric relative to the reported gains?
  • I would be interested in the wall clock costs compared to other methods, including the task generation, scoring and the rejection?
5 comments
AuthorsOpenReview ↗

Response to Reviewer UPNv (1/2)

Thank you for your comments and questions! We are happy to hear that you describe the proposed TEPIG as ''A relatively simple and cheap per task score with a pattern across the three priors established''.

We address your questions below and have also updated the manuscript with the content in the linked documents. Section, figure, table and line numbers (L) refer to the updated manuscript.

Weaknesses

Tiling seems to just move mass to the shortest length scales as shown in Figure 12b and 14b?

We have now set up an improved version of our Gaussian Process (GP) experiments (Appendix M, L2268; referenced in Section 6.1, L409) where we have defined an extended range of the prior over the lengthscale parameter. Our new prior also varies the kernel and noise variance of the GP.

In this experiment, we observe a rather complex interaction (Appendix M.2, L2339, and Figure 28, L2841) of TEPIG-tilting with the prior. For example, TEPIG-tilting allocates less prior mass to both very small and large lengthscale values since both pose tasks with little learnable predictive structure. (Please also compare to Figure 2 and Appendix C for why both very small and large lengthscales pose tasks with little learnable structure.)

The experiments don't seem to test whether "TEPIG" really adds anything over a simpler task scores.

We now have introduced two new baselines.

  • Tilting based on training loss (for the same proxy models used for TEPIG-tilting).
  • The Extra Trees (ET) filter used for TabICL v2 [2].

We evaluate those alternative methods in the following setups:

  • On the new GP setup (Appendix M.4, L2754) mentioned above. Here, we find that loss-based tilting leads to a higher final loss and slower convergence than untilted training. The ET filter proposed for TabICL v2 does not lead to a significantly lower final loss. It reduces the number of training steps needed to reach the untilted loss by 13.7%, compared to an almost twice as large reduction for TEPIG-based tilting.
  • On the SCM prior (Appendix P.4, L3605), we also find that loss-based tilting substantially hurts performance and convergence speed. The ET filter from TabICL v2 improves in-prior performance early in training, then falls below untilted training, and ends with only a small improvement.
  • In a new prior with a step-function component (Appendix O, L3374), the ET filter from TabICL v2 rejects almost no datasets and therefore has no effect, while TEPIG-based tilting reduces the number of training steps needed by 41% compared to the untilted prior.
  • On the TabICL v2 regression prior (Appendix Q.6, L4364) we now explicitly compare TEPIG-based tilting and the ET filter and find that TEPIG-based tilting yields a consistently lower in-prior loss throughout training, with the largest gains early on. It also improves real-world performance (Appendix Q.8, L4474) on 26 out of 39 additional datasets from two established tabular benchmarks.
  • In new experiments on the TabICL v2 classification prior (Appendix Q.4, L4212) we also find TEPIG-based tilting to perform better than the ET filter.

Two related ideas also go undiscussed in the paper for the predictive information of [1], and the TabICL v2 prior's existing filter for datasets on which the ExtraTrees cannot beat a constant baseline [2].

Thank you for pointing out this interesting paper. We added a discussion of it to the manuscript (Section 3, L158). In general, EPIG [3] can be seen as a special case of the predictive information defined in [1]. Note that TEPIG differs from EPIG in two key ways: 1) the outer expectation is conditioned on θ\theta, 2) the distribution with which the expectation is taken (the data generating process) is different to the distribution inside the expectation (proxy model). Rather than a quantity about the data generating process (like EPIG), TEPIG measures the reduction in log loss achieved by a proxy model.

We also added the following explanation (Section 3, L151, and Appendix N, L3348) of the filter from TabICL v2 to the related work section in our paper.

AuthorsOpenReview ↗

Response to Reviewer UPNv (2/2)

The learning curves show faster early training, but the gains shrink?

We believe this to be the case as the TEPIG-tilted models start learning faster than the untilted models. This can be seen more clearly in Figure 42 (L3860) for the SCM prior, and Figure 47 (a) (L4207) for the TabICL prior.

In general, the results conditioned on the TEPIG decile show that we gain on high TEPIG tasks without losing much on lower TEPIG tasks. Hence, the final performance on a prior is subject to what the distribution over TEPIG for that prior is. For example, if a prior has high mass over low TEPIG, we don't expect the mean performance to improve much. We see here that the TabICL v2 prior puts more mass on low-TEPIG datasets than the SCM prior, which explains why the gains are larger on the SCM prior.

[...] with no significant difference on TabArena for Section 6.3.

We believe there are two reasons for this.

  1. As tilting towards higher TEPIG allows for learning a prior better (by improving on the high TEPIG tasks), the performance on real world datasets is subject how well the prior describes the real world dataset. We do not make claims on real world improvements apart from those that are a consequence from learning a fixed prior better. It is entirely possible to learn a given prior better but the prior itself does not describe real world datasets well.
  2. We show (Figure 6 (b), Figure 5 (c)) that the tilted model improves on high TEPIG tasks without losing out too much on low TEPIG tasks. Figure 46 (b) (L4116) shows a similar slight upward trend in performance increase with TEPIG value on real world datasets.

Nonetheless, we add 39 new real-world datasets (Appendix Q.8, L4474) where we win in 26 cases by tilting towards high-TEPIG, compared to the standard ET filtering.

Questions

How many training runs does each of the SCM and TabICL v2 arm use for this project?

Due to the high computational cost (around 128 GPU (A100) hours for our TabICL v2 experiment and 40 GPU hours for each SCM run), we have only conducted one run per tilting method so far for those priors, while running larger numbers of seeds for the GP-experiments.

However, in response to your question we ran 5 additional KNN tilting and 2 additional TabICL v2 tilting runs on the SCM prior and 2 more runs on the TabICL v2 prior. Our new results on the SCM prior (Appendix P.5, L3726) as well as our new results for TabICL v2 (Appendix Q.5, L4320) show the significance of the improvement across seeds.

I would be interested in the wall clock costs compared to other methods, including the task generation, scoring and the rejection?

In terms of overall wall-clock training time, we generally train in the same amount of time with and without tilting. This is because the rejection sampling and data-generation can be done in parallel to training by running multiple CPU cores at once that generate data and compute the TEPIG scores.

In terms of overall FLOPs, model training costs roughly 2.3⋅10122.3·10^{12} FLOPs per training dataset on the SCM prior and 3.6⋅10113.6·10^{11} FLOPs per training dataset on the TabICL v2 prior. Training-data generation with the strongest KNN-based tilting (τ=20\tau = 20 and τ=2.5\tau = 2.5) costs roughly 6⋅1096·10^9 (SCM prior) and 1.2⋅10101.2·10^{10} (TabICL v2 prior) operations per training dataset on average, i.e. 0.25% and 3% of training compute. Without tilting, data generation accounts for 0.003% and 0.13% of training compute.

The time of computing TEPIG with the proposed cheap KNN tilting is of the same order as data generation itself. On one CPU core (AMD EPYC 7662 from 2020), scoring one dataset takes about 0.22 s on the SCM prior (512 context and 512 query points) and 0.75 s on the TabICL v2 prior (1024 context and 512 query points). Generating one dataset takes about 0.18 s with the SCM prior and 0.45 s with the TabICL v2 prior.

We detail our calculations and results here (Appendix K, L1971).

Please let us know if you have any further questions or remaining concerns; we are happy to address them. If not, we would be grateful if you would consider raising your score.

References

[1] William Bialek et al. Predictability, Complexity, and Learning. Neural Computation 2001

[2] Jingang Qu et al. TabICLv2: A Better, Faster, Scalable, and Open Tabular Foundation Model. ICML 2026

[3] Freddie Bickford Smith et al. Prediction-Oriented Bayesian Active Learning. AISTATS 2023

AuthorsOpenReview ↗

The Rebuttal Period Will Be Over Soon (Less Than 48 Hours Left)

Thank you again for your review!

We address your comments and questions in our answers above.

In particular, our new baselines address the comparison with simpler task scores, and our additional seeds address the number of training runs.

If you have further questions, we are very happy to answer them.

If everything is addressed, we would greatly appreciate you increasing your score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Our Responses

We thank all reviewers for their helpful comments and questions! Since less than 48 hours remain in the rebuttal period, we summarise below the main changes we have made in response to the reviews.

We have incorporated all changes into the updated manuscript (marked in blue).

1. Comparison with Baselines (Reviewers UPNv, E561, g3eT, e4Uj)

We now compare TEPIG-based tilting with tilting based on the loss of the same proxy model, and with the Extra-Trees (ET) filter used for TabICL v2.

  • Loss-based tilting leads to slower convergence and a higher final loss than untilted training, both on a new, broader GP prior (Appendix M.4, L2754) and on the SCM prior (Appendix P.4, L3605).
  • The ET filter reduces the number of training steps needed on the GP prior by 13.7%, compared to an almost twice as large reduction for TEPIG-based tilting, and ends with only a small improvement on the SCM prior.
  • On the TabICL v2 regression prior (Appendix Q.6, L4364), TEPIG-based tilting yields a consistently lower in-prior loss than the ET filter throughout training.

2. Compute Cost of Tilting (Reviewers UPNv, HeZ2, g3eT, e4Uj)

Our new cost analysis (Appendix K, L1971) shows that generating the training data with our strongest KNN-based tilting costs 0.25% (SCM prior) and 3% (TabICL v2 prior) of the training compute. Wall-clock training time stays the same, because datasets are generated and scored on CPU cores in parallel to training.

3. More Training Seeds (Reviewers UPNv, HeZ2, e4Uj)

We ran five more seeds for KNN-based tilting and two more for TabICL v2-based tilting on the SCM prior (Appendix P.5, L3726), and two more seeds on the TabICL v2 prior (Appendix Q.5, L4320). These runs support our initial findings.

4. Presentation and Related Work (Reviewers UPNv, E561, HeZ2, g3eT)

We added a step-by-step description with pseudocode of how TEPIG is computed (Appendix D, L1001). In Section 3, we now discuss predictive information (L158), the ET filter of TabICL v2 (L151) and curriculum learning (L177).

5. More Real-World Datasets (Reviewers UPNv, E561, HeZ2, e4Uj)

We evaluate on 39 additional real-world datasets (Appendix Q.8, L4474) from two further benchmarks. TEPIG-based tilting improves over the standard ET filter of TabICL v2 on 26 of them (16 significantly). On TabArena, we now also report ranks and win rates (Appendix Q.7, L4419), where TEPIG-based tilting improves the mean rank from 5.55 to 4.45.

6. Classification (Reviewers HeZ2, g3eT)

In new experiments on the TabICL v2 classification prior (Appendix Q.4, L4212), TEPIG-based tilting improves over both untilted training and the ET filter.

7. Further Analysis of TEPIG (Reviewer g3eT)

In a new experiment (Appendix H, L1613), TEPIG measured with cheap proxy models is strongly correlated with the amortisation gap of trained models (Spearman correlations between 0.69 and 0.95), which provides an explanation for why tilting towards higher TEPIG speeds up training. We also add an empirical study (Appendix E, L1080) of the predictive-convergence assumption.

We are very happy to answer any further questions!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.