acceptodds
Under review as a conference paper at ICLR 2027

Slow Prediction Concentration in Tabular Foundation Models

Abstract

How do tabular foundation model predictions change with more labelled rows at inference? We measure prediction spread across contexts and the effect of deleting a single row for six released checkpoints from four model families. On disjoint higgs contexts of – rows, four checkpoints have fitted probability spread slopes between and , compared with for logistic regression. At rows, their accuracy is – percentage points higher than logistic regression's. An exact-tree XGBoost reference refit on the same supports also concentrates faster. These slopes describe the observed range. They give no asymptotic guarantee. Deleting a support row can also alter the fingerprint features of retained rows. For TabPFN v2.5, we compare fingerprints on and off across two datasets, six support seeds, four sizes (–), and 50 fixed, unfiltered queries per dataset. Disabling fingerprints makes mean pairwise influence decay faster on both probability and log-odds scales. All four paired contrasts survive Holm correction. The log-odds exponent rises by on higgs and on magic. Faster influence decay does not by itself imply better prediction or a population-wide effect. Whether spread eventually concentrates faster at larger context sizes remains open.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Tabular foundation models, in-context learning, prior-data fitted networks, TabPFN, prediction stability, concentration rates, influence analysis

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ijS9OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the context dependence of tabular foundation models (TFMs). It examines two aspects: prediction spread, measured as the standard deviation of predictions across different contexts, and the influence of a single context row on the prediction. The authors evaluate several TFMs, including Mitra, PFNs, TabICL, and TabFM.

Critical strengths

None

Critical weaknesses

The paper present something in a useful direction, but the contribution feels somewhat limited in its current form, since it largely consists of empirical checks of the results in Nagler (2023). It would help if the authors clarified what new insights the work offers beyond this prior study. As it stands, I am not yet convinced the paper meets the bar for ICLR.

My main concern is the clarity of the writing. The paper is currently quite difficult to follow, which makes it hard to properly assess the central claims. For example (one of several such instances), “higgs” and “magic” are introduced without explanation, so the reader can’t tell whether they are datasets or something else. The paper also presents many specific numbers with little context, including in the abstract. Some of these details may fit better in the experiments section or the appendix, which would let the main narrative and claims come through more clearly. I believe the presentation needs a substantial revision before the paper is ready for submission, and I hope the authors find these comments helpful in strengthening it.

Reference:

Nagler, Thomas. "Statistical foundations of prior-data fitted networks." International Conference on Machine Learning. PMLR, 2023.

Questions for authors

None

Reviewer_8XAjOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

In this paper, the authors empirically analyze how increasing the number of rows affects the final prediction in tabular foundation model frameworks. Specifically, they mainly consider two metrics. The prediction spread metric measures the standard deviation of a query's prediction across different draws of the contexts. The single-row influence metric measures the change in prediction when a single row is removed. Empirical results on two datasets show that the rate is slower than n−1/2n^{-1/2}.

Critical strengths
  1. The empirical finding may provide important insights for further research.
Critical weaknesses
  1. The contribution of the paper is limited. The empirical finding is potentially interesting, but the authors do not provide a more detailed analysis of the underlying reason. I would expect the authors to provide more insights in the theoretical part.
  2. The experiments are conducted on only two specific datasets, although there are a large number of tabular datasets. The authors do not provide any explanation for why these two datasets were selected. Moreover, as shown in Table 1, the slope for the HIGGS dataset is much smaller than -1/2, while the slope for the MAGIC dataset is much closer to -1/2. As a result, it is possible that some datasets perfectly fit the n−1/2n^{-1/2} rate. As a result, the experiments on two datasets may not be sufficient to support the claim that the rate is slower than n−1/2n^{-1/2}.
  3. There are also many unclear parts in the paper. For example, in line 78, the authors do not provide details about the support draws. What does this mean? How are they drawn? What distribution are they drawn from? Also, the experiments use only 50 queries for each configuration, which I think is insufficient to support the claims.
Questions for authors

See the weakness part.

Reviewer_P1ZeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(−1)
Read review
Paper summary

The authors study how tabular foundation models depend on their context, analysing prediction spread across different contexts and the effect of dropping an individual row across multiple leading models. Specifically, they find that more data points reduce the prediction spread much slower than classical models like XGBoost and Logistic Regression, even though the foundation models achieve better predictive performance. While investigating deleting a single row, they find that the fingerprint feature in TabPFN’s preprocessing distorts and increases single row deletion effects, since the deletion changes the features of all remaining rows. Furthermore, the authors find that the influence of the most influential row shrinks slowly as data grows, while the average row influence can shrink much more quickly.

Critical strengths
  • The theoretical foundation and experiments are explored and conducted rigorously, with quantifications of exactly what a result shows and how it can be interpreted. Furthermore, the authors also mention exactly which interpretations are not supported by the results.
  • The insight that tabular foundation model predictions stabilise more slowly than classical models, while being more accurate, is interesting. It could be relevant in areas like uncertainty estimation and might be important information for practitioners switching between classical models and foundation models.
Critical weaknesses
  • While the paper is very detail-oriented, I did not see a clear hypothesis when reading the paper, and it appears more as a collection of individual results. A clear motivation on why these experiments are important for the community, how they are connected and outlining the impact of these findings on foundation models would greatly improve the clarity. Furthermore, the results get somewhat lost within all the details, making the paper overall challenging to read.
  • While the paper describes the slow stabilisation in detail, it does not explain why this happens. It would be interesting to investigate via additional experiments what causes this behaviour and whether it changes based on dataset properties such as noise level within the observed datapoints. Without this, the significance of the results remains limited.
Questions for authors
  • Do you have insight into what characterises the most influential rows? Were these outliers or rows near the decision boundary? Also, did they improve or drag down the performance when removed?
  • Given the findings, what should practitioners take away from them and do differently when using these models or building new ones?
  • Did you find any relationship between the prediction concentration and the maximum pretraining context size of the model, or are these independent of each other?
  • Given your investigation of fingerprint feature preprocessing inside TabPFN, should such preprocessing steps where a single row impacts the full context be avoided in model design?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.