acceptodds
Under review as a conference paper at ICLR 2027

Information-Set Confounding in Consistency Tests for Tabular Foundation Models

Abstract

Recent work tests compatibility of univariate tabular foundation model (TFM) predictions by comparing marginals with autoregressive conditionals. The tested identities, however, combine posterior predictives based on different observed training columns. Because the law of total probability and the chain rule apply under a common information set, a nonzero cross-information gap need not imply a failure of Bayesian coherence across observed datasets. After making the conditioning explicit, we bound both reported gaps by ordinary posterior updates. We also construct an exact coherent Bayesian model for which both gaps equal , while a predictor that ignores cross-target information attains zero gap at strictly worse joint risk. This reversal appears in a 60-task exact Bayesian benchmark, controlled experiments with TabICLv2 and TabPFNv2, and all ten classification datasets in the original study. On each dataset and for both models, a fixed mixture of the native conditional chains improves joint negative log likelihood over the zero-gap independent product; the combined pooled relative NLL gains are 10.62% and 10.17%. The queried heads can be incompatible when their different contexts are suppressed; our result shows why this does not establish that the predictor lacks a coherent Bayesian interpretation. A valid consistency test must hold the observed information fixed; practical evaluations should also include an information-blind control and held-out predictive scores.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
tabular foundation models, probabilistic consistency, information sets, Bayesian prediction, evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vnsnOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper rebuts a recent paper [1] on the consistency behaviour of univariate tabular foundation models (TFMs) when deployed on multivariate classification tasks by using autoregressive prediction. It argues that [1]’s approach of dropping a certain column from the context set leads to an incorrect analysis of the consistency properties, because dropping the column from the context set amounts to a change in the evidence, not a marginalization. The paper then performs an analysis similar to that of [1], but restores the information sets to correctly analyse the consistency behaviour of TFMs.

[1] Klötergens, Christian, et al. "Do Tabular Foundation Models Agree with Themselves?." arXiv preprint arXiv:2608.06004 (2026).

Critical strengths
  • The paper correctly fixes the assumption from [1] and provides grounded theoretical reasoning in Section 2. It fixes major problems in [1] and gives a more grounded treatment of the analysis.
  • The main contribution of the paper is supported by theoretical justification and adequate experiments.
Critical weaknesses
  • The paper could be made stronger by explaining why this behaviour is observed, especially in connection with the architecture design of these tabular foundation models.
  • In addition, the study seems limited to two different TFMs, and the TabPFN used is an older version rather than the latest one. Given the paper goal, Introducing more TFM models would strengthen the paper.

Minor but important: The paper is a bit dense, and the writing could still be improved.

Questions for authors
  1. Would it make sense to extend the analysis to regression tasks? If so, what are the challenges and limitations? (Future work)

  2. Related to the cons above, are the properties discussed in the paper tied to the choice of the TFMs’ architecture? It would be nice if the paper discussed further which architectural components of the TFMs might be responsible for the observed behaviours. (Clarification)

Minor: It might be nice to also connect the findings to the regression case and discuss related work, especially in the Neural Processes literature, such as [2,3].

[2] Bruinsma, Wessel P., et al. "Autoregressive conditional neural processes." arXiv preprint arXiv:2303.14468 (2023).

[3] Hassan, Conor, et al. "Efficient autoregressive inference for transformer probabilistic models." International Conference on Learning Representations. Vol. 2026.

Reviewer_fHXaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper points out that existing consistency tests for TFMs might be misleading because they compare predictions made with different training information. The authors show this theoretically and empirically using TabICLv2 and TabPFNv2.

Critical strengths

The paper makes a simple but relevant conceptual point for evaluation of TFMs in the context of probabilistic consistency testing. This is well-supported with an appropriate amount of theory and some experiments. The Bayesian counterexample is specifically interesting.

Critical weaknesses

*The empirical results are not as convincing as the theoretical contribution. Specifically, the conducted experiments show that the existing consistency metric can be misleading, but they do not show whether the tested TFMs themselves are actually consistent when the information set is held fixed. Also, the evaluation is limited to TabICLv2 and TabPFNv2, while other TFMs available at the time of submission (TabPFNv2.5 and v3, LimiX and TabFM) partially use different architectures or modeling objectives. Evaluating on a more diverse set of models would help to assess how broadly the observed issue actually applies. Specifically LimiX could be interesting here, since it also supports joint/masked modeling and imputation.

Questions for authors

Can the proposed tests also be applied to (current) SOTA TFMs? Specifically for models like LimiX-2, EXAONE Tabular, Kumo Tabular, and TabPFN-3.5 (which partially appeared very recently so this is not a criticism that they were not included in the first place) it would be interesting to understand whether their interfaces make such a test easier in future work.

Reviewer_t194OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper examines consistency tests for tabular foundation models that compare marginal and autoregressive predictions. It argues that these tests combine predictions conditioned on different observed training columns, so a nonzero gap does not necessarily imply Bayesian incoherence. The authors derive an information-update bound, construct exact Bayesian counterexamples, and show that a zero-gap independent predictor can have worse predictive risk. Experiments include an exact finite-prior benchmark, controlled synthetic tasks, and two tabular foundation models across ten classification datasets.

Critical strengths
  1. A precise and consequential correction to consistency-test interpretation. The paper demonstrates that changing the observed training columns changes the conditioning information, making the usual probability identities inapplicable without qualification. The exact counterexamples establish that the tests can reject coherent Bayesian predictors.
  2. Supporting evidence across theoretical and practical settings. The exact benchmark, controlled experiments, and real-data comparisons show why a zero consistency gap should not be treated as a standalone measure of predictive quality. The fixed mixture avoids choosing the favorable autoregressive order using test outcomes.
Critical weaknesses
  1. The contribution is narrow, and the practical coherence question remains unresolved. The central observation is that probability identities require a common conditioning information set, and the bound follows from standard total variation arguments. Applying this observation to an existing evaluation protocol is useful, but the paper does not quantify its contribution to the gaps measured in actual TFMs or implement a corrected diagnostic for those models. It therefore challenges one interpretation of the existing test while leaving the coherence of current TFM predictions unresolved.

  2. The empirical comparisons do not establish the proposed explanation. The independent product discards target dependence, whereas the native mixture uses conditional predictions and averages two chains. Better predictive performance therefore does not isolate information set confounding. The source study already treats predictive quality and consistency as distinct evaluation axes, so demonstrating that a zero gap construction predicts worse provides limited additional insight. Differences in categorical handling and MIC preprocessing further complicate direct comparison.

Questions for authors

see comments above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.