acceptodds
Under review as a conference paper at ICLR 2027

STRUCTURE-ALIGNED SYNTHETIC PRIORS FOR TABULAR FOUNDATION MODELS

Abstract

Tabular foundation models such as TabPFN, TabICL, and MITRA are pre-trained on millions of synthetic datasets sampled from randomly generated Structural Causal Models (SCMs). In the dominant paradigm the SCM is realised as a randomly wired multi-layer perceptron (MLP) whose activations are read off as features and targets, so the dependency graph of the training data is an artifact of MLP connectivity rather than a deliberate choice. We argue that this topol- ogy is an independent design variable, and that it diverges systematically from real causal structure. Comparing∼36,000 TabICL SCMs against 33 real-world causal graphs of three independent provenances, we find that MLP priors produce conditional dependence structures an order of magnitude wider: median Markov- blanket width 43.8 versus 4.0, maximum parent count 31 versus 3, maximum minimum-d-separator cardinality 20 versus 3. This is not a restatement of graph size—MLP blanket width scales with node count (∝N0.73) whereas real-world width is size-invariant (N0.06, n.s.), and the gap persists at 10.2×within a matched size band. We therefore introduce BN-ICL, a prior whose dependency graph is an explicit Bayesian network matched to real structural statistics, and train in-context learners from scratch under an ablation in which the only difference is DAG topol- ogy. On an augmented suite of 219 real classification datasets (TALENT plus a high-dimensional OpenML expansion specified in advance), the BN prior im- proves mean accuracy and log-loss, but not uniformly: the effect is concentrated in, and grows with, feature dimensionality, and the priors are indistinguishable on the median dataset. Above 100 features the BN prior wins by +1.5–1.6 percent- age points (bootstrap 95% CI excluding zero, Holm-corrected p<0.05), replicat- ing across two independently trained prior configurations and on external datasets disjoint from TALENT—ruling out a benchmark-composition artifact—and sur- viving truncation of those tables to the training feature width when the retained features are informative, which rules out width extrapolation as the explanation. Low- and mid-dimensional gains are smaller or fragile. DAG topology is thus a meaningful, under-explored axis of prior design, sharpest where the structural discrepancy is largest.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Bayesian Networks, Causality, Transformer, Foundational Models, Priors

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 2 reviews, above 0% of the others.
Reviewer_mKZ5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper demonstrates a systematic gap in descriptive statistics of graphs from real world data generating processes from those sampled from SCM priors used in training tabular foundation models. To improve performance of TFMs on real world graphs, a new synthetic data generator is proposed, replacing SCM generation via BN from pruning MLPs to a mixture of other classes of graphs (e.g. Erdos-Renyi or scale-free graphs). Proposal is evaluated empirically on held out real world datasets.

Critical strengths

The motivation and idea are sound. I appreciate the need for a more systematic study (in public) of synthetic data generation for the purpose of training TFMs.

Critical weaknesses

The paper feels currently quite underdeveloped. Specifically:

  1. Insufficient theoretical depth
  • Graph G (or scm S) induces observed distribution P which is the actual input into a TFM. This should be elaborated upon and investigated (e.g. different complexity S can induce similar if not same observed distribution)
  • The theoretical insight as to why matching these graph properties to real world data should improved performance is not developed in detail (it does intuitively make sense)
  • representativeness of the reference graphs is not well established
  1. Empirical investigation is shallow
  • Mixture of five graph families is given without ablation or analysis of individual component contribution. In other words the potential of the paper's own method is not systematically investigated
  • The performance improvement seems barely statistically significant.
  • Anonymized code repo supplement would be highly appreciated (while not required by conference, I see it as an established good scientific standard) for reproducibility and checkability of work.
  1. Presentation is excessively long in details, particularly Sections 7-8. Stronger prioritization of key findings and overall maintaining prose with a narrative thread throughout the paper would make it more readable. More thorough treatment of related work would also be appropriate.

In summary, the work in its current form does not do justice to an interesting research direction that it proposes. Better theoretical treatment and better accompanying empirical evaluation are necessary before I can see this is a valuable contribution. Considering the idea is interesting I am open to increasing my score but conditional on addressing the current significant insufficiencies of the manuscript.

Questions for authors

Can the authors:

  • establish deeper theory, including but not limited to treatment of the observed distribution P?
  • provide better experiments including ablations, possibly more seeds (than the current one) and show a more clearly superior result following their method?
  • elaborate on the robustness of their results?
  • provide an anonymized code repository?
Reviewer_CZbmOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

N/A

Critical strengths

N/A

Critical weaknesses

N/A

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.