acceptodds
Under review as a conference paper at ICLR 2027

Learning Heterogeneous Tabular Tasks through Multiregime Pretraining of Prior-Data Fitted Networks

Abstract

Tabular datasets can combine observations from different populations, environments, or operating conditions, each governed by a different relationship between features and labels. These coexisting relationships define distinct regimes. Regime membership, however, may be unobserved, requiring classifiers to account for heterogeneity within a single dataset. Prior-data fitted networks (PFNs) are pretrained on synthetic datasets to predict new observations from labelled examples without task-specific retraining. We investigate whether incorporating regime heterogeneity into this pretraining enables compact PFNs to learn multiregime classification. We extend TabICL's structural-causal-model prior with regime-dependent score functions and label mappings, and compare fixed-mixture and curriculum pretraining with canonical NanoTabPFN pretrained on the native single-regime prior, published tabular foundation models, and conventional classifiers. On held-out synthetic tasks without a supplied regime ID, the largest effect occurs in feature-dependent, multiregime multiclass tasks: curriculum pretraining improves a six-layer NanoTabPFN's excess cross-entropy from to nats, a 10.8% relative reduction. Across 2,880 matched soft-gate episodes, the paired cross-entropy difference is nats (95% episode-bootstrap interval ) and the accuracy-gain difference is points (). Across all matched multiregime tasks with three to five classes, the curriculum model exceeds majority-class accuracy by 6.69 points, compared with 6.30 for the native single-regime model, a 6.2% relative increase; its excess cross-entropy improves from to nats, a similarly sized 10.5% relative reduction. Fixed-mixture pretraining produces a similar pooled improvement, so these experiments do not isolate an advantage of the curriculum schedule. Evaluation on 36 real-world BeyondArena tasks and a merged multi-site heart-disease case study finds no consistent advantage over the native single-regime control.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Prior-data fitted networks, tabular foundation models, multiregime classification, synthetic pretraining, prior design, in-context learning, heterogeneous tabular data

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_h8BJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper considers tabular data settings in which the relation between the target variable and the source variables are conditional on latent "regimes" - which they term as multiregime data. They extend the data prior of tabular foundation models with this type of data to see if it helps downstream generalization.

Critical strengths
  1. Practicality: The considered multiregime settings appear practical downstream; for instance when learning tasks are to be done on datasets that are pooled from multiple different sources; e.g. in the paper: prediction on patient data from three different hospitals.

  2. New and different ways to create multiregime priors: They propose 2 distinct modifications to the SCM based TFM prior. First creates a different target node/variable per regime, and uses regime-specific cut-off points to translate the scalar target variable to class labels. Second shares the same SCM structure but employs different cutoff points per regime. They also play with the assignment of regimes; either encoded in input/observed data or latent membership not directly derivable from input.

Critical weaknesses
  1. Minor improvements: Biggest concern for this work is that the idea does not seem to pan out; improvements are quite marginals and even at places they are significant, the effect size is practically irrelevant. Various versions of TabICL readily perform well on multiregime settings simulated in-distribution. (Table 1 and Fig 2)

  2. AI generated content: While it alright for the authors to benefit from AI in generating, say, drawings/figures for the paper, the text also appears to be written by AI, including the interpretation of the results. This leaves the reviewer with lack of confidence in terms of how the findings are communicated.

Questions for authors
  • Did you try inferring if real world datasets follow multi-regime? If so, how would one go about doing it? If they did in fact follow multi-regime, however it is defined, that would be a clearer motivation for the work.

  • If target y follows different functions of the input, as in multiregime, it is hard to imagine standard pretrained TFMs to directly be able to * infer * those multiple functions in-context (via in context learning). Thus, how could the authors explain TabICL variants to do readily so well in such datasets? Perhaps the created multi-function input is not challenging enough or constructed to probe where they would fall short.

It seems that the paper needs 1) more real-world motivation, and 2) better investigation of the shortcomings of in-context learning (ICL) to multi-regime input.

Reviewer_r53rOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

To the best of my understanding, I think that the paper suggests drawing the labels of rows in each pretraining dataset from a mixture of different labeling mechanisms instead of a single one. The features come from one structural causal model (SCM), and each row has a latent zz that selects its labeling mechanism. The mechanism is either a deterministic function of the first feature or drawn independently of the features, and the PFN is not given access to it. It then evaluates this slightly adapted pretraining prior scheme to test whether it improves in-context classification. The authors pretrain nanoTabPFN at three sizes with these tasks mixed in, at a fixed 30% rate or on a curriculum rising to 50%. They compare against a matched control trained on the unmodified TabICL prior, on held-out synthetic tasks from the same prior, BeyondArena tasks, and a heart-disease dataset. They report a small cross-entropy gain on synthetic multiclass tasks, mostly where the mechanism depends on the first feature, and no gain on real data.

Critical strengths

None

Critical weaknesses
  • The paper is very hard to follow because it names its components with terms that either clash with established meanings or are internal pipeline labels. I had to sort of reverse engineer Figure 1 to find out what the proposed pretraining prior modification actually is. Examples are the use of "score function", e.g., in the abtract, "soft gate", "excess cross-entropy" and "routing".
  • Section 3 never defines the method formally. It gives only a "schematic form" y=hz(rz(x))y = h_z(r_z(x)) and never states p(z∣x)p(z \mid x), how KK is sampled, or how cut points and class permutations are drawn. Those details appear only as script paths and code names in the Appendix.
  • The paper never shows that the problem it targets even exists. On the evidence of published PFN checkpoints, capacity and training scale matter more than the proposed prior change. The paper also shows no real dataset where a "native-prior" PFN (as the authors call it) fails because of label heterogeneity.
  • Also, the novelty is very limited, as adding families to the pretraining prior is a standard and already exhaustingly visited part of the related literature which, however, is not discussed in this submission.
  • The test tasks for evaluation come from the adapted pretraining prior and are therefore in distribution. The PFN spends a substantial part of the pretraining batches on datasets from exactly this adapted prior that also produces the headline evaluation subset, whereas the baseline PFN does not see these datasets during pretraining. A PFN approximates the posterior predictive under its training prior, so of course it does better on tasks from a prior if it has seen datasets from that prior extensively during pretraining. On real data, the gains are non-existent.
  • Experiments only use one repetition (one seed), the baselines are untuned, and there is no oracle reference to compare the proposed prior change against.
  • Paper layout and formatting is poor with Figures 3-6 and 10 being almost impossible to read. Also, filepaths and slurm commands take up a lot of space in the appendix and add nothing to the paper.

In my opinion, the submission currently falls substantially short of the standards of the venue.

Questions for authors
  1. What model is "TabPFN v2.2", for instance in Table 1?
Reviewer_XcHbOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper extends TabICL’s SCM-based synthetic data generator to construct a multi-regime prior for studying prediction when different data-generating rules coexist within a single dataset.

Specifically, it considers an rzr_z mechanism in which both scoring functions and label mappings vary across regimes, and a gzg_z mechanism in which the score is shared but label mappings vary, while comparing regime assignments based on observed features with assignments based on unobserved latent groups.

The authors pretrain NanoTabPFN on these synthetic tasks without providing regime IDs as prediction inputs, and compare a fixed proportion of multi-regime tasks with a curriculum that gradually increases this proportion to investigate how the prior and training schedule affect predictive performance.

Critical strengths

The paper addresses a clearly defined problem, namely in-context prediction in settings where different data-generating rules coexist, and also, the proposed method is clearly described.

Critical weaknesses
  1. (weak novelty) The proposed method appears to extend TabICL’s SCM-based prior over synthetic tasks by introducing row-level heterogeneity while using the existing NanoTabPFN as the predictive model. However, describing the method as incorporating multiple data-generating rules within a single task does not, by itself, sufficiently establish its novelty.

  2. (very bad readability) Despite the disclosure in the AI Use Statement that AI was used to create the figures, regarding the presentation, the quality of Figure 1 should be improved.

Questions for authors
  1. Could the authors clarify how the proposed design for heterogeneity differs from existing approaches based on synthetic task priors and why these differences constitute a meaningful and novel contribution?

  2. Could the authors improve the quality of Figure 1 to enhance its readability?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.