Learning Heterogeneous Tabular Tasks through Multiregime Pretraining of Prior-Data Fitted Networks

Tabular datasets can combine observations from different populations, environments, or operating conditions, each governed by a different relationship between features and labels. These coexisting relationships define distinct regimes. Regime membership, however, may be unobserved, requiring classifiers to account for heterogeneity within a single dataset. Prior-data fitted networks (PFNs) are pretrained on synthetic datasets to predict new observations from labelled examples without task-specific retraining. We investigate whether incorporating regime heterogeneity into this pretraining enables compact PFNs to learn multiregime classification. We extend TabICL's structural-causal-model prior with regime-dependent score functions and label mappings, and compare fixed-mixture and curriculum pretraining with canonical NanoTabPFN pretrained on the native single-regime prior, published tabular foundation models, and conventional classifiers. On held-out synthetic tasks without a supplied regime ID, the largest effect occurs in feature-dependent, multiregime multiclass tasks: curriculum pretraining improves a six-layer NanoTabPFN's excess cross-entropy from $-0.1562$ to $-0.1730$ nats, a 10.8% relative reduction. Across 2,880 matched soft-gate episodes, the paired cross-entropy difference is $-0.0168$ nats (95% episode-bootstrap interval $[-0.0179,-0.0158]$) and the accuracy-gain difference is $+0.84$ points ($[+0.76,+0.91]$). Across all matched multiregime tasks with three to five classes, the curriculum model exceeds majority-class accuracy by 6.69 points, compared with 6.30 for the native single-regime model, a 6.2% relative increase; its excess cross-entropy improves from $-0.0838$ to $-0.0926$ nats, a similarly sized 10.5% relative reduction. Fixed-mixture pretraining produces a similar pooled improvement, so these experiments do not isolate an advantage of the curriculum schedule. Evaluation on 36 real-world BeyondArena tasks and a merged multi-site heart-disease case study finds no consistent advantage over the native single-regime control.