acceptodds
Under review as a conference paper at ICLR 2027

Can Tabular Foundation Models Amortize Statistical Inference?

10 views

Abstract

For decades, statistical inference has largely been developed one problem at a time. Given a scientific target, such as a treatment effect or a regression function, statisticians design a problem-specific estimator together with a procedure for quantifying its uncertainty. This paper proposes a different paradigm. We focus on a classical problem in statistical inference, confidence interval construction, and develop TabCon, an amortized inference system built on a tabular foundation model that produces confidence intervals for new datasets through a simple forward pass. The key methodological ingredients of TabCon are a sparse mixture-of-experts architecture and reinforcement-learning-based post-training that calibrate the resulting confidence intervals to a desired coverage level. Across a wide range of benchmark datasets, TabCon attains near-nominal coverage while producing short confidence intervals. At inference time, it also offers considerably greater computational efficiency, running 50 times faster than the classical bootstrap procedure, even when the latter uses only 50 bootstrap samples.

Primary area:
foundation or frontier models, including LLMs
Keywords:
tabular foundation models, confidence intervals, uncertainty quantification, statistical inference, Mixture-of-Experts

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_56VoOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TabCon and explores whether TFMs can construct confidence intervals for the conditional mean in regression tasks in an amortized manner. Building on point estimates from a frozen TabPFN, TabCon trains estimation error prediction heads using synthetic data and constructs confidence intervals from their quantiles. Experiments on synthetic and semi-synthetic data demonstrate good coverage and short interval lengths.

Critical strengths

The paper makes an interesting attempt to explore how well the existing pretraining paradigm of TFMs can achieve calibration in a statistical sense, and demonstrates good performance under the paper’s experimental settings.

Critical weaknesses

For a rigorous statistical inference problem, the current experiments have limited persuasiveness, particularly regarding the statistical clarity of the experimental settings and the boundaries of the model’s capabilities.

Questions for authors
  1. What is the joint distribution PXP_X of the covariates used in training and evaluation, and how is it generated?

  2. Holding the other data-generating mechanisms fixed, how does performance change when PXP_X changes, e.g., when some variables shift from symmetric to one-sided distributions?

  3. Under what conditions does TabCon fail (undercoverage) substantially or completely?

  4. In an i.i.d. setting with (X⊤,Y)⊤∼MVN(μ,Σ)(X^\top,Y)^\top\sim MVN(\mu,\Sigma) (Σ\Sigma positive definite, n>d+1n>d+1), how does TabCon compare with the exact OLS t-interval? Does it offer an IL advantage at comparable coverage?

  5. Could the authors include stronger CI baselines with theoretical coverage guarantees under explicit assumptions, and evaluate them where these assumptions hold?

(If these concerns are adequately addressed, I would be willing to raise my score.)

9 comments

We thank the reviewer for the comments and for the specific questions, which helped us to improve the statistical rigor of our experimental evaluation and better characterize the capabilities and limitations of TabCon. Since your questions detail the weakness, we focus on answering them point by point below. In particular, we describe the generation of the covariates (C1), study the sensitivity of TabCon to the covariate distribution (C2), characterize the conditions under which TabCon undercovers (C3), and compare TabCon with exact or theoretically guaranteed CIs in settings where their assumptions hold (C4 and C5).

Response C1: Covariate distribution and generation (Q1)

We appreciate this question. The joint distribution of the covariates PXP_X is defined at the task level, and each task is generated with its own covariate distribution. Our main generating mechanism, similarly to the prior of TabPFN [1,2], is a structural causal model (SCM) defined on a random directed acyclic graph (DAG), to which we add parametric covariate families and real covariates, as follows.

  • (i) Training data for the Global head and testing data for ID evaluation: in the SCM, we generate a sequence of random variables V1,…,VpV_1,\dots,V_p, whose dependencies are given by the random DAG, and select the covariates among them. (ia) The first variables are drawn independently from Gaussian, uniform, Gaussian-mixture, Student-tt or categorical distributions, possibly with a common latent factor that makes some of them dependent. (ib) Each subsequent variable is a random function of a few previously generated variables, Vk=gk(Vpa(k);θk)V_k=g_k\big(V_{\mathrm{pa}(k)};\theta_k\big), where Vpa(k)V_{\mathrm{pa}(k)} denotes the selected previous variables (its parents in the DAG), and gkg_k is drawn from a family of linear and nonlinear mappings, such as nonlinear transformations (e.g., tanh, sine and step functions), random Fourier features, Gaussian-process draws and tree-based mappings, with randomly sampled parameters θk\theta_k. (ic) The covariates are then selected as a random subset of V1,…,VpV_1,\dots,V_p, and their joint distribution therefore combines different types of marginal distributions with linear and nonlinear dependencies. (id) To reproduce common issues of real data, some covariates may finally be replaced by independent variables unrelated to the response or by copies of other covariates, discretized into categories, or masked with missing values.
  • (ii) Training data for the specialists and the router: most tasks are generated as in (i). To increase the heterogeneity of the covariate distributions, the remaining tasks draw the covariates from parametric families with different dependence structures, including equicorrelated, block-correlated, low-rank and anisotropic Gaussian distributions, as well as heavy-tailed, skewed and multimodal distributions. As in (i), some covariates may then be converted into categorical variables or masked with missing values.
  • (iii) OOD evaluation: the covariates follow a Gaussian factor model with random loadings, X=ZL+EX=ZL+E, to which we add nonlinear dependencies among the covariates, random rescaling and shift of each covariate, resulting in correlated and generally non-Gaussian distributions.
  • (iv) CC18 and CTR23 benchmarks: we use the real covariates of each dataset.

In all synthetic settings, the response is then generated as Yi=fi+εiY_i=f_i+\varepsilon_i, where the regression target fif_i is computed by the generator and is therefore known at every query point, and εi\varepsilon_i follows the noise mechanisms described in Appendix C. We will add a detailed description, including all parameter ranges, to our manuscript.

References

[1] N. Hollmann, S. Müller, K. Eggensperger and F. Hutter (2023). TabPFN: A transformer that solves small tabular classification problems in a second. International Conference on Learning Representations (ICLR).

[2] N. Hollmann, S. Müller, L. Purucker, A. Krishnakumar, M. Körfer, S. B. Hoo, R. T. Schirrmeister and F. Hutter (2025). Accurate predictions on small data with a tabular foundation model. Nature, 637, 319–326.

Response C2: Performance when PXP_X changes (Q2)

This is an excellent suggestion. We therefore designed a new experiment in which only the covariate distribution changes, while the regression function, the noise distribution and all other generating parameters are kept fixed. Specifically, we consider three families of regression functions, corresponding to RFF, MLP and Gaussian-process draws, with independent Gaussian noise. To move from symmetric to one-sided covariates, we start from Gaussian covariates and change the marginal distribution of 0%, 50% or 100% of them to a half-normal distribution with the same mean and variance.

Table 1. Coverage (%) / mean interval length of TabCon at the 95% nominal level. From left to right, the proportion of covariates with a half-normal marginal distribution increases.

Function family0% half-normal50% half-normal100% half-normal
RFF97.35 / 3.795197.19 / 3.697697.24 / 3.6276
MLP98.21 / 3.177398.23 / 3.108598.01 / 3.0532
GP98.32 / 3.425098.80 / 3.434099.08 / 3.4942
Overall97.96 / 3.465898.07 / 3.413398.11 / 3.3917

Table 1 shows that the coverage of TabCon remains stable, and above the nominal level, as the covariates move from symmetric to one-sided distributions. These results provide additional evidence that TabCon is robust to this type of change in the covariate distribution.

Response C3: TabCon limitations and failures (Q3)

This is another excellent point which helped us identify the conditions under which TabCon may fail. We address it from two perspectives.

First, on real covariates, undercoverage is limited to a few datasets, mostly with discrete, mixed-type or strongly dependent covariates or a high noise level (see our response B2 to Reviewer AZfp).

Second, we simulated two covariate shifts of increasing severity between the context data and the query points. In the first one, the context data follow a Gaussian linear model and the query points are placed at increasing distance from them (Table 2). As the distance grows, TabCon produces wider intervals and its coverage decreases, whereas the exact OLS tt-interval remains close to the nominal level.

Table 2. Coverage (%) of TabCon and of the exact OLS tt-interval at the 95% nominal level in a Gaussian linear model with n=64n=64, d=16d=16 and X∼N(0,Id)X\sim\mathcal N(0,I_d). From top to bottom, the query points are placed at increasing distance from the context data.

Norm of the query pointTabConOLS tt-interval
d\sqrt d99.0694.43
1.5d1.5\sqrt d97.5594.38
2d2\sqrt d95.3694.27
3d3\sqrt d91.5694.38

In the second setting, the query points lie outside the support of the context data. We restrict the context to x1≤0x_1 \le 0 and set fA(x)=0.4+x⊤β+A,1x1>0f_A(x)=0.4+x^\top\beta+A\\,\mathbf 1\\{x_1 > 0\\}, which introduces a jump of size AA in the region x1>0x_1 > 0. Since the jump does not affect the context data, ff is not identified outside their support, and no method can guarantee coverage there without structural assumptions on ff. Accordingly, TabCon covers 45% of the out-of-support query points for A=2A=2 and none for A=4A=4 and A=8A=8, while its coverage within the support remains above the nominal level.

In summary, TabCon may substantially undercover when the query points lie in low-density regions or outside the support of the context data.

Response C4: TabCon vs OLS intervals (Q4)

Interesting suggestion. We consider the Gaussian linear model Yi=0.4+Xi⊤β+εiY_i=0.4+X_i^\top\beta+\varepsilon_i, with Xi∼N(0,Id)X_i\sim\mathcal N(0,I_d), εi∼N(0,1)\varepsilon_i\sim\mathcal N(0,1) independent of XiX_i, for n∈32,64,128,256n\in\\{32,64,128,256\\} and d/n∈0.25,0.5,0.75d/n\in\\{0.25,0.5,0.75\\}. We further consider two settings for the regression coefficients: a dense setting, in which all coefficients are nonzero, and a sparse setting, in which only two coefficients are nonzero. We compare TabCon with the exact OLS interval for the conditional mean, based on the Student-tt distribution with n−d−1n-d-1 degrees of freedom.

At the 95% nominal level, TabCon attains higher coverage than OLS (98.88% versus 94.77%) with shorter intervals (3.32 versus 4.20). Since TabCon is conservative in this setting, we also follow your suggestion and compare the two methods at the same empirical coverage level. For each method, we select the nominal level that yields 95% coverage directly on the test data to provide a descriptive comparison at matched in-sample coverage. Results for the two settings are reported in Tables 3 and 4, respectively. We note that, in both settings, the OLS interval remains exact. At comparable coverage, TabCon yields intervals about 46% shorter than OLS on average.

Table 3. Coverage (%) and mean interval length (IL) of TabCon and of the exact OLS tt-interval in the dense Gaussian linear model at similar coverage, with nominal levels directly selected on test data, averaged over n∈32,64,128,256n\in\\{32,64,128,256\\}. The last column reports the reduction in IL of TabCon relative to OLS.

d/nd/nTabCon CPOLS CPTabCon ILOLS ILIL reduction
0.2595.0795.082.6702.185-22.2%
0.5095.1095.073.1313.73516.2%
0.7595.0895.123.8746.73242.5%
Overall95.0895.093.2254.21723.5%

Table 4. Coverage (%) and mean IL of TabCon and of the exact OLS tt-interval in the sparse Gaussian linear model at similar coverage. The rest is the same to Table 3.

d/nd/nTabCon CPOLS CPTabCon ILOLS ILIL reduction
0.2595.0895.071.2462.16742.5%
0.5095.0895.101.2803.82666.5%
0.7595.1295.071.3926.82979.6%
Overall95.1095.081.3064.27469.4%

This result is perhaps surprising, given that the OLS confidence interval is the classical benchmark under the Gaussian linear model. Yet, TabCon, which is pretrained on a much broader class of data generating processes, achieves substantially shorter intervals while maintaining comparable coverage. A deeper look at the two tables suggests that this improvement may stem from two sources:

  1. The most substantial gains are observed in the sparse setting. This is intuitive, as standard OLS does not explicitly exploit the sparsity structure and may therefore be inefficient in this setting. The results suggest that TabCon can adapt to such structure, potentially incorporating variable selection when constructing confidence intervals.
  2. The improvement becomes more pronounced as the ratio d/nd/n increases. This is consistent with the James–Stein phenomenon [3], a classical result in multivariate estimation, which demonstrates that introducing shrinkage can improve estimation accuracy. In a similar spirit, TabCon can leverage regularities learned through pretraining across a broad collection of tasks, hence borrowing structural information across datasets rather than relying only on the information available in the current dataset. This cross-task information allows TabCon to achieve more efficient uncertainty quantification in high-dimensional settings.

In summary, even in the classical Gaussian linear model, TabCon performs well and can outperform the classical OLS benchmark, when the regression coefficients are sparse or the covariates are high-dimensional.

References

[3] James, W. and Stein, C. (1961). Estimation with Quadratic Loss. Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Vol. 1, 361–379.

Reviewer_AZfpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TABCON, a prior-data fitted network that amortizes confidence-interval construction for a conditional mean function. The method reframes this task as learning the distribution of the residual f^(x)−f(x)\hat f(x)-f(x). It combines a frozen TabPFN-v3 point prediction with three residual heads: a global head trained across all data-generating processes, an expansion specialist targeting undercoverage, and a shrinkage specialist targeting overcoverage. A router trained through supervised fine-tuning and reinforcement learning adaptively combines their outputs. Across synthetic and semi-synthetic benchmarks, TABCON achieves near-nominal average coverage and relatively short intervals while offering lower inference latency than the evaluated sampling- and resampling-based alternatives.

Critical strengths
  • The main idea of amortized statistical inference via quantifying the distristatistical of the prediction error is neat and potentially impactful.
  • The empirical results are promising: TABCON generally provides a favorable balance between coverage, interval length, and inference-time efficiency.
Critical weaknesses
  • The implementation appears to be driven primarily by ad hoc engineering choices rather than a clearly justified methodological principle. For instance, it is unclear why three heads and top-2 routing are necessary designs, and the current ablation does not isolate the effects of the specialists, sparse routing, or RL post-training.

  • The evaluation is limited to synthetic or semi-synthetic tasks and omits TabMGP, a highly relevant peer-reviewed baseline. Aggregate coverage also hides undercoverage on individual datasets, while the theoretical result applies only to oracle rather than learned quantiles.

  • The claimed 50-fold speedup over bootstrap is not particularly informative because bootstrap inherently requires repeated model evaluations; a stronger efficiency analysis should compare against optimized recent martingale-posterior methods under matched coverage and include TABCON's training cost and amortization break-even point.

Questions for authors
  1. Why are three residual heads and top-2 routing needed? How does TABCON compare with a single residual head trained using the same coverage-aware objective under matched parameter and training budgets?
  2. How were the 50,000/50,000/100,000 task allocations, specialist thresholds, and RL reward coefficients selected? How sensitive are the results to these choices?
  3. Why is TabMGP not included as a baseline?
  4. Can the authors report variability across training seeds and task-level or worst-case coverage rather than only pooled coverage?
9 comments

We sincerely thank the reviewer for the insightful comments, which gave us the opportunity to clarify the motivation behind our methodology and to obtain new results that enrich our experiments and our understanding of the behavior of TabCon in several relevant settings. Below, we address the critical weaknesses W1-W3 and the questions Q1-Q4. In particular, we mainly have (i) assessed the role of each component of our method (e.g., the mixture-of-experts architecture), (ii) extended the comparison to TabMGP, and (iii) further motivated the hyperparameters involved in the analysis.

Response B1: Design principles and ablations (W1)

Our model architecture is not merely ad hoc engineering choices. Each component is motivated by a specific challenge in learning reliable confidence intervals across DGPs. We explain the rationale behind our three main design choices: (i) specialist heads, (ii) sparse routing, and (iii) RL post-training.

  • (i) Specialist heads. Our mixture of experits architecture procedures are motivated by the challenge of learning the full distribution of the estimation error f^(x)−f(x)\hat f(x)-f(x), whose shape can vary substantially across DGPs. As a consequence, a single head trained across all DGPs may achieve near-nominal coverage on average while under- or overcovering in specific regimes. The Expansion and Shrinkage specialists are trained on these two regimes to bring coverage closer to its nominal level.
  • (ii) Sparse routing. Since a given task typically exhibits either undercoverage or overcoverage, but not both simultaneously, it is therefore not reasonable to combine all three heads. We thus adopt Top-2 routing, allowing the router to select and combine only the two most relevant heads for each task; for instance, Global can be paired with the Expansion specialist on tasks where it undercovers.
  • (iii) RL post-training. We employ RL to directly optimize the coverage and interval length. Although the NLL and CRPS objectives used in SFT are effective for learning the distribution of estimation errors, they do not directly optimize the final CI performance criteria of interest. Meanwhile, empirical coverage involves non-differentiable indicator functions, making direct gradient-based optimization challenging. The use of RL is thus necessary to optimize a reward that explicitly accounts for coverage and interval length.

Following your suggestion, we have also substantially expanded the ablation study, in our response A5 to Reviewer dHYU, and, in part, in our response B4 to you.

Response B2: Experimental scope and theoretical statement (W2)

We address each of these points below.

  1. We mainly focus on synthetic and semi-synthetic tasks because they allow us to directly assess CI coverage for the conditional mean f(x)=E[Y∣X=x]f(x)=\mathbb{E}[Y\mid X=x], which is generally unknown in real-world observational datasets, while the semi-synthetic benchmarks also retain realistic covariate structures. To bring the evaluation closer to real data, we have further evaluated TabCon on all 35 OpenML-CTR23 datasets, in which both the covariates and the regression function are derived from real data (see our response A4 to Reviewer dHYU), providing additional evidence on the effectiveness of the procedure in more realistic settings.
  2. We agree that TabMGP is a relevant comparison and have added it alongside AMP in Table 1. TabMGP defines its target through a loss function. In its regression experiments, it uses the squared-error loss for a linear model, so that its target is the best linear approximation of ff. Under nonlinear DGPs, these two targets generally differ, which explains the substantial undercoverage of TabMGP when evaluated against f(x)f(x), as reported in Table 1. For this reason, we previously compared against AMP, which targets the conditional mean f(x)f(x) directly, as the more direct comparison. We further discuss the computational cost of TabMGP in our response to W3.
  3. We agree that average coverage can hide under- and overcoverage on individual datasets. Appendix D.1 already reports per-dataset coverage and interval length on CC18. At the 95% nominal level, TabCon attains at least nominal coverage on 9 of the 13 datasets and, among the methods reaching this threshold, yields the shortest intervals in all 9 cases. The few undercovered datasets mostly have mixed-type or strongly dependent covariates (e.g., credit-approval and mfeat-fourier), which we will further discuss as a limitation in the revision. Moreover, we will further add the per-dataset results on the CTR23 datasets, as already specified in our response A4 to Reviewer dHYU.
  4. Finally, Proposition 1 is intended to formalize the inferential principle underlying the proposed construction, by establishing how the quantiles of the estimation error f^(x)−f(x)\hat f(x)-f(x) translate into a valid CI for f(x)f(x). In practice, these oracle quantiles are replaced by learned estimates, whose coverage we assess empirically. Extending the coverage analysis to account for quantile estimation error is certainly possible by incorporating finite sample error bounds based on standard quantile estimation arguments and existing theoretical analyses of PFNs. However, such an extension would be technically involved, as it requires to control the approximation and generalization errors arising from learning the estimation error distribution across tasks. More importantly, the resulting bounds would primarily quantify the discrepancy between the learned and oracle quantiles, without providing further insight into the underlying inferential principle. We therefore focus our theoretical analysis on establishing this fundamental principle.

Table 1. Coverage probability (CP, %) and interval length (IL) on 32 nonlinear out-of-distribution DGPs, with one context realization per DGP and 512 query points in total. From left to right, results are reported at the 90%, 95% and 99% nominal coverage levels, and each entry reports CP / IL. TabMGP uses 100 trajectories of 500 steps. TabMGP (†\dagger) targets the best linear approximation of ff, whereas coverage is evaluated against f(x)f(x).

MethodNominal 90%Nominal 95%Nominal 99%
AMP93.16 / 1.60195.90 / 2.26198.44 / 7.798
TabMGP$^\dagger$35.94 / 1.17342.19 / 1.38049.22 / 1.749
TabCon94.14 / 1.69898.24 / 2.15699.61 / 3.316

Response B3: Bayesian baseline and efficiency (W3)

In the manuscript, we mainly highlighted the comparison with the bootstrap and asymptotic procedures based on linear models or basis functions, as they are the standard frequentist procedure for approximating the sampling distribution of an estimator, and our proposal lies within the same framework.

Nevertheless, we agree with the reviewer that comparisons with martingale-posterior methods are informative. We had already included AMP in the original submission. To further address this concern, we have now added TabMGP to the runtime comparison in Table 2. As already observed, TabCon is more computationally efficient than AMP, with its advantage increasing as the number of query points grows. The advantage is particularly pronounced over TabMGP, which requires more than three orders of magnitude longer to run. Finally, TabCon requires 16.3 hours of one-off training, which is amortized after 35–51 new tasks when replacing TabMGP (Table 3).

Table 2. Median runtime in seconds on an NVIDIA RTX 6000 Ada Generation GPU.

CCQQAMPTabMGPTabCon
64640.25791154.51700.2036
642560.27421156.15350.2072
6410240.47731217.95980.2297
128640.25951250.72120.2054
1282560.27581252.72400.2082
12810240.48031253.87990.2320
256640.26191310.45030.2072
2562560.27721313.44110.2097
25610240.48211317.38250.2340
512640.26701452.82040.2120
5122560.28251463.29780.2146
51210240.48591458.41260.2376
1024640.27021709.65870.2155
10242560.28821723.74790.2201
102410240.48901711.72420.2403

Table 3. Number of new tasks required for TabCon's one-off training cost to be recovered through inference-time savings over each baseline. The second column reports the break-even count at the largest context and query sizes (C=Q=1,024C=Q=1{,}024), corresponding to the most computationally demanding setting. The final column reports the range across all 15 context and query size configurations.

BaselineMost computationally demanding setting (C=Q=1,024C=Q=1{,}024)Range across 15 settings
Bootstrap5,1945,194–5,682
AMP236,130236,130–1,084,886
TabMGP3535–51
Reviewer_dHYUOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes TabCon: a method for confidence interval estimation based on a tabular foundation model (TabPFN-v3). In particular, additional heads are added to the model and trained to predict the estimation error in addition to the point estimate. The method is evaluated on a set of synthetic data generation processes, measuring coverage and interval length.

Critical strengths

Reduced interval length and coverage closer to nominal levels across evaluated dataset generation processes, compared to a set of baselines.

Critical weaknesses

Limited novelty of the method. The fundamental idea is an additional set of heads for TabPFN-v3, trained for a different task. While authors add significant complexity to this basic setup (specialist heads, MoE, etc.), the value of this additional complexity is not clear due to limited ablation experiments (see below).

Current evaluation is rather limited. Prior misspecification (OOD) evaluation could be stronger: at the moment held-out DGP's are still standard function classes, so might be well-covered by the prior. The main results (Table 1 and Figure 2) don't report error bars, which makes it hard to asses the variance within method's results, and assess the significance of improvement. Most results are based on simple synthetic DGPs: more realistic evaluations (e.g. fit a model to real data and sample from it) would make the results more informative for practical applications. Finally, ablations could be stronger: at the moment, only CP numbers are reported for a subset of the DGP's (no CP/IL numbers across all DGPs), again, without error bars, which makes it hard to understand the importance of the various components. In addition, many choices are not ablated, e.g. dense vs. top-2 routing, the number of experts, RL vs. SFT with a coverage loss, individual reward components, etc.

Questions for authors

How do authors view the problem of scaling this method to large datasets, given the limited context size of transformer-based in-context learners like TabPFN-v3?

What other key limitations of the method and experimental methodology that authors would focus on if continuing this work?

7 comments

We thank the reviewer for the detailed comments, which gave us the opportunity to clarify the novelty and the statistical motivation of our methodology, and to strengthen the empirical evaluation of TabCon in several relevant settings. Below, we address the critical weaknesses W1-W5 and the questions Q1-Q2. To summarize, we mainly have (i) added more challenging out-of-distribution DGPs, (ii) evaluated TabCon on real-data-driven semi-synthetic tasks from all 35 OpenML-CTR23 datasets, (iii) completed the main results with error bars, and (iv) extended the ablation study to assess the role of each component of our method.

Response A1: Methodological novelty and additional complexity (W1)

We would like to clarify that the novelty of our work lies primarily in its inferential target and in the underlying learning principle, rather than in the architecture. At the architectural level, TabCon indeed adds residual heads to a frozen TabPFN backbone. Its contribution, however, is threefold. First, it targets confidence intervals rather than prediction intervals. TabPFN's predictive distribution quantifies the uncertainty of the response YY, whereas TabCon quantifies the estimation uncertainty of f(x)=E[Y∣X=x]f(x)=\mathbb{E}[Y\mid X=x]; for instance, as the sample size grows, a prediction interval does not shrink below the noise level, while a CI for f(x)f(x) shrinks towards zero. We regard amortized CI construction as an important statistical insight, as it represents a first step towards new directions in statistical inference with tabular foundation models and may be further extended to other inferential tools, such as hypothesis tests. Second, existing frequentist methods for this target mainly rely on resampling or refitting and, as in the bootstrap, require the whole procedure to be repeated for every new dataset. TabCon is instead trained once and returns CIs for any new dataset in a single forward pass. Third, both advances rest on the observation that the estimation error f^(x)−f(x)\hat f(x)-f(x), although never observed in real data, is available in every task simulated from the prior, where ff is known. Learning its distribution directly is, to the best of our knowledge, new in the context of tabular foundation models for CI construction.

Beyond the inferential target, the additional components of TabCon are themselves statistically motivated. We discuss this motivation in detail in our response B1 to Reviewer AZfp and provide the corresponding ablations in our response A5 below.

Response A2: Prior misspecification (W2)

We find this a valuable suggestion, as it allows us to test TabCon in scenarios beyond common function families and to better assess the robustness of our methodology under prior misspecification. We therefore added 32 OOD DGPs based on rational functions, B-splines, multiscale wavelets, implicit equations and the Lorenz dynamical system, denoted OOD2 hereafter, whereas OOD1 denotes the OOD families of the manuscript. These mechanisms broaden the functional structures and complexity of the original evaluation and are not generated by the training prior.

On OOD2, TabCon attains 96.08% coverage at the 95% nominal level, with intervals 8.69% shorter than those of AMP, which undercovers (93.89%), and 13.2% shorter than those of PI (Table 1). These results support the generalization of TabCon to function classes beyond those considered in the manuscript.

Table 1. Coverage probability (CP, %) and interval length (IL) on the 32 OOD2 DGPs at the 95% nominal level, evaluated against the known conditional mean f(x)f(x). Brackets report 95% bootstrap intervals obtained by resampling the DGPs. The coverage closest to the nominal level and the shortest IL among methods with coverage at least 95% are shown in bold.

MethodCP (%)IL
Linear62.30 [54.82, 69.54]2.900 [2.172, 3.715]
RFF80.04 [77.49, 82.57]3.122 [2.611, 3.643]
PI97.80 [96.95, 98.56]2.505 [1.931, 3.095]
Bootstrap68.18 [60.98, 75.14]0.828 [0.678, 0.978]
AMP93.89 [92.18, 95.50]2.381 [1.838, 2.946]
TabCon96.08 [94.36, 97.50]2.174 [1.672, 2.685]

Response A3: Error bars and statistical uncertainty (W3)

We thank the reviewer for this observation, which allowed us to complete Table 1 and Figure 2 of the manuscript with 95% bootstrap intervals (Tables 2 and 3).

The error bars allow us to assess the variability of the results across DGPs and datasets. The coverage of TabCon is stable, with error bars narrower than those of Linear, RFF and Bootstrap and, on ID and OOD1, than those of AMP. Moreover, the error bars of TabCon include the nominal level on ID, OOD1 and Stress, whereas those of AMP include it only on ID and OOD1 and lie entirely above it on Stress and CC18. Even in the worst case, the error bars of TabCon do not indicate a substantial undercoverage, as their lower end remains above 93.7% in all suites, whereas that of AMP falls to 89.9% on ID. A similar pattern holds across the stress-test regimes of Figure 2, where the error bars of TabCon never lie entirely below the nominal level and include it in seven of the ten regimes, against three for AMP. Finally, in all four suites of Table 2, TabCon yields the shortest intervals among the methods attaining at least nominal coverage.

In summary, the error bars confirm the conclusions of the manuscript, showing that TabCon achieves stable, near-nominal coverage with shorter intervals than the methods attaining comparable coverage.

Table 2. Coverage probability (CP) and interval length (IL) of Table 1 of the manuscript at the 95% nominal level, with 95% bootstrap intervals over the evaluation units (DGPs or datasets) in brackets. The CP closest to the nominal level and the shortest IL among methods with CP at least 0.95 are shown in bold, and the same rules are applied to the interval bounds.

MethodID CPID ILOOD1 CPOOD1 IL
Linear0.8653 [0.7902, 0.9263]8.5699 [4.7308, 12.9325]0.6814 [0.6201, 0.7415]2.5411 [1.9182, 3.2226]
RFF0.8511 [0.8145, 0.8847]4.0549 [2.4213, 5.8969]0.8029 [0.7738, 0.8326]2.5039 [2.0945, 2.9211]
PI0.9792 [0.9666, 0.9897]4.0261 [2.2202, 6.0593]0.9695 [0.9565, 0.9810]2.1793 [1.7336, 2.6448]
Bootstrap0.5935 [0.5208, 0.6644]0.9200 [0.5194, 1.3762]0.6820 [0.6215, 0.7430]0.8107 [0.6545, 0.9754]
AMP0.9369 [0.8994, 0.9689]3.9612 [2.1798, 5.9658]0.9351 [0.9131, 0.9549]2.0706 [1.6528, 2.5111]
TabCon0.9548 [0.9375, 0.9703]2.6824 [1.5479, 3.9542]0.9531 [0.9384, 0.9666]1.9288 [1.4906, 2.3946]
MethodStress CPStress ILCC18 CPCC18 IL
Linear0.7673 [0.7210, 0.8112]6.9267 [3.2538, 12.8809]0.5377 [0.4244, 0.6439]78.0342 [0.3095, 233.2685]
RFF0.7381 [0.7056, 0.7690]2.6404 [1.9061, 3.4948]0.6238 [0.5301, 0.7170]0.7514 [0.4562, 1.0665]
PI0.9902 [0.9863, 0.9935]3.2091 [2.4510, 4.0482]0.9953 [0.9933, 0.9970]2.6531 [2.1283, 3.1946]
Bootstrap0.6970 [0.6674, 0.7254]0.6651 [0.4550, 0.9274]0.7328 [0.6902, 0.7743]0.4084 [0.3254, 0.4960]
AMP0.9729 [0.9639, 0.9802]3.2663 [2.4228, 4.2364]0.9818 [0.9771, 0.9860]2.4117 [1.9627, 2.8721]
TabCon0.9561 [0.9465, 0.9651]1.8581 [1.2266, 2.6505]0.9654 [0.9519, 0.9779]1.0686 [0.7945, 1.3715]

Response A3: Error bars and statistical uncertainty (W3), continued

Table 3. Coverage (%) and interval length (IL) across the ten stress-test regimes of Figure 2 of the manuscript at the 95% nominal level, with 95% bootstrap intervals in brackets. Bold entries follow the same rule as in Table 2.

(a) Coverage (%)

Stress regimeLinearRFFPIBootstrapAMPTabCon
Near-constant74.70 [69.40, 79.90]72.52 [68.59, 76.26]99.37 [98.93, 99.71]71.67 [67.77, 75.43]98.21 [97.19, 98.99]95.56 [93.94, 96.95]
Bounded74.36 [69.11, 79.36]71.82 [67.88, 75.72]99.37 [98.93, 99.71]71.59 [67.72, 75.40]98.22 [97.24, 98.98]95.55 [93.91, 96.93]
Quantized72.66 [67.07, 78.00]71.11 [67.18, 74.91]99.29 [98.83, 99.65]67.72 [63.91, 71.44]97.98 [96.99, 98.78]94.00 [91.78, 95.85]
Discontinuous74.30 [68.27, 79.82]73.75 [69.00, 78.19]99.39 [99.15, 99.62]70.48 [65.03, 75.61]98.08 [97.12, 98.79]98.42 [97.93, 98.86]
Near-zero noise65.22 [59.57, 70.85]60.09 [56.42, 63.69]98.04 [97.35, 98.66]70.69 [66.43, 74.95]95.41 [93.89, 96.72]95.16 [94.07, 96.14]
High noise88.22 [84.58, 91.54]88.74 [86.38, 90.91]99.91 [99.77, 100.00]76.56 [73.47, 79.50]99.57 [99.07, 99.92]97.09 [95.63, 98.29]
Heavy-tailed noise78.20 [73.01, 83.08]75.28 [71.47, 78.92]99.28 [98.83, 99.65]71.91 [68.09, 75.63]98.22 [97.38, 98.94]94.60 [92.76, 96.25]
Asymmetric outliers83.29 [78.78, 87.55]80.85 [77.37, 84.23]97.90 [95.97, 99.21]67.12 [62.75, 71.28]93.18 [90.91, 95.14]94.00 [91.40, 96.22]
Heteroscedastic79.18 [74.22, 83.95]75.24 [71.44, 78.99]99.46 [99.10, 99.76]72.07 [68.44, 75.54]98.36 [97.47, 99.10]97.04 [95.90, 98.02]
Categorical + missing77.14 [72.10, 81.87]68.68 [64.63, 72.52]98.18 [97.29, 98.89]57.16 [53.20, 61.08]95.69 [94.26, 96.91]94.73 [93.31, 96.02]

(b) Interval length

Stress regimeLinearRFFPIBootstrapAMPTabCon
Near-constant0.0359 [0.0148, 0.0639]0.0207 [0.0128, 0.0300]0.0199 [0.0123, 0.0294]0.0043 [0.0027, 0.0062]0.0196 [0.0125, 0.0282]0.0110 [0.0064, 0.0171]
Bounded3.2461 [1.6181, 5.7313]1.5454 [1.1066, 2.0616]1.7427 [1.2999, 2.2401]0.4241 [0.2848, 0.5952]1.7923 [1.2999, 2.3599]1.1160 [0.7168, 1.6017]
Quantized3.4248 [1.7403, 5.9470]2.0273 [1.2840, 3.0202]1.8310 [1.3723, 2.3671]0.4548 [0.3034, 0.6389]1.8953 [1.3847, 2.5072]1.2283 [0.7968, 1.7557]
Discontinuous6.0256 [3.3421, 9.4570]2.3454 [1.6874, 3.0904]2.5439 [1.9034, 3.2471]0.6711 [0.4675, 0.9005]2.7435 [2.0448, 3.5177]2.1095 [1.4007, 2.9134]
Near-zero noise2.4134 [1.0099, 4.8133]1.4672 [1.0413, 1.9501]0.6205 [0.3657, 0.9620]0.2473 [0.1644, 0.3428]0.6456 [0.3886, 0.9740]0.6552 [0.3920, 0.9901]
High noise13.6168 [7.6785, 20.9729]5.8670 [4.2805, 7.7242]9.6276 [7.5755, 11.8426]1.6825 [1.1434, 2.3545]9.7841 [7.5311, 12.3093]4.7008 [3.1289, 6.6154]
Heavy-tailed noise5.1986 [2.6324, 8.7397]2.5593 [1.7483, 3.5648]3.0465 [2.2054, 3.9749]0.6445 [0.4196, 0.9191]3.1927 [2.2740, 4.2485]1.7041 [1.0229, 2.5690]
Asymmetric outliers13.5983 [6.6416, 22.0067]5.6424 [3.3786, 8.5032]6.8455 [4.7566, 9.3375]1.1702 [0.6776, 1.8256]6.6426 [4.2988, 9.8232]3.1727 [1.3809, 5.7451]
Heteroscedastic5.5041 [2.8251, 9.1442]2.8392 [1.8482, 4.2241]3.3166 [2.4600, 4.2954]0.7290 [0.4756, 1.0425]3.3943 [2.4690, 4.5154]1.8532 [1.1738, 2.7234]
Categorical + missing16.2034 [2.9300, 41.4511]2.0897 [1.4967, 2.7362]2.4965 [1.8280, 3.2897]0.6235 [0.4172, 0.8765]2.5532 [1.8568, 3.3711]2.0306 [1.3711, 2.8081]

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.