acceptodds
Under review as a conference paper at ICLR 2027

Debiased Prior-Data Fitted Networks for Amortized Bayesian Semi-Parametric Inference

Abstract

Prior-data fitted networks (PFNs) have recently gained popularity as an end-to-end approach to amortized Bayesian inference. Yet, their statistical behavior for semi-parametric targets, such as means under missingness or causal effects, remains poorly understood. In this work, we develop theoretical foundations for debiased PFNs to perform inference on finite-dimensional targets in general semi-parametric settings. First, we show that, despite being Bayes-optimal with respect to the target, an idealized PFN can exhibit a posterior concentration rate that implicitly depends on the concentration rates of (latent) nuisance functions and can become slower as the prior becomes more diverse. Then, to speed up the target posterior concentration rates, we introduce two generic debiasing strategies based on efficient influence functions: (i) a label correction and (ii) a prior correction. Finally, we show that the debiased PFNs (i) & (ii) possess concentration rates that asymptotically match a convergence rate given by traditional frequentist bias-corrected estimators. In this way, we establish a novel connection between amortized Bayesian inference with PFNs and frequentist semi-parametric efficiency theory. Under additional assumptions, this allows us to guarantee local and global optimality of PFN-based estimators (namely, the Bernstein-von Mises property and minimax optimality, respectively). Therefore, our work provides guidelines for designing debiased PFNs for semi-parametric inference with principled and provable asymptotic guarantees.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
semi-parametric inference, prior-fitted networks, amortized Bayesian inference, causal inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ixqSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper explores statistical properties of PFNs, a previously overlooked aspect of PFNs. Primarily, the paper explores the convergence rate of PFN based posterior, and how it can be slow since it inherits the convergence rate of the latent nuisance functions which are implicitly learned by the PFNs. This is a generalisation of the existing theory of the prior-induced bias to the field of PFNs. The authors first outline the theory to show this is true in an infinite training data regime for a PFN. Then the authors then offer two debiasing techniques borrowed from the field of semi-parametric Bayesian statistics and then show theoretically that these debiasing techniques will result in faster convergence rate, improving upon the theoretical convergence rate shown in theorem 1.

Critical strengths

The paper explores aspects of PFNs which have previously been under-explored: the convergence rate of posteriors learned with PFNs. It ties together previously well understood theories in semi-parametrics such as prior-induced bias, and applies them to theory in PFNs to understand the convergence rate of PFN posteriors. The authors propose a de-biasing technique to solve this problem and show that it theoretically, it should lead to benefits in the convergence rate and bias of the learned posterior.

Critical weaknesses

Though theoretically, the authors show that their de-baising techniques would lead to faster convergence rate of the posterior, in appendix F.9 we see that with four times the amount of training data, i.e. sampling 4 times from each world rather than once, the standard PFNs convergence rate greatly improves, with the RMS conditional bias reaching the same level as that of the debiased PFN with n=2048. Its evident that with smaller n, there is some benefit to the de-biased PFN in terms of reducing bias. However, the reduction in bias from 1 sample verses 4 sample for the training data for the standard PFN is notable. This illustrates that the convergence rate of the standard PFN was limited by the amount of data available, which is extremely cheap to generate. The experiments in the main paper, figure 3 is carried out with 1 sample per world to generate the training data. However, much of the difference between the bias standard PFN and the de-biased PFN is not attributed to statistical bias, (the bias due to the difference between the true nuisance parameters and the estimated) but as a result of but as a result of amortisation bias (bias as a result of not being able to fit the nuisance functions due to lack of data). As a result, the theory is not fully supported by the experiments.

Questions for authors

Would it be possible to run experiments that support the theoretical claims such that the bias can be attributed to statistical error or amortisation error? Having some estimate of the true posterior and experiments to support that their de-biasing techniques would result in a reduction of the statistical error would result in their theoretical claims being supported by the experiments.

4 comments
AuthorsOpenReview ↗

Rebuttal [1/2]

We are grateful for your review! Thank you for recognizing the importance of studying posterior convergence rates in PFNs, as well as our contributions in connecting PFNs with semiparametric theory and developing principled debiasing strategies. In the following, we want to address your comments.

Response to Weaknesses & Questions

Though theoretically, the authors show that their de-baising techniques would lead to faster convergence rate of the posterior, in appendix F.9 we see that with four times the amount of training data, i.e. sampling 4 times from each world rather than once, the standard PFNs convergence rate greatly improves, with the RMS conditional bias reaching the same level as that of the debiased PFN with n=2048. … This illustrates that the convergence rate of the standard PFN was limited by the amount of data available, which is extremely cheap to generate. … As a result, the theory is not fully supported by the experiments. …

Upon reading your comment, we realized that the results in the original Appendix F.9 were potentially misleading: Increasing the pretraining data resulted in an unfair comparison, and, ideally, we would like to compare all the PFNs under the same training budget (wrt. both the amount of synthetic data and the number of epochs). Thank you for motivating a more rigorous evaluation!

Action. To address your concerns, we have substantially revised and extended our experimental section:

  • We removed the original Appendix F.9, as its comparison of differently optimized networks did not provide conclusive evidence about the statistical effects of debiasing.
  • We redesigned our experimental protocol to better reflect our theoretical results: We now report the posterior L2L_2 concentration errors (Theorems 2/4) and total variation distances (Theorem 3) instead of RMS conditional bias. Importantly, we distinguish between evaluation under the training prior Π\Pi and a modified evaluation prior Πˇ\check\Pi emphasizing worst-case distributions (Theorem 1).
  • We added experiments for Example A (mean value) in a new Appendix J to provide further evidence for the effectiveness of our debiasing strategies.

The new results now clearly support our theoretical findings: Our debiased PFNs asymptotically outperform the standard PFN for target ψ\psi when evaluated at the evaluation prior Πˇ\check\Pi (that emphasizes worst-case distributions), both in terms of L2L_2 errors and frequentist properties (i.e., total variation to the efficient frequentist estimators and frequentist coverage).

However, much of the difference between the bias standard PFN and the de-biased PFN is not attributed to statistical bias, (the bias due to the difference between the true nuisance parameters and the estimated) but as a result of but as a result of amortisation bias (bias as a result of not being able to fit the nuisance functions due to lack of data).

Thank you for pointing this out! To better situate our work, we distinguish three groups of baselines for Bayesian semi-parametric inference with different types of errors:

  1. Exact posterior inference (statistical error only). Here, statistical errors arise from inherent uncertainty about the nuisance functions: uncertainty about the marginal covariate distribution typically contributes at the parametric rate, whereas uncertainty about conditional nuisance functions can lead to slower concentration. Example baselines: idealized PFNs (= exact posterior).
  2. Amortized nuisance inference (ammortization error) + post-hoc correction (statistical error). Here, we might use PFNs as inference models for the nuisance functions, which are subsequently used for an explicit post-hoc correction (as suggested in [1]). Hence, in addition to (1) statistical errors, amortization errors arise from approximating the exact nuisance posteriors. Example baselines: nuisance PFN + OSPC ([1]).
  3. End-to-end amortized inference (amortization error only). Here, in addition to (1) statistical errors, amortization errors arise from directly approximating the target posterior with a neural network (which implicitly includes the error from (2)). Example baselines: Standard and debiased PFNs (=our approach).

Action. As a remedy, we added (2) a post-hoc correction approach from [1], where we fit a comparable posterior-predictive PFN that (i) targets μa\mu_a and π\pi (= nuisance PFN) and then (ii) applies a one-step posterior correction (OSPC); see updated Sec. 5 Experiments. Also, we performed (3) the exact posterior experiments (more on that later).

In our new results, the new baseline (Nuisance PFN + OSPC) often outperforms the Standard and Debiased PFNs (as expected, as it eliminates the part of the amortisation error, yet at the cost of an explicit post-hoc computation). However, when considering the exact posteriors, the debiased posteriors outperform the nuisance PFN + OSPC, which conforms to our theory.

AuthorsOpenReview ↗

Rebuttal [2/2]

Would it be possible to run experiments that support the theoretical claims such that the bias can be attributed to statistical error or amortisation error?

Thank you for the suggestion! Indeed, our initial experiments did not directly isolate the theoretical benefits of debiasing (i.e., it is difficult to distinguish statistical improvements from amortization errors).

Action. We added experiments with exact posteriors (= idealized PFNs) in the new Appendix I.2. These experiments eliminate neural amortization errors and thus directly isolate the statistical benefits of debiasing. The results are very encouraging: Under Πˇ\check\Pi, the standard posterior concentrates markedly slower than the parametric rate, whereas both debiased posteriors approximately attain the parametric rate, with substantially lower L2L_2 errors. Furthermore, the debiased posteriors generally approach the efficient Gaussian distribution and maintain close-to-nominal coverage, unlike the standard posterior. => Thus, the benefits of debiasing persist (and become even more pronounced) without amortization errors.

References:

  • [1] Valentyn Melnychuk, Vahid Balazadeh, Stefan Feuerriegel, and Rahul G. Krishnan. Frequentist consistency of prior-data fitted networks for causal inference. In International Conference on Machine Learning, 2026.

Thank you for addressing my concerns, I believe that the authors sufficiently addressed the issues I raised related to the empirical results. I will keep my score however, since I am cognisant of the fact that my expertise does not cover the theoretical contributions of the paper, which is a significant contribution of the paper.

Thank you for revisiting our work and confirming that we addressed your empirical concerns (your suggestion motivated our new exact-posterior experiments)! We believe this has significantly strengthened our manuscript, and we greatly appreciate your thoughtful feedback!

Reviewer_srzDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper attempts to theoretically explain how directly using existing PFNs to estimate parameters such as the ATE can be inefficient due to poor posterior contraction properties. To fix this, it proposes incorporating semiparametric bias correction into PFN pretraining through corrected labels or a modified prior. It provides theoretical justification of these fixes as well as some experimental evidence.

Critical strengths
  1. The paper addresses an important question: when can PFN posteriors provide reliable frequentist inference for semiparametric targets? Its connection between amortized Bayesian inference and semiparametric efficiency provides a useful framework for studying this question.
  2. Incorporating correction into pretraining is appealing, particularly because synthetic prior draws provide access to the true nuisance functions.
Critical weaknesses
  1. The theory assumes the conclusions, and leaves assumptions unverified. In the main paper, Theorems 1 and 2 are stated as "short version", whose full versions in the appendix include a lot more assumptions. This is both misleading and furthermore the assumptions largely assume the most important conclusions. To be more concrete, Theorem 1 asserts that a posterior can attain the nuisance rate, but the appendix establishes sharpness only under additional posterior-mass assumptions (55)–(56), without verifying them for a concrete prior. For the corrected methods, equation (71) assumes the required first-order cancellation, while the BvM proof assumes Gaussian convergence of the leading term and invokes an unspecified total-variation stability condition.

  2. The experiments do not establish the claimed rate improvement. Appendix F.9 shows that increasing pretraining data substantially narrows the bias gap and changes the fitted slopes. Furthermore, separate networks are fitted at each sample size with single training seeds. The incomplete optimization make it difficult to distinguish statistical rates from training error.

Questions for authors
  1. Can you describe or give evidence that a concrete prior and truth satisfies the posterior-mass assumptions (55)–(56), so that Theorem 1 yields genuinely slower-than-parametric target contraction?
  2. Does the empirical rate advantage persist across training seeds when both methods are adequately optimized? Given F.9, how can the observed slopes be attributed to the proposed statistical mechanism rather than differences in training error?
8 comments
AuthorsOpenReview ↗

Rebuttal [1/3]

Thank you for your detailed review of our paper! We appreciate your recognition of the importance of our work and your constructive feedback, which helped us strengthen both the theory and experiments. In the following, we want to address your concerns and questions.

W1/Q1. Assumed conclusions & unverified assumptions

Thank you for raising these important issues! Let us elaborate on each point:

In the main paper, Theorems 1 and 2 are stated as "short version", whose full versions in the appendix include a lot more assumptions.

We appreciate your comment. Given the limited length of the main part, we moved almost all the technical details to the Appendix and only mentioned the main assumptions in the main text (e.g., slow concentration of the second-order remainders).

Action. To increase the transparency and readability of our theoretical results, we structured the assumptions throughout Appendix E (Theoretical Results): now they are labeled with letters from (a) to (n) (marked in blue; see our updated PDF). In the main text, we now also explicitly summarize all the assumptions for Theorems 1–4, while keeping their detailed formulations and proofs in Appendix E.

The theory assumes the conclusions, and leaves assumptions unverified. …

Indeed, to demonstrate posterior concentration rates, the properties of debiasing, and asymptotic optimality, it is common in semi-parametric Bayesian theory to assume multiple regularity conditions. Furthermore, those are relatively nontrivial (see, e.g., the original works on prior and posterior corrections [1,2]).

Importantly, several of our assumptions can be verified explicitly, such as the von-Mises expansion and Lipschitz continuity ((a)-(b), derived for Examples A-C), finite-moment conditions ((c), (h-i), under suitable regularity), and the scaling of the prior correction variance (controlled by design). Moreover, in Theorem 2, the parametric first-order rate of the label correction is derived from the Bayesian bootstrap construction, rather than assumed. On the other hand, the more demanding assumptions concern the behavior of the posterior itself: e.g., posterior mass in target-relevant directions ((d)--(f), Theorem 1), stochastic equicontinuity and first-order cancellation under prior correction ((g), (h-ii), Theorem 2), negligible second-order remainders and BvM regularity ((j)-(k), Theorem 3), and uniform risk bounds and matching minimax rates ((l)-(n), Theorem 4). Their rigorous verification requires a separate analysis of the specific prior and data-generating distribution (which we leave for future work). To the best of our knowledge, even general posterior concentration guarantees for publicly available PFNs like TabPFN remain unknown.

In our experiments, several regularity conditions are satisfied by construction (e.g., Gaussian outcome models, strong overlap, and access to the ground-truth nuisance functions). However, our experiments are primarily empirical in nature (in the same way that PFNs are): While they do not formally verify all the assumptions, they still support our theoretical findings. Overall, we acknowledge that verifying all the conditions for general PFN priors is an important direction for future work (as we now mention in Appendix K “Limitations & Future Work”). Nevertheless, our results establish a general theoretical framework characterizing when slow concentration arises and when debiasing yields improved rates and valid uncertainty quantification.

Action. We added the considerations above and the detailed verification/discussion of every assumption (a)-(n) to a new Appendix G “Verification of Assumptions & Practical Considerations”.

… the assumptions largely assume the most important conclusions … Theorem 1 asserts that a posterior can attain the nuisance rate, but the appendix establishes sharpness only under additional posterior-mass assumptions (55)–(56) …

Our Theorem 1 provides a general upper bound for any pathwise differentiable target, and further acknowledges the existence of specific “worst-case” posteriors that achieve this upper bound.

Action. We clarified in Sec. 4.1 (see the updated PDF) that “... for a posterior given by a generic non-parametric Bayesian model (and idealized PFNs can be viewed as such), the squared <posterior L2L_2> error is only upper-bounded by <the first-order error>, with equality achieved for specific, worst-case posteriors…”

AuthorsOpenReview ↗

Rebuttal [2/3]

…without verifying them for a concrete prior. … Can you describe or give evidence that a concrete prior and truth satisfies the posterior-mass assumptions (55)–(56), so that Theorem 1 yields genuinely slower-than-parametric target contraction?

Great suggestion!

Action. We have added a new Figure 2 and a new Remark 2 in Appendix E:

  • In Figure 2, we illustrate the sharpness mechanism of Theorem 1 using two concrete Gaussian process (GP) posteriors for Example B (mean under MAR). We show that, unlike a generic Matérn GP posterior, an oversmoothed GP posterior can induce nuisance errors that barely cancel when integrated over the covariate distribution. Consequently, the target posterior concentrates at nearly the same slow rate as the nuisance functions, illustrating how the worst-case behavior of Theorem 1 can arise in practice.
  • In Remark 2, we establish a formal connection between the first-order error in Theorem 1 and the concept of prior-induced confounding/selection bias in causal inference. Specifically, we derive a lower bound on the first-order error FψF_\psi in terms of the difference between the true degree of observed selection bias and its posterior-averaged magnitude. Thus, when the prior favors nearly unconfounded distributions despite substantial selection bias in the ground truth, the first-order error cannot vanish, providing a concrete interpretation of the mechanism behind slow posterior concentration.

For the corrected methods, equation (71) assumes the required first-order cancellation …

We agree that Eq. (71) (= Eq. 78 in the updated PDF) is a nontrivial assumption, but it can be verified for concrete nonparametric priors. For example, consider Gaussian process with f∼GP(0,k)f\sim\mathrm{GP}(0,k), unit noise variance, and least-favorable perturbation fλ=f+λf_\lambda=f+\lambda, where λ∼N(0,σn2)\lambda\sim N(0,\sigma_n^2). By Gaussian conjugacy,

λ∣f,Dn∼N(nPnϕfn+σn−2,1n+σn−2).\lambda\mid f,\mathcal D_n \sim N\left(\frac{nP_n\phi_f}{n+\sigma_n^{-2}},\frac{1}{n+\sigma_n^{-2}}\right).

Thus, with σn2=n−1/2\sigma_n^2=n^{-1/2}, the required OP0(n−1/2)O_{P_0}(n^{-1/2}) cancellation follows whenever the original GP posterior has bounded second moments of PnϕfP_n\phi_f. Conversely, the condition may fail when the original prior assigns extremely little mass near the truth, leading to large perturbations and non-negligible higher-order errors.

We acknowledge that verifying such assumptions for concrete PFN priors requires further work. Nevertheless, we view our paper as an important first step toward theoretically grounded semi-parametric inference with PFNs, establishing a general framework for understanding and correcting their statistical properties.

while the BvM proof assumes Gaussian convergence of the leading term and invokes an unspecified total-variation stability condition.

Indeed, in our original formulation, we directly assumed Gaussian convergence of the leading term and invoked an implicit total-variation stability condition. We agree that these assumptions were too close to the desired conclusion.

Action. We have substantially revised Theorem 3 and its proof (Appendix E), by replacing these assumptions with more general, explicit regularity conditions: (i) consistency of the empirical EIF variance for label correction, and (ii) posterior localization, local asymptotic normality, and prior flatness for prior correction. Importantly, we now derive the Gaussian convergence from the Bayesian bootstrap CLT and a local Laplace argument, respectively, and explicitly address convergence in total variation. Thus, our revised theorem no longer directly assumes the desired asymptotic normality.

AuthorsOpenReview ↗

Rebuttal [3/3]

W2/Q2. Experiments & claimed rate improvement

Appendix F.9 shows that increasing pretraining data substantially narrows the bias gap and changes the fitted slopes. … The incomplete optimization make it difficult to distinguish statistical rates from training error.

Given F.9, how can the observed slopes be attributed to the proposed statistical mechanism rather than differences in training error?

Upon reading your comment, we realized that the results in the original Appendix F.9 were potentially misleading: Increasing the pretraining data resulted in an unfair comparison, and ideally we would like to compare all the PFNs under the same training budget (wrt. both the amount of synthetic data and number of epochs). More importantly, these experiments did not directly isolate the theoretical benefits of debiasing (i.e., it is difficult to distinguish statistical improvements from amortization errors). Thank you for motivating a more rigorous evaluation!

Action. To address your concerns, we have substantially revised and extended our experimental section:

  • We removed the original Appendix F.9, as its comparison of differently optimized networks did not provide conclusive evidence about the statistical effects of debiasing.
  • We redesigned our experimental protocol to better reflect our theoretical results: We now report the posterior L2L_2 concentration errors (Theorems 2/4) and total variation distances (Theorem 3) instead of RMS conditional bias. Importantly, we distinguish between evaluation under the training prior Π\Pi and a modified evaluation prior Πˇ\check\Pi emphasizing worst-case distributions (Theorem 1).
  • We added experiments with exact posteriors (= idealized PFNs) in the new Appendix I.2. These experiments eliminate neural amortization errors and thus directly isolate the statistical benefits of debiasing. The results are very encouraging: Under Πˇ\check\Pi, the standard posterior concentrates markedly slower than the parametric rate, whereas both debiased posteriors approximately attain the parametric rate, with substantially lower L2L_2 errors. Furthermore, the debiased posteriors generally approach the efficient Gaussian distribution and maintain close-to-nominal coverage, unlike the standard posterior. => Thus, the benefits of debiasing persist (and become even more pronounced) without amortization errors.
  • We added experiments for Example A (mean value) in a new Appendix J to provide further evidence for the effectiveness of our debiasing strategies.

Furthermore, separate networks are fitted at each sample size with single training seeds.

This design choice is largely orthogonal to our theoretical contributions, which concern posterior concentration for each fixed dataset size nn. We trained separate networks to simplify benchmarking and isolate errors across sample sizes. Nevertheless, further amortization across different dataset sizes is possible and remains an interesting direction for future work.

Does the empirical rate advantage persist across training seeds when both methods are adequately optimized?

We agree that evaluating robustness across training seeds is important. As noted above, we added exact-posterior experiments to isolate the statistical effects of debiasing.

Action. Thank you for the suggestion! We have launched experiments with four additional random seeds for both synthetic data generation and PFN training on a larger compute cluster. Given the substantial training time, we expect to have the results by the end of the rebuttal period and will update our response accordingly.

We hope to have resolved all of your concerns! Let us know if you have any follow-up questions.

References:

  • [1] Kolyan Ray and Aad van der Vaart. Semiparametric Bayesian causal inference. The Annals of Statistics, 48(5):2999–3020, 2020.
  • [2] Andrew Yiu, Edwin Fong, Chris Holmes, and Judith Rousseau. Semiparametric posterior corrections. Journal of the Royal Statistical Society Series B: Statistical Methodology, 87(4):1025–1054, 2025.
Reviewer_M9ibOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies posterior concentration, debiasing, and convergence rates for prior-data fitted networks (PFNs) in the semi-parametric setting, where the target is finite-dimensional but the model includes infinite-dimensional nuisance parameters. The paper demonstrates that, in the semi-parametric setting, a PFN can be Bayes-optimal and still suffer slow posterior concentration. The paper proposes label and prior corrections to remedy the slow concentration and formalizes the asymptotics, showing that the debiased PFN can achieve a fast concentration rate and asymptotic normality. The paper validates its label correction scheme against a standard PFN, showing concentration rates consistent with its theoretical claims.

Critical strengths

The paper identifies that, by Bayesian self-consistency, a PFN’s training labels act as posterior draws, so a posterior correction previously applied post hoc can instead be built into the training labels. This yields the proposed correction scheme, which the authors show inherits the guarantees of classical debiased estimators, including parametric concentration rates, a Bernstein–von Mises property, and minimax-rate optimality. To my knowledge, this is the first work to provide such foundations for PFNs used in semi-parametric inference, and it gives PFN developers a principled, estimand-agnostic way to obtain valid uncertainty quantification.

Critical weaknesses

From an application perspective, two issues limit the paper’s impact. First, the label correction requires computing the nuisance functions of each prior draw during pretraining. In the synthetic experiments this is easy by construction, but for richer priors, such as those used in CausalPFN, the nuisances may not be available in closed form. The paper does not discuss how the correction would be computed in such settings or how approximation errors would affect Theorems 2 and 3. Second, the experiments compare only standard and debiased PFNs on a single synthetic prior for the ATE, without frequentist debiased estimators (A-IPTW, TMLE) or the post-hoc correction [1]. As a result, it is unclear when a practitioner would prefer a debiased PFN over these existing alternatives.

Reference

[1] Melnychuk, Valentyn, et al. "Frequentist Consistency of Prior-Data Fitted Networks for Causal Inference." arXiv preprint arXiv:2603.12037 (2026).

Questions for authors
  • For a PFN like Do-PFN or CausalPFN, how can we compute the nuisance function during training to perform label correction? In particular, when they are not available in close form?
4 comments
AuthorsOpenReview ↗

Rebuttal [1/2]

We are grateful for your positive review of our work! We appreciate that you recognize the novelty of our theoretical contributions and their potential to provide a principled foundation for valid uncertainty quantification with PFNs in semi-parametric settings. In the following, we address the mentioned weaknesses and questions.

W1/Q1. Computability of nuisance functions

Great point; access to the explicit nuisance functions is crucial for our method!

In our work, we distinguish two groups of nuisance functions: (1) those that have a direct variational influence on the target, and (2) those that additionally appear in the efficient influence function (EIF) or as a solution to the operator equation in prior correction (we can also refer to (1) as primal nuisance functions and to (2) as dual nuisance functions, in line with the terminology of [1]). In the context of ATE, (1) the primal nuisance functions are the conditional outcome mean μa(x)\mu_a(x) and the marginal covariate density P(x)P(x); and (2) the dual nuisance functions are, e.g., the propensity score π(x)\pi(x) and the conditional outcome variance σ2(x)\sigma^2(x). Then, in order to perform the Bayesian semi-parametric inference with PFNs, we inadvertently require access to (1) the primal nuisance function, given that they define the target of inference ψ\psi. Specifically, they either have to be inferred post-hoc (as suggested by [2]) or at the PFNs' training stage (i.e., as part of the synthetic prior, as it is done in our paper). Our debiasing strategies then require additional access to (2) the dual nuisance functions. Thus, we see the mentioned weakness as a fundamental trade-off: For the reliable semi-parametric inference with PFNs, we must sacrifice some flexibility in defining the synthetic prior process (e.g., the original prior for TabPFN, Do-PFN, or CausalPFN would need to be modified, as it samples data directly from complex SCMs w/o explicitly defining conditional densities / conditional expectations/propensity scores).

Still, there is a computational workaround here: Given that we can sample any amount of synthetic data, we can approximate the implicit nuisance functions arbitrarily well with a Monte Carlo estimator. In fact, in our ATE experiments, we never inferred the explicit marginal densities P(x)P(x) but rather used a much larger sample from it (up to 2^22 ≈ 4M samples compared to the maximum dataset size Dn\mathcal{D}_n with $ n=4096$) for the MC estimator of the ground-truth synthetic labels ψ\psi. Here, the MC sample size was chosen relative to the asymptotic variance given by the EIF, and, thus, its relative contribution to the final error was negligible by design. We refer to the computational details in Appendix H.1.1 Prior Processes of the updated PDF (we highlighted the relevant details in blue). In principle, the same MC-based approximation can be extended to other nuisance functions, e.g., the conditional outcome mean μa(x)\mu_a(x) or the propensity score π(x)\pi(x), provided that the approximation error is sufficiently controlled. This, in turn, would enable richer synthetic prior processes.

Action. We added the above-mentioned trade-off of our method to the “Experimental limitations” section (Appendix K.1 in the updated PDF).

AuthorsOpenReview ↗

Rebuttal [2/2]

W2. Additional baselines and experiments

Thank you for the suggestions!

Action. We have refactored and extended our experimental section:

  1. We added the experiments for Example A (mean value); see Appendix J in the updated PDF.
  2. For Example C (ATE) experiments, we added the frequentist debiased estimators (A-IPTW, TMLE) with the ground-truth nuisance functions (that serve as oracle estimators); see updated Sec. 5 Experiments.
  3. Also, for Example C (ATE), we added a post-hoc correction approach of [2], where we fit a comparable posterior-predictive PFN that (i) targets μa\mu_a and π\pi (= nuisance PFN) and then (ii) applies a one-step posterior correction (OSPC); see updated Sec. 5 Experiments.
  4. Finally, we added the frequentist debiased estimators (A-IPTW, TMLE) with the nuisance functions estimated by the nuisance PFN (the posterior mean from the PFN is taken as a frequentist nuisance estimator). In this way, we ensure fair comparison with other baselines; see updated Sec. 5 Experiments.

After additionally refactoring our experimental protocol (e.g., now we report the L2L_2 errors (=Theorems 2/4) and total variation distances (=Theorem 3) instead of RMS bias, and we split the evaluation between the training prior Π\Pi and a modified evaluation prior Πˇ\check\Pi emphasizing worst-case distributions), our new results better reflect our theoretical findings. Specifically, under the evaluation prior Πˇ\check\Pi, both of our debiased PFNs outperform the standard PFN for sufficiently large nn and attain faster concentration rates. The nuisance PFN + OSPC [2] and the A-IPTW/TMLE with the nuisance PFN achieve even lower L2L_2 errors, yet at the cost of an explicit post-hoc computation. However, judging by the uncertainty quantification benchmark (new Appendix I.1), our (ii) prior correction attains the highest coverage among all the comparable methods. Moreover, w/o amortization errors (i.e., with the exact posteriors), our debiasing strategies outperform both the standard posterior and the nuisance PFN + OSPC and approximately attain the parametric rate. Hence, our debiased PFNs are preferable whenever post-hoc computation at inference time should be avoided.

We hope to have resolved all of your concerns! Let us know if you have any follow-up questions.

References:

  • [1] Chernozhukov, Victor, Whitney K. Newey, and Rahul Singh. "Debiased machine learning of global and local parameters using regularized Riesz representers." The Econometrics Journal 25.3 (2022): 576-601.
  • [2] Valentyn Melnychuk, Vahid Balazadeh, Stefan Feuerriegel, and Rahul G. Krishnan. Frequentist consistency of prior-data fitted networks for causal inference. In International Conference on Machine Learning, 2026.
Reviewer_M9ibOpenReview ↗

Reviewer responds

Thank you for the detailed response. The distinction between primal and dual nuisance functions clarifies which parts of the method need explicit access to nuisance functions and which do not. I appreciate that you now state the resulting trade-off openly, namely that the synthetic prior has to be restricted, or the nuisance functions approximated by Monte Carlo, and you’ve added this to the limitations section.

I have read the updated PDF and confirm that the changes you describe are there. The frequentist baseline additions make the empirical comparison considerably more convincing.

I have no further questions. However, I will hold my score until the new theoretical material added in response to the other reviewers has been checked, since it changes parts of the paper my current assessment was based on.

Thank you for taking the time to carefully review our updated manuscript and for your positive feedback! Your suggestions helped us substantially improve the paper, and we are truly grateful for your constructive engagement throughout the rebuttal.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all the reviewers for their thoughtful and insightful reviews and actionable recommendations on how to improve our work! The reviewers highlighted the importance of studying posterior concentration in PFNs, the novel connection to semi-parametric efficiency theory, and the potential of our debiasing strategies.

We addressed all the concerns and questions and made the following main improvements to our paper (all the changes are marked in blue; see our updated PDF):

  1. Restructured & refactored experiments. We redesigned our experimental protocol to better reflect our theoretical results: We now report the posterior L2L_2 concentration errors (Theorems 2/4) and total variation distances (Theorem 3) instead of RMS conditional bias. Importantly, we distinguish between evaluation under the training prior Π\Pi and a modified evaluation prior Πˇ\check\Pi emphasizing worst-case distributions (Theorem 1).
  2. Additional baselines. We added the frequentist debiased estimators (A-IPTW, TMLE) with both the ground-truth and the PFN-estimated nuisance functions, the post-hoc correction of Melnychuk et al. (2026) (= nuisance PFN + one-step posterior correction), and experiments with the exact posteriors (i.e., w/o amortization errors).
  3. More transparent structure of assumptions & practical considerations. We reorganized the assumptions in Appendix E (Theoretical Results) by labeling them from (a) to (n) and now explicitly summarize them in the main text. Furthermore, we introduced a new Appendix G (Verification of Assumptions & Practical Considerations) and expanded Appendix K (Limitations & Future Work), providing concrete examples, practical insights, and directions for extending our theoretical framework to broader PFN settings.

Thanks to your suggestions, we have greatly improved the quality of our paper during the rebuttal. We are confident that our paper is a good fit for the ICLR 2027 conference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.