- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper explores statistical properties of PFNs, a previously overlooked aspect of PFNs. Primarily, the paper explores the convergence rate of PFN based posterior, and how it can be slow since it inherits the convergence rate of the latent nuisance functions which are implicitly learned by the PFNs. This is a generalisation of the existing theory of the prior-induced bias to the field of PFNs. The authors first outline the theory to show this is true in an infinite training data regime for a PFN. Then the authors then offer two debiasing techniques borrowed from the field of semi-parametric Bayesian statistics and then show theoretically that these debiasing techniques will result in faster convergence rate, improving upon the theoretical convergence rate shown in theorem 1.
Critical strengths
The paper explores aspects of PFNs which have previously been under-explored: the convergence rate of posteriors learned with PFNs. It ties together previously well understood theories in semi-parametrics such as prior-induced bias, and applies them to theory in PFNs to understand the convergence rate of PFN posteriors. The authors propose a de-biasing technique to solve this problem and show that it theoretically, it should lead to benefits in the convergence rate and bias of the learned posterior.
Critical weaknesses
Though theoretically, the authors show that their de-baising techniques would lead to faster convergence rate of the posterior, in appendix F.9 we see that with four times the amount of training data, i.e. sampling 4 times from each world rather than once, the standard PFNs convergence rate greatly improves, with the RMS conditional bias reaching the same level as that of the debiased PFN with n=2048. Its evident that with smaller n, there is some benefit to the de-biased PFN in terms of reducing bias. However, the reduction in bias from 1 sample verses 4 sample for the training data for the standard PFN is notable. This illustrates that the convergence rate of the standard PFN was limited by the amount of data available, which is extremely cheap to generate. The experiments in the main paper, figure 3 is carried out with 1 sample per world to generate the training data. However, much of the difference between the bias standard PFN and the de-biased PFN is not attributed to statistical bias, (the bias due to the difference between the true nuisance parameters and the estimated) but as a result of but as a result of amortisation bias (bias as a result of not being able to fit the nuisance functions due to lack of data). As a result, the theory is not fully supported by the experiments.
Questions for authors
Would it be possible to run experiments that support the theoretical claims such that the bias can be attributed to statistical error or amortisation error? Having some estimate of the true posterior and experiments to support that their de-biasing techniques would result in a reduction of the statistical error would result in their theoretical claims being supported by the experiments.
4 comments
Rebuttal [2/2]
Would it be possible to run experiments that support the theoretical claims such that the bias can be attributed to statistical error or amortisation error?
Thank you for the suggestion! Indeed, our initial experiments did not directly isolate the theoretical benefits of debiasing (i.e., it is difficult to distinguish statistical improvements from amortization errors).
Action. We added experiments with exact posteriors (= idealized PFNs) in the new Appendix I.2. These experiments eliminate neural amortization errors and thus directly isolate the statistical benefits of debiasing. The results are very encouraging: Under , the standard posterior concentrates markedly slower than the parametric rate, whereas both debiased posteriors approximately attain the parametric rate, with substantially lower errors. Furthermore, the debiased posteriors generally approach the efficient Gaussian distribution and maintain close-to-nominal coverage, unlike the standard posterior. => Thus, the benefits of debiasing persist (and become even more pronounced) without amortization errors.
References:
- [1] Valentyn Melnychuk, Vahid Balazadeh, Stefan Feuerriegel, and Rahul G. Krishnan. Frequentist consistency of prior-data fitted networks for causal inference. In International Conference on Machine Learning, 2026.
Thank you for addressing my concerns, I believe that the authors sufficiently addressed the issues I raised related to the empirical results. I will keep my score however, since I am cognisant of the fact that my expertise does not cover the theoretical contributions of the paper, which is a significant contribution of the paper.
Thank you for revisiting our work and confirming that we addressed your empirical concerns (your suggestion motivated our new exact-posterior experiments)! We believe this has significantly strengthened our manuscript, and we greatly appreciate your thoughtful feedback!
Rebuttal [1/2]
We are grateful for your review! Thank you for recognizing the importance of studying posterior convergence rates in PFNs, as well as our contributions in connecting PFNs with semiparametric theory and developing principled debiasing strategies. In the following, we want to address your comments.
Response to Weaknesses & Questions
Upon reading your comment, we realized that the results in the original Appendix F.9 were potentially misleading: Increasing the pretraining data resulted in an unfair comparison, and, ideally, we would like to compare all the PFNs under the same training budget (wrt. both the amount of synthetic data and the number of epochs). Thank you for motivating a more rigorous evaluation!
Action. To address your concerns, we have substantially revised and extended our experimental section:
The new results now clearly support our theoretical findings: Our debiased PFNs asymptotically outperform the standard PFN for target ψ when evaluated at the evaluation prior Πˇ (that emphasizes worst-case distributions), both in terms of L2 errors and frequentist properties (i.e., total variation to the efficient frequentist estimators and frequentist coverage).
Thank you for pointing this out! To better situate our work, we distinguish three groups of baselines for Bayesian semi-parametric inference with different types of errors:
Action. As a remedy, we added (2) a post-hoc correction approach from [1], where we fit a comparable posterior-predictive PFN that (i) targets μa and π (= nuisance PFN) and then (ii) applies a one-step posterior correction (OSPC); see updated Sec. 5 Experiments. Also, we performed (3) the exact posterior experiments (more on that later).
In our new results, the new baseline (Nuisance PFN + OSPC) often outperforms the Standard and Debiased PFNs (as expected, as it eliminates the part of the amortisation error, yet at the cost of an explicit post-hoc computation). However, when considering the exact posteriors, the debiased posteriors outperform the nuisance PFN + OSPC, which conforms to our theory.