- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces adaptive PFNs, a prior-fitted network model that supports post-training adaptation by separating inference over the semantic paths M from the prediction conditional. The semantic paths are interpretable latents tied to the training prior, representing task-specific parameters. The prior over M can be elicited from historical data, similar to empirical Bayes. The method is evaluated on controlled toy experiments and on two real-world forecasting datasets.
Critical strengths
Interpreting the proposed method as Bayesian updating (Equation 4) offers a very interesting perspective.
Critical weaknesses
-
The experimental evaluation appears limited. (1) It would be valuable to go beyond the linear-regression training prior, for example by using a Gaussian Processes training prior with the kernel type and lengthscale as M. (2) An ablation on the dimensionality of M would also be informative, to show how well the method scales to higher-dimensional semantic paths.
-
Baselines: the set of baselines is quite limited. Although the paper mentions PriorGuide [1] and multi-task Bayesian in-context learning [2], it does not compare against them. For example, M could be implemented as the latent parameter in PriorGuide, with the prior over it constructed via empirical Bayes from historical data.
-
Other than above I also have a concern on the broad applicability of semantic path concepts, especially in the case where the semantic paths / latent parameters are not easily defined.
[1] Yang, Yang, et al. "Priorguide: Test-time prior adaptation for simulation-based inference." International Conference on Learning Representations. Vol. 2026. 2026.
[2] Zhu, Qingyang, Eric Karl Oermann, and Kyunghyun Cho. "Multi-Task Bayesian In-Context Learning." arXiv preprint arXiv:2606.20538 (2026).
Questions for authors
-
Would it make sense to extend this approach to a more general setting, such as the SCM-based training priors used by TabPFN/TabICL, i.e., a tabular foundation model rather than a prior-specific PFN or neural process? In particular, can meaningful semantic paths M even be defined for such foundation models? (future works/clarification)
-
The choice of the prior tilt in Equation 2 needs more justification, as its motivation is currently not entirely clear. For example, how does it compare with the power prior [3]? (clarification)
-
Could the target head be made simpler, e.g., a single Gaussian or a GMM with a small number of components as in [4]? Since the conditional prediction given a specific semantic path M should be considerably simpler, would such a head be easier to learn and better behaved? (clarification)
References:
[3] Ibrahim, Joseph G., and Ming-Hui Chen. “Power prior distributions for regression models.” Statistical Science (2000): 46–60.
[4] Chang, Paul E., et al. “Amortized probabilistic conditioning for optimization, simulation and inference.” arXiv preprint arXiv:2410.15320 (2024).
3 comments
Response (2/2)
3. Baselines: PriorGuide and Multi-Task Bayesian ICL.
- The reference these methods would have to match (§3, App. E.6). Like Adaptive-PFN, these methods require dedicated pretraining, here to condition on a prior or on related tasks. Training on the correct deployment prior is what they would need to emulate. A direct PFN retrained from scratch on the deployment prior with the same budget is therefore an informative reference.
- The frozen Adaptive-PFN recovers 97% [89, 104] of its improvement at history length 8.
- At length 32 it is more accurate than the retrained network.
- PriorGuide. It applies the same prior-ratio principle to simulation-based inference with a supplied prior. With M as its latent parameter and a prior estimated from historical data, it corresponds to our density-ratio variant, which we evaluate on Rossmann with a regression-estimated prior (6.4–8.2% vs the direct PFN, three seeds). We state this relation in App. A.
- Real data with informative histories. On Rossmann, Adaptive-PFN reduces WQL by 4.8–6.7% relative to the direct PFN (three seeds, all intervals exclude zero). An output-level correction given the same population information does worse than no correction there.
- External baselines. We also add full and readout-only fine-tuning, and a Chronos-2 reference.
4. Applicability when semantic paths are hard to define. We added a Scope paragraph (§1) and guidance on choosing M (App. I). We recommend exposing simulator choices that are:
- interpretable, so a population change has a stated meaning;
- plausibly population-dependent;
- not determined by a single history.
Q1 — TabPFN-style SCM priors. Natural semantic paths there are high-level generator choices: graph density, mechanism family, noise type. Exposing them requires T1 to infer these choices from one dataset, which is a substantive question we leave to future work and now discuss (App. I). We note, however, that in many tabular PFN constructions the synthetic datasets are generated from a directed acyclic graph (DAG), which offers a natural structure for semantic paths: for example, a path that follows the directed edges connecting the nodes of the DAG.
Q2 — Justifying the tilt; comparison with the power prior (§2, App. B.2).
- Minimum-KL justification. Among all priors whose shift features have given means, the exponential tilt uniquely minimizes KL(p‖p0) (I-projection; Csiszár, 1975). It is therefore the smallest change of the training prior consistent with the shifted moments; we include a short proof.
- Power prior. The power prior of Ibrahim & Chen (2000) is the special case s(m) = log L(m | D_hist), η = a0.
- The difference: the power prior fixes the feature through one historical dataset and treats a0 as given. Adaptive-PFN fixes interpretable features of M and estimates η from a population of deployment histories.
Q3 — A simpler target head. A Gaussian or small Gaussian-mixture head, as in ACE, can be substituted without changing the reweighting, and the adapted mixture over paths stays flexible even if each component is simple. We keep the Riemann head because forecasting targets are skewed and the same head is used by the direct PFN (which was the way it was originally constructed in the literature), keeping comparisons matched (App. I).
Also new. We added a corollary giving a sufficient condition for adaptation to improve on the unadapted predictor, and Rossmann with the paper's own moment estimator (4.8–6.7%, three seeds).
We hope the Gaussian-process experiment, which implements your suggestion directly, and the minimum-KL justification of the tilt address your main concerns. If so, we would be grateful if you would consider updating your score. We would also welcome further suggestions during the discussion.
Dear Reviewer,
We thank you again for your constructive comments. As the discussion closes soon, we wanted to make sure our response reached you. We followed your suggestions directly:
-
GP prior with kernel type and length-scale as M, as you proposed (§3.4, App. H). Adaptation lowers MSE by 7.7–7.8% in every seed. It recovers 95–100% of the log-loss gain that exact Bayes obtains from the deployment prior.
-
Dimensionality of M. We added an ablation over the number of tilted coordinates (1–3), in which an extra unshifted coordinate costs nothing. We also report the path ESS for every experiment (App. E.6, App. I).
-
PriorGuide / Multi-Task BICL. These methods condition on a prior, so their target is a network trained on the deployment prior. The frozen Adaptive-PFN recovers 97% of that network's gain (§3).
-
The tilt in Eq. 2. It is the unique KL-minimal change of the training prior, and the power prior is a special case (§2, App. B.2).
-
SCM priors, a simpler head, and paths that are hard to define. These are discussed in App. I and the Scope paragraph (§1).
If these address your concerns, we would be grateful if your assessment reflected them.
Response (1/2)
We thank the reviewer for concrete, actionable suggestions; we implemented the main one exactly as proposed. All changes are shown in red in the updated PDF.
1. Beyond the linear-regression prior: a GP with kernel type and length-scale as M (new §3.4, App. H). We trained Adaptive-PFN on one-dimensional GP regression with M = (kernel family ∈ {RBF, Matérn-3/2, Matérn-1/2}, length-scale). Exact Bayes is computed on a fine grid as a reference. Deployment tilts toward rough, short-length-scale functions. We fit it with the paper's moment estimator from 128 histories, with three training seeds × three deployment repetitions.
2. Dimensionality of M. The semantic path can be large: in the controlled experiment M has 527,898 discrete configurations plus a continuous coordinate. What matters for reweighting is the dimension and concentration of the tilt. We added two pieces of evidence.
We do not claim scalability to many simultaneously shifted coordinates. App. I states that the method assumes a tilt with few coordinates, and limitation (iii) lists the overlap failures we observe; a systematic sweep over the number of shifted coordinates remains open.
(continued in the next comment)