acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Causal Prior-Fitted Networks under Unobserved Confounding: Identifiability and Randomized Evidence

Abstract

Prior-Fitted Networks (PFNs) amortize the cost of causal effect estimation across tasks, enabling one-pass in-context prediction on new datasets without task-specific optimization. Yet efficient amortized inference does not by itself ensure causal identifiability: accurate effect estimation also depends on whether the observed context contains sufficient information to determine the causal target. Existing identifiability guarantees for causal PFNs rely on unconfounded settings such as strong ignorability, leaving what can be identified from PFN contexts unresolved under unobserved confounding, where the same context distribution may be compatible with different causal effects. We formalize this through context identifiability: when all compatible mechanisms agree on the target effect, ideal posterior inference recovers it asymptotically; otherwise, an irreducible bias remains that cannot in general be eliminated by more data or more accurate approximation. This framework further shows that broader causal priors can preserve ambiguity and that instrumental-variable identification still depends on additional structural assumptions. We therefore consider limited randomized evidence alongside confounded observational data and show that retaining source labels can restore identification of the target effect. Guided by this result, we propose Randomization-Aware PFN (RA-PFN), which adapts a pretrained causal PFN through mixed-source prior fitting, source embeddings, and randomization-aware attention while preserving one-pass inference. Across three public benchmarks, RA-PFN achieves the lowest causal effect estimation errors under unobserved confounding with limited randomized evidence, while retaining the inference-efficiency advantage of PFN-based estimation without downstream fitting.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Prior-Fitted Network, Causal effect estimation, Unobserved confounding

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_Ht9ROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies identifiability in causal Prior-Fitted Networks under unobserved confounding and establishes conditions under which source-labeled randomized evidence identifies causal effects. It proposes RA-PFN, which adapts CausalPFN using mixed-source prior fitting and corresponding architectural adaptations. Experiments on three semi-synthetic benchmarks report improved CATE and ATE estimation.

Critical strengths
  1. The mixed observational-randomized identification strategy provides a transparent way to construct identifiable synthetic training tasks. Although the conditional mean generalizability from RCT to the observational population generally cannot be verified from the available real-world data, it can be enforced exactly in the simulator. This creates a clear connection between the identification assumptions and the training-task construction.
  2. Reasonable empirical improvement.
Critical weaknesses
  1. There seems to be limited evidence for deployment robustness. Enforcing identification assumptions during synthetic training does not ensure that independently collected observational and randomized datasets satisfy them at deployment. The paper's reported evaluations preserve these assumptions by construction. A controlled sensitivity analysis, particularly introducing violation of conditional mean generalizability from trial participants to the target population, would clarify the method’s practical robustness.

  2. Inspired by equation (5), we can further decompose the error by: \begin{aligned} \left|\widehat{g}_N-g(\psi)\right| \leq \left|\widehat{g}_N-m_N\right| + \left|m_N-\bar{g}_{\pi}(R)\right| + \left|\bar{g}_{\pi}(R)-g(\psi)\right|. \end{aligned} The last two terms are examined by Theorem 3.2.(i) and Theorem 4.1, respectively, but the control of the first term, which is essentially network approximation error, is not established in the paper. Consequently, it does not establish consistency guarantee for the implemented estimator. Maybe an RA-PFN-specific error analysis would help facilitate better understanding of what drives the network approximation error.

  3. The adaptation prior seems to cover a narrow family of hidden-confounding mechanism and mainly focus on low-dimensional settings. Even starting from CausalPFN does not necessarily make the adapted predictions retain the pretrained model’s generalization capabilities. We either need a sensitivity analysis on the prior coverage, or extend the coverage of adaptation priors to more scenarios, e.g. high-dimensional covariates, correlated covariates, multiple latent confounders, nonlinear latent outcome effects, or latent treatment-effect modification.

Questions for authors
  1. Under the identification strategy of causal data fusion, observational outcomes are no longer needed for identification, but they might be helpful for estimation. How much does access to observational treatment and outcome information improve estimation, beyond access to the same randomized sample and observational covariates?
  2. A potential advantage of PFN-based causal estimation is its ability to provide posterior distributions, yet the experiments evaluate only point estimates derived from posterior means. Could the authors also examine the posterior distribution of CATE?
  3. How do accuracy, source weighting, and computational cost change outside the 128-row context window setting?
  4. What explains CausalFM’s relative ATE error of 1.000±0.0001.000\pm0.000 on both CPS and PSID in Table 1?
  5. For Table 1, Appendix C.5 selects among ten configurations for each fusion baseline using source-treatment-stratified validation and a source-balanced R-loss. Given that the main experiment contains only 13 randomized observations, how sensitive are the selected configurations and reported performance to the validation split?
Reviewer_z4JbOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The work proposes a new causal foundational model for treatment effect estimation from the combination of experimental data and confounded observational data (namely, Randomization-Aware Prior-Fitted Network (RA-PFN)). The authors broadly discuss the issue of identifiability under confounded data and how it is revealed in ordinary causal PFNs. As a remedy, they suggested fitting a PFN based on a mixture of both experimental and observational data (which, under the right assumptions, recovers point identification). Finally, the authors demonstrated the effectiveness of the proposed model on several synthetic benchmarks.

Critical strengths

The paper is well-written and mostly easy to follow. To the best of my knowledge, the proposed approach is novel (in the sense of the literature stream of causal PFNs).

Critical weaknesses

My main criticism concerns the limited novelty of the proposed approach: both the causal PFNs and causal identification under exp./obs. data mixture were already broadly covered in the related work. Hence, the paper seems to be a simple re-implementation of a PFN in a new (yet identifiable) causal setting. Given that the previous work already established a blueprint to construct PFNs for any identifiable setting [1], the paper is (in my opinion) a special case of the latter.

References:

  • [1] Ma, Yuchen, et al. "Foundation models for causal inference via prior-data fitted networks." International Conference on Learning Representations. Vol. 2026. 2026.
Questions for authors
  • The authors assume a prior over the SCMs class (ψ∈Ψ\psi \in \Psi). Is it a parametric or a non-parametric prior? Can the authors provide more details?
  • What is the expectation taken over in Eq. 4?
  • What does line 169 mean? “... the identical context laws also rule out consistency under both mechanisms”
  • Why do we need a tailored PFN design (i.e., what the authors did in Sec. 4.2)? Wouldn’t a regular transformer suffice (e.g., as done in TabPFN)? I would expect some sort of ablation study here.

I would be open to raising my score if the authors address all of my concerns and questions.

Reviewer_SkWuOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper is about amortized causal effect estimation with prior-fitted networks (PFNs): a transformer is trained on synthetic SCMs drawn from a prior, and at deployment it receives a dataset as context and outputs, in one forward pass, (an approximation of) the posterior predictive of the conditional expected potential outcome; the point estimate is the posterior mean, and CATE/ATE are obtained by contrasts. The prior work mainly frames around ignorability assumption, and the paper attempts to address unobserved confounding.

Critical strengths

The paper emphasizes an important point for its audience of LLM-powered causal inference: Leveraging heterogenous data in pretraining the PFNs may improve inference even when the effect of interest is not identifiable from conventional adjustment techniques.

Critical weaknesses
  1. The theoretical contribution is unclear: it reads as a restatement of facts in the established prior work, also acknowledged by the paper. Theorem 3.2 seems to be merely a corollary of the results in [1] among other prior work, and does not constitute a novel contribution. Proposition 3.3 is a known fact about causal effects, and the only relevance here is that additional heterogeneity may identify previously non-identifiable causal effects. The remark that follows at line 270 reads as a tautology, although the name "context identifiability" suggests relationship with LLMs, but I can't identify such relationship. Theorem 4.1, again, is a restatement of standard identifiability/transportability results.

  2. The experimental results are not a fair comparison. The baselines are all taken frozen, and the proposed method is the only one getting fine-tuned on the novel data. I assumed that the contribution/message is not "fine-tune the LLM on confounded data before using it for confouded inference", and if this is in fact the main message, I suggest that the authors do not pose the work as a novel method with its own name.

[1] Balazadeh, Vahid, et al. "CausalPFN: Amortized Causal Effect Estimation via In-Context Learning. arXiv preprint, 2025." URL https://github. com/vdblm/CausalPFN.

Questions for authors
  1. How is Def 3.1 any different from regular causal effect identifiability? As far as I can tell, the definition doesn't have anything to do with LLMs, yet it carries a new notion of "context" identifiability?

  2. Have you tried providing the same fine-tuning data to the baselines, and compare the results? The experimental results as presented are significantly improving upon the baseline, but it is hard to understand what is the decisive difference in RA-PFN and any of the baselines.

Reviewer_tX3YOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This study extends the framework of Prior-Fitted Networks (PFNs; Muller et al., 2022) to causal-effect inference under latent confounders. Rather than some existing PFN studies for causal inference based on ignorability assumption (CauslPFN; Balazadeh Meresht et al., 2026) or those requiring additional structural assumptions (Robertson et al., 2026, Ma et al., 2026), this study instead proposes to use limited randomized controlled trials (RCTs) to identify the causal effect. Based on standard arguments for causal effect estimation using RCTs, it shows that conditional expected potential outcomes (CEPOs) can be estimated from randomized evidence. The framework performs prior fitting on mixed observational and randomized data, with bias parameters added to the encoder and attention networks, and predicts potential outcomes for a given query. The numerical experiments show that the proposed method outperforms existing baselines.

Critical strengths
  1. The incorporation of randomized controlled trial (RCT) samples into PFN architecture for causal inference under latent confounders seems new.

  2. The study illustrates some examples where the existing PFN approaches cannot uniquely identify the causal effects under possible influence of unobserved confounders.

Critical weaknesses
  1. Limited performance gains and evaluation scope: Although consistent, the experimental results show rather limited gains over the baselines. Moreover, the evaluation is restricted to three benchmarks (ACIC, CPS, and PSID, with 10, 100, and 100 realizations, respectively), leaving it unclear whether the results generalize across sufficiently diverse data-generating mechanisms.

  2. Experimental results: Further explanation is necessary to clarify the main contribution of this study. In particular, PEHE appears largely unchanged as the RCT data ratio varies from 1% to 30%. How much of the improvement is attributable to randomized evidence rather than continued prior fitting? Although the benefits of randomized evidence may already saturate at 1%, comparisons including a 0% RCT ratio under the same continued prior-fitting procedure, together with results broken down by experimental setting, would help distinguish these explanations.

  3. Limited explanation of the causal prior: The construction of the causal prior in Appendix C.2 is not clearly explained, making it difficult to assess what primarily drives the performance improvements, as discussed above.

Questions for authors

See Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.