acceptodds
Under review as a conference paper at ICLR 2027

A Causal Foundation Model for Structure and Outcome Prediction

10 views

Abstract

We introduce TabPFN-CFM, a causal foundation model that addresses causal structure learning and causal outcome estimation within a single amortized inference network. TabPFN-CFM is a prior-fitted network trained purely on synthetic structural causal models. Using only an observational dataset and, optionally, prior knowledge of the causal graph, it predicts the posterior causal structure, including unobserved confounding, and distributions for observational, interventional and counterfactual queries, covering all three levels of Pearl's Causal Hierarchy. Representing structure with acyclic directed mixed graphs lets the model express latent confounding, and we show that conditioning on a known graph can only improve predictive accuracy. Evaluated on in-distribution, out-of-distribution and semi-synthetic datasets with established causal graphs, TabPFN-CFM improves over meta-learner and existing prior-fitted-network baselines for interventional prediction and over both classical and deep structure learners for graph recovery, while also supporting counterfactual queries. A set of backbone and optimizer changes reduces the number of training steps needed to reach a target loss by roughly .

Primary area:
causal reasoning
Keywords:
Prior fitted network, Causal inference, Causal structure prediction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8uTrOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces TabPFN-CFM, a foundation model trained exclusively on synthetic SCMs, that allows for both Structure Learning and inference on all three rungs of Pearl's Causal Ladder. As its name suggests, the model is adapted off TabPFN. The results are strong, especially on more causality intensive tasks: it beats every baseline they tested against, except on Observational generation, where it still holds up. The advantage also holds to out of distribution and real world data.

Critical strengths
  • The model seems clearly state of the art, makes the reasonable assumption that Structure Learning and Causal Inference are parts of the same task (SCM modelling, I guess it could called). And uses that assumption to make a model that is the best of both worlds.
  • The paper is overall clear, and the authors made sure to evaluate their model even in unfavorable conditions, where it still performs, even though the lack of access to ground truth SCMs in real world systems is always a challenge for the evaluation of these models.
Critical weaknesses

I wouldn't say there are "critical" weaknesses in this paper.

Questions for authors

1- How does your model compare to the others in terms of compute? Is there a trade-off here?

2- I find that the baseline of N-ADMG [1] is missing, could you share some results including it?

[1] Ashman et al., “Causal Reasoning in the Presence of Latent Confounders via Neural ADMG Learning,” ICLR 2023.

1 comment

We appreciate your feedback and the time you took to review our paper. We are glad that you found our work clear and informative, and we appreciate your positive comments regarding the strengths of our model.

Compute times comparison

Compared to the outcome prediction methods, our method uses a similar amount of time to make predictions. On our H100 GPU, all methods make predictions in under 100ms / example (10 samples per second) in our tests with up to 1024 observations. This is because all models are tabular transformers with similar sizes and setup. The overhead of our graph embedding layer is only a few ms per example.

Compared to the structure prediction, our method is again similar in speed compared to AVICI and Arrow which are tabular transformers. The classical baselines use the causal-learn package and are run on the cpu. With 1024 observations on a 16 core CPU. LiNGAMICA (52ms / example) and PC (760ms / example) are faster than GES (3.7s / example) and FCI (10s / example). Apart from LiNGAMICA, the other methods are substantially slower than the neural methods.

N-ADMG

We evaluated N-ADMG on the same out-of-distribution Fourier-feature dataset with 1024 observations as used in the main paper. The TabPFN-CFM outcome results below are those reported in Table 1.

Graph prediction (edge accuracy in %, AUROC; higher is better)

ModelEdge typeAUROC
N-ADMG [1]Directed0.625
N-ADMG [1]Hidden0.392
TabPFN-CFMDirected0.875
TabPFN-CFMHidden0.795

Outcome MSE (lower is better)

ModelObservationalInterventionalCounterfactual
N-ADMG [1]0.7221.1381.203
TabPFN-CFM (no graph)0.5340.7550.603
TabPFN-CFM (with graph)0.5340.7340.575

TabPFN-CFM has substantially better accuracy for outcome prediction and graph prediction than N-ADMG. We attribute this to our method's PFN setting, where we are less likely to overfit on the relatively small datasets as well as our refined training procedure. Furthermore, N-ADMG requires training a new model for each dataset, which is much slower, at around 5 minutes per dataset compared to TabPFN-CFM which takes around 0.1 second.

Reviewer_KCpuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces a network trained on synthetic SCM which achieves the capability to predict the observational, interventional and counterfactual distributions given the observational data, the query and optionally the causal graph as input. This approach is novel compared to existing work as they can address the presence of unobserved confounders. The authors show extensive empirical results on various synthetic and real-world datasets.

Critical strengths

The addressed problem is very important since unobserved confounders are very common in real-world datasets. I appreciate the presentation quality of the paper. The authors performed detailed experiments demonstrating the approach capability in all three causal layers compared to baselines. The ablation studies help to understand the role of the causal graphs and evaluation in real-world dataset helps to understand the capability in complex setups.

Critical weaknesses

Below i share my concerns.

#1. There are many interventional and counterfactual queries that are non-identifiable, i.e., they cannot be estimated uniquely [1,2,3]. Existing research instead constructs bounds for such queries. Also, there are observational distributions where the causal graph cannot be uniquely determined. How does the proposed method solve such scenarios? For example, how are confounding, ancestors, and causal effects detected in non-identifiable situations: X→Y, X←Y, X↔Y, X→Y + X↔Y, and X←Y + X↔Y? All of these graphs represent the same observational distribution. Also, Figure 1 in [3] shows causal graphs where P(y|do(x)) is non-identifiable. Can the authors share their results on those cases and show where their predictions lie within the bounds?

[1] Nasr-Esfahany, Arash, and Emre Kiciman. "Counterfactual (non-) identifiability of learned structural causal models." arXiv preprint arXiv:2301.09031 (2023).
[2] Shpitser, Ilya, and Judea Pearl. "What counterfactuals can be tested." arXiv preprint arXiv:1206.5294 (2012).
[3] Shpitser, Ilya, and Judea Pearl. "Complete identification methods for the causal hierarchy." (2008).

#2. I understand Eq. 2 allows the model to learn how to predict interventional distributions (Proposition 1). But which step is enabling counterfactuals? What is the guarantee that the model is doing the abduction, action, and prediction steps? Is it enough to only receive the factual outcome and match the cross-entropy loss? To my understanding, the authors did not take any additional steps for counterfactual generation, such as abduction. All queries just use the cross-entropy loss.

#3. To my understanding, the authors did not empirically demonstrate the difference between the values of observational, interventional, and counterfactual queries. In trivial scenarios, all three causal layers can collapse and become very similar, resulting in a small prediction error. Can the authors show how their performance changes when the distance between these queries gradually increases?

#4. There are many approaches that perform interventions, not just T-learners and S-learners. For example, [4] estimates causal effects by parameterizing an SCM with neural networks; [5] CausalGAN represents a DAG with GAN networks mimicking an SCM and generates high-dimensional interventions; [6] Modular-DCM learns an SCM containing high-dimensional variables in a modular fashion in the presence of unobserved confounders; [7] extends NCM for continuous data; and [8] proposes causal normalizing flows. Can you compare with these approaches on any specific dataset?

[4] Xia, Kevin, Kai-Zhan Lee, Yoshua Bengio, and Elias Bareinboim. "The Causal-Neural Connection: Expressiveness, Learnability, and Inference." Advances in Neural Information Processing Systems 34 (2021).\ https://proceedings.neurips.cc/paper_files/paper/2021/file/5989add1703e4b0480f75e2390739f34-Paper.pdf [5] Kocaoglu, Murat, et al. "Causalgan: Learning causal implicit generative models with adversarial training." arXiv preprint arXiv:1709.02023 (2017).
[6] Rahman, Md Musfiqur, and Murat Kocaoglu. "Modular Learning of Deep Causal Generative Models for High-dimensional Causal Inference." International Conference on Machine Learning. PMLR, 2024.
[7] Balazadeh Meresht, Vahid, Vasilis Syrgkanis, and Rahul G. Krishnan. "Partial identification of treatment effects with implicit generative models." Advances in Neural Information Processing Systems 35 (2022): 22816-22829.
[8] Javaloy, Adrián, Pablo Sánchez-Martín, and Isabel Valera. "Causal normalizing flows: from theory to practice." Advances in Neural Information Processing Systems 36 (2023): 58833-58864.\

#5. From my observation, the authors did not compare with any counterfactual approaches. However, there exist many counterfactual estimation approaches. The authors should do a literature search and compare with the most recent approaches. A few examples are [10] and [11].

[10] Chao, Patrick, Patrick Blöbaum, and Shiva Prasad Kasiviswanathan. "Interventional and counterfactual inference with diffusion models." arXiv preprint arXiv:2302.00860 4 (2023): 16.
[11] Melistas, Thomas, et al. "Benchmarking counterfactual image generation." Advances in Neural Information Processing Systems 37 (2024): 133207-133230.\

Questions for authors
  1. Line 189, why does the training example contain only prediction and counterfactual query and not an interventional query ?
  2. See weakness.
4 comments
AuthorsOpenReview ↗

Response to KCpu Part 1

Thank you for your detailed review and feedback. We appreciate brining up points for better evaluating our model.

Identifiability

Non-identifiability is a fundamental challenge in causal algorithms. However, our method can still make plausible predictions in certain non-identfiable cases by taking advantage of inductive biases. For the case of having two nodes X and Y, the general case is non-identifiable. If some basic assumptions are made about the underlying functions (e.g. non-Gaussian additive noise), then identification becomes plausible. To demonstrate, we evaluate our method on two-node linear SCMs with different noise types (Gaussian and Laplace) and confounding distributions. 10 random weights are sampled for each graph, we take the models predicted graph and average them.

n_fitNoise / confoundingTrue graphIdentifiableP(Y→X)P(X→Y)P(X↔Y)
128LaplaceX -> Yyes0.330.650.16
128LaplaceY -> Xyes0.600.380.15
128GaussianX -> Yno0.430.550.17
128GaussianY -> Xno0.430.550.17
128Gaussian confoundingX <-> Yno0.430.550.17
128Binary confoundingX <-> Yno0.430.560.42
1024LaplaceX -> Yyes0.050.940.07
1024LaplaceY -> Xyes0.930.060.06
1024GaussianX -> Yno0.490.510.08
1024GaussianY -> Xno0.490.510.08
1024Gaussian confoundingX <-> Yno0.490.510.08
1024Binary confoundingX <-> Yno0.510.520.54

Our method correctly identifies the direction of the edge in the identifiable cases, and is particularly confident with more data. In the non-identifiable cases, our method reports equal probabilities of the three possible graphs, which is the correct behavior under uncertainty and the bayesian framework. If there is non-Gaussian confounding, our model places much higher weight on a bidirected edge. As with any causal identification, fundamentally non-indentifable cases cannot be resolved but our model can still make progress with basic assumptions.

Predictions on non-identifiable graphs

Next, we evaluate the model predictions on non-identified graphs in [3] as suggested. Since these graphs are non-identifiable, there is no single correct predicted distribution. However, under the PFN formulation, we expect the model to make predictions to be the posterior, given our graph prior and input data. For each graph, we make a linear SCM with binary treatment and noise distributions. Then, given the parameters of the SCM, we use monte-carlo sampling with respect to our graph prior to compute plausible distributions that match the observed distribution, weighted to the likelihood of our prior. This allows us to compute the ideal PFN posterior distribution (Equation 1). We then sample 1024 observations from the model and compute the model predicted distribution and compare with the monte-carlo posterior distribution. The results are shown in the new Figures 7, 8 9 in the Appendix. For each graph, we plot out p(y | do(T)) for several compatible distributions, as well as the ideal posterior. In these cases, our model predictions are close to the ideal posterior. Hence, the predictions are consistent with the Bayeisan PFN framework and, assuming the prior is accurate, predictions remain useful.

Varying interventional and counterfactual queries.

We use a four-node SCM with observed treatment T, mediator M, and outcome Y, and a binary latent variable U. The graph has T -> M -> Y, T -> Y, and U -> T and U -> Y, so U confounds the treatment-outcome relationship. We vary the strength of the intervention by changing the weights of the edges. At strength 0, there is no causal effect and observational, interventional and counterfactual distributions are identical. As the strength increases, the distributions diverge and split apart. We evaluate our models predicted distributions compared to an oracle with the analytic posterior. This graph is identifiable. The results are shown in new Figure 10. Our model accurately predicts the shape of the distributions as the intervention strength increases, from the delta function at strength 0 to bimodal observational and interventional distributions.

AuthorsOpenReview ↗

Response to KCpu Part 2

Comparison with causal normalizing flows

Thank you for suggesting the other baselines to compare to. Unfortunately, they are not directly comparable to our method since they require the causal graph to be known beforehand, or work with image data rather than tabular data. In our response to reviewer 8uTr, we compare against Neural-ADMG, a comparable method that does both graph prediction and outcome prediction including counterfactuals that may be useful as well.

We evaluated causal normalizing flows [8] on the out-of-distribution Fourier-feature SEMs with 1024 observations. This method does not use the full SCM, but still requires a causal ordering. This method is also substantially slower than ours, requiring around 2 minutes per dataset compared with our 0.1 second per dataset, since a new model is trained rather than using the PFN framework.

The corresponding TabPFN-CFM results with and without a supplied graph are from Table 1 of the paper. We supply Causal Normalizing Flows the ground truth causal structure. Outcome MSE (lower is better):

ModelObservationalInterventionalCounterfactual
Causal normalizing flows [8]0.7391.0780.954
TabPFN-CFM (no graph)0.5340.7550.603
TabPFN-CFM (with graph)0.5340.7340.575

TabPFN-CFM outperforms causal normalizing flows on all three causal layers. We attribute this difference to the PFN setting, which helps mitigate overfitting and allows for better generalization. PFNs have previously been shown to outperform the traditional training of neural networks in many settings, in both causal and general tabular learning. For example, when tuning Causal normalizng flows, we found the full 20000 steps heavily overfitted and a shorter training run yielded better results, which we use here.

How does the model perform counterfactuals?

Our model is trained to entirely on the cross-entropy loss (Equation 2). We believe this is sufficient to learn counterfactuals, experimentally, we verify the model converges to the correct counterfactual distribution. In our setup, since the target is the counterfactual value, our model internally learns to predict the counterfactuals. Since predicting counterfactuals requires the abduction, action and prediction steps, the model must learn to perform these steps internally. There is no explicit abduction "module" or "architecture", rather the transformer is flexible enough to perform these internally. For example in Figure 3 and 4, the model correctly gets the counterfactual outcome, which necessarily requires the abduction step to estimate the latent noise value.

Thank you for your response. However, I still have some unresolved concerns.

Firstly, the authors should explicitly mention in the paper, their limitation for non-identifiable queries.

Next, I mentioned several approaches that can generate interventional and counterfactual samples [4, 6, 7, 8, 10, 11]. However, I do not see any comparison with these methods for any specific causal query (except comparison with causal normalizing flow). The experiments shown in Figures 7, 8, and 9 of the PDF could also be performed for these baselines.

Additionally, what does the true SCM in Figure 10 look like? The observational, interventional, and counterfactual distributions appear to be nearly identical, differing mainly by a location shift. I would request that the authors clarify this and the previous point.

AuthorsOpenReview ↗

Follow up experiments

We thank the reviewer for their follow-up comments and appreciate the opportunity to clarify our work further. We will mention the identifiability limitations in the paper more prominently.

Comparison with baseline approaches.

We have run new comparisons with a few of the suggested papers, the Neural Causal Model (NCM) [4], Modular-DCM [6] and Diffusion Causal Models (DCM) [10]. Firstly, we highlight the differences with our setup. None of these methods predict the distribution of outcomes, only samples. Secondly, all of these methods require the full ground truth causal graph to be known. The NCM was further modified compared to the original paper to handle continuous data using Gaussian latents rather than Bernoulli latents. DCM assumes no hidden confounding, which is not the case for our setup so accuracy will be lower. Modular-DCM does not produce counterfactuals, we modified the method by using test-time training to perform abduction by training the noise variables for 80 steps. Results are shown below:

ModelObservationalInterventionalCounterfactual
Causal normalizing flows [8]0.7391.0780.954
Neual Causal Model [4]0.5841.0000.881
Diffusion Causal Models [10]0.6971.0520.981
Modular-DCM[6]0.9551.3031.279
N-ADMG (Response 8uTr)0.7221.1381.203
TabPFN-CFM (no graph)0.5340.7550.603
TabPFN-CFM (with graph)0.5340.7340.575

The Neural Causal Model had the best observational performance of the baselines, just behind TabPFN-CFM. All baselines struggled in the interventional and counterfactual case. These baselines are also substantially slower to run since a new model is trained for every case, around 30 seconds to 2 minutes per dataset.

For the other suggested baselines, CausalGAN [5] and Benchamrking Counterfactual Image Generation [11] are aimed at the image modality, so we are unable to fairly compare on our tabular setting without substantially redesigning the methods.

Evaluating distribution prediction of CNF

We repeat the experiment in new figures 7, 8, 9 with Causal Normalizing Flows (CNF). The results are given in new Figure 11. We see that CNF captures a single plausible distribution, and does not fully capture the posterior of plausible distributions and underestimates the variance. This is particularly visible in the tails of the Mediated and Mediated + Direct graphs. TabPFN-CFM correctly captures the full posterior.

Impact of varing causal strength

In Figure 10, we evaluate on a Linear SCM with binary treatment T. Since the SCM is linear, switching treatment from 0 to 1 is limited to affine transformations. We run the experiment on a new nonlinear SCM in Figure 12 with a nonlinear response function on Y. The full setup is described in the Appendix. This demonstrates heteroskedasticity with the different treatments which our model successfully captures.

Reviewer_ypUmOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes TabPFN-CFM, a causal foundation model that jointly learns causal structures and predicts observational, interventional, and counterfactual outcomes. The model is pretrained on synthetic structural causal models (SCMs), with an auxiliary structure prediction objective that outputs directed adjacency, ancestry, and latent confounding relations represented through an ADMG. It also supports optional conditioning on a known causal graph.

The authors evaluate the model on synthetic SCMs, including an ood setting with unseen functional mechanisms, and on two real tabular datasets with model-generated causal outcomes. The reported results show improvements over several existing causal discovery and effect estimation methods. An ablation study further suggests that structural supervision can improve interventional and counterfactual prediction.

Critical strengths
  1. Jointly studying causal structure learning and causal outcome estimation is a meaningful direction. I particularly appreciate the attempt to investigate whether structural supervision improves interventional and counterfactual predictions, rather than treating the tasks as entirely independent.

  2. The paper goes beyond standard DAG learning by considering latent confounding and counterfactual queries. The evaluation also includes a held-out functional mechanism family, which is a useful step toward studying generalization beyond the training prior.

Critical weaknesses

1. The foundations and empirical validation of the claimed causal structure learning capability are insufficient.

The paper aims to recover causal structures, including latent confounding, from observational data using an amortized model. This is already a highly challenging research problem on its own. In particular, learning ADMGs involves different graphical representations, identifiability assumptions, and equivalence classes. Even with unlimited observational data, the underlying causal structure is generally not uniquely identifiable without additional assumptions.

The paper supervises predictions of adjacency, ancestry, and confounding matrices using labels from simulated SCMs. However, it is unclear what aspects of these structures are identifiable under the assumed data-generating processes, how the learned predictions respect graphical consistency constraints, or whether their uncertainties meaningfully reflect observational non-identifiability. Predicting each structural relation with an elementwise loss is not necessarily equivalent to learning a coherent posterior over causal graphs.

These questions matter because the paper claims substantially more than predictive accuracy on simulated graph labels. It presents the model as a general-purpose approach to causal structure learning, including latent confounding. Establishing such a capability would require a much more systematic characterization of the assumptions under which it succeeds, the structures it can reliably recover, and its behavior when those structures are not identifiable.

The empirical evidence is also insufficient for this claim. Most evaluations use synthetic SCMs drawn from the authors' training prior. Although the paper includes an OOD experiment with random Fourier mechanisms, this remains a limited test of generalization across the much broader space of causal graphs, functional mechanisms, noise distributions, and latent confounding structures. The real-data evaluation is restricted to two datasets and does not provide ground-truth confounding information.

Related work that would help clarify the problem formulation and evaluation includes Learning Identifiable Structures Helps Avoid Bias in DNN-based Supervised Causal Learning (Zhang et al., 2025), Causal Discovery with Unobserved Confounding and Non-Gaussian Data (Wang and Drton, 2023), and Causal Identification under Markov Equivalence: Completeness Results (Jaber et al., 2019).

2. The outcome prediction claims are not sufficiently supported, particularly for counterfactual inference and the proposed unification of the three tasks.

The paper brings causal structure learning, interventional prediction, and counterfactual prediction into one pretrained network. I find the overall direction interesting, but each of these tasks has substantial unresolved challenges, and the current evaluation does not convincingly establish that the proposed model can address them in a generalizable manner.

One fundamental concern is the counterfactual prediction setting. Section 5.3 and Appendix D.2 appear to supply the model with counterfactual covariate values generated by the underlying SCM, including variables recomputed after the treatment intervention. Such values would generally not be available when answering a real counterfactual query. Conditioning on them is different from inferring a counterfactual outcome using only factual observations. This distinction needs to be resolved before the reported counterfactual performance can be interpreted as evidence of the claimed capability.

The real-data experiments do not fully address this issue. Although the observational datasets are real, the interventional and counterfactual evaluation targets are generated using DoWhy models. Agreement with these targets does not independently validate the model's predictions against real causal outcomes.

I also find the methodological contribution of jointly learning structure and outcomes insufficiently established. The model shares representations across the tasks, but its predicted ADMG is not explicitly used as an intermediate input to outcome prediction when no graph is supplied. The structure-loss ablation shows modest improvements in interventional and counterfactual prediction, but this is also consistent with the benefits of ordinary auxiliary-task regularization. It does not establish that the model has learned a structure-based causal inference procedure, or clarify when structure learning is necessary for estimating the target causal effect.

More broadly, the evaluation of supervised causal learning needs to distinguish genuine causal generalization from statistical regularities introduced by synthetic data generators. The concerns raised in Beware of the Simulated DAG! Causal Discovery Benchmarks May Be Easy to Game (Reisach et al., 2021) are particularly relevant here.

Overall, the paper demonstrates that several causal prediction objectives can be learned jointly under a synthetic training distribution. However, I do not find sufficient evidence that this constitutes a reliable or general-purpose foundation model for causal discovery, intervention, and counterfactual inference.

Questions for authors
  1. Causal structure learning. What is the precise identifiability setting for the predicted ADMG? Which structural properties are expected to be identifiable from observational data, and how does the model represent uncertainty when multiple causal structures are observationally equivalent? How is consistency among the predicted adjacency, ancestry, and confounding relations ensured?

  2. Counterfactual inference. Does the model receive counterfactual covariates generated under the alternative treatment assignment, including treatment descendants? If so, can the authors evaluate counterfactual prediction using only information available in the factual world? This distinction seems essential to interpreting the results.

  3. Generality and the benefit of joint learning. What evidence establishes that the model's causal capabilities generalize beyond the synthetic training prior? In particular, can the authors provide broader evaluations using independent SCM generators and clarify whether the improvement from structural supervision reflects actual use of causal structure rather than a generic multi-task learning benefit?

2 comments
AuthorsOpenReview ↗

Response to ypUm Part 1

Thank you for the detailed review and for pointing us to these references.

Counterfactual inputs and baselines

The model never receives counterfactual or post-intervention covariates in either training or evaluation. Interventional queries pass (x∗,do(T=1−t∗))(\mathbf{x}^{\ast}, \mathrm{do}(T = 1 - t^{\ast})) and counterfactual queries additionally pass the observed outcome y∗y^{\ast}, where x∗\mathbf{x}^{\ast} and y∗y^{\ast} are the factual values observed under t∗t^{\ast} (including any observed descendants of TT). Writing xt∗\mathbf{x}^{\ast}_t in Sections 4 and 5.3 and Appendix D.2 was a notation mistake on our part: when generating targets, descendants of TT are recomputed inside the SCM to obtain yt∗y^{\ast}_t, and those recomputed covariates are discarded before constructing the query. All counterfactual results in the paper therefore already evaluate factual-only inference, and we will fix the notation in those sections. Our code in the supplement correctly implements this; we always use counterfactual=True in the config file. This can also be seen in Table 3 on Amazon Sales, where the mediators Revenue and Operational Cost determine Profit: an S-learner gets 0.002 observational MSE but 7.269 interventional MSE because the query inputs hold the factual mediators rather than post-intervention ones.

To compare directly against baselines that support counterfactual queries, we evaluated N-ADMG (Ashman et al., 2023) and Causal Normalizing Flows (Javaloy et al., 2023, which is given the true causal ordering, though it assumes no latent confounding) on the OOD Fourier dataset (n=1024n = 1024):

ModelObservationalInterventionalCounterfactual
N-ADMG (Ashman et al., 2023)0.7221.1381.203
Causal normalizing flows (Javaloy et al., 2023)0.7391.0780.954
TabPFN-CFM (no graph)0.5340.7550.603
TabPFN-CFM (with graph)0.5340.7340.575

TabPFN-CFM improves on both across all three tasks and runs in under 0.1s per dataset, versus 2+ minutes for the baselines, which fit a new model on each dataset.

Identifiability, uncertainty, and graph consistency

Observational data alone identify causal structure only up to an equivalence class (PAG), while nonlinearities and non-Gaussian noise or confounding in our prior p(ψ)p(\psi) make additional edges identifiable (Shimizu et al., 2006; Wang and Drton, 2023). Trained with elementwise BCE, our structure head predicts marginal posterior edge probabilities P(Mij=1∣Dobs)P(M_{ij}=1 \mid \mathcal{D}^{\mathrm{obs}}) under p(ψ)p(\psi). To test how these predictions behave on identifiable versus non-identifiable structures, we evaluated the model on standardized two-node linear SCMs (n=1024n = 1024, averaged over 10 datasets per setting):

Noise / confoundingTrue graphP(X→Y)P(Y→X)P(X↔Y)
Laplace (identifiable)X -> Y0.940.050.07
Laplace (identifiable)Y -> X0.060.930.06
Gaussian (identical data)X -> Y / Y -> X / X <-> Y0.510.490.08
Binary latent U + Gaussian noiseX <-> Y0.520.510.54

In the identifiable Laplace cases, edge probabilities sharpen as sample size grows (from 0.65 at n=128n=128 to 0.94 at n=1024n=1024). In the non-identifiable Gaussian cases, the model predicts roughly 50/50 for the two directions, while adding a binary latent confounder raises P(X↔Y)P(X\leftrightarrow Y) from 0.08 to 0.54. Similarly, when the interventional distribution P(y∣do(t))P(y \mid \mathrm{do}(t)) is non-identifiable even given the true graph G\mathcal{G}, the model's predicted densities track the average over compatible SCMs (Appendix H, Figures 7–9 and our response to Reviewer KCpu).

Our model predicts edgewise adjacency, ancestral and bidirected edges, the outputs are marginal probabilities rather than a full joint posterior (Zhang et al., 2025). Because outcome prediction never conditions on a sampled or thresholded predicted graph, this does not introduce downstream graph-discretization error. Even without hard constraints, on OOD graph recovery (n=1024n=1024) our predictions reach 0.880 directed and 0.804 confounding AUROC, compared to 0.625 and 0.392 for N-ADMG (which explicitly enforces acyclicity and bow-free constraints) and 0.535 and 0.560 for FCI. Graph identification is generally a hard task on larger graphs, and our method gives users more accurate predictions about the general structure of the system through ancestral and confounding edges.

AuthorsOpenReview ↗

Response to ypUm part 2

Generalization and joint learning

Our data generation procedure follows an enhanced version of the priors used in other causal PFN and graph discovery methods (Do-PFN, AVICI, Arrow, CausalPFN etc). In particular, we standardize each feature on the fit set and add empirical CDF categories which removes the primary marginal-variance varsortability (Reisach et al., 2021). Random column embeddings also remove column-ordering shortcuts.

On our OOD benchmark, we replace our training prior with unseen Random Fourier functions and raise the max latent probability from 0.5 to 0.75 which causeses only a small drop in accuracy (Tables 1 and 2). In Section 6.3, our datasets are semi-synthetic based on real data, taken from the Do-PFN paper (Robertson et al., 2025). In both the Amazon-sales dataset and Law-race dataset, the underlying causal graph and mechanisms are understood, so we expect the dataset to have realistic counterfactuals. While we would like to evaluate on fully real datasets, real counterfactuals and true graphs are generally not available, and why synthetic and semi-synthetic datasets are widely used. For example, the Law School Admissions dataset is widely used for causal and counterfactual benchmarking (Kusner et al., 2017, Zhou et al 2024, Robertson et al 2025, Fernandes et al 2025, Almodovar et al 2025, Guldogan et al 2026).

Causal structure

Causal structure is needed for outcome estimation whenever covariates contain mediators, colliders, or hidden confounders. When no graph is supplied, we do not feed a thresholded predicted graph back into the model for two reasons. Firstly, this would require sampling a single graph which would discard uncertainty about the graph structure present with non-identifiable graphs or finite sample sizes, so predictions would not be a true posterior (Equation 1). Secondly, our graph prediction is a gated inner product between the model's hidden states, training the model to predict graph structure forces the model to encode this information inside its latent embeddings which the model can use to improve predictions. We do not see explicitly feeding in the final prediction back into the model as neccesary. Furthermore, with graph inputs, we see jointly training with the graph structure, adding the structure loss barely changes error on observational predictions, (-0.4%), but improves performance much more on interventional (-2.6%) and counterfactual (-4.9%) tasks (λ>0\lambda > 0, Table 4). The graph structure task benefits causal tasks more than a generic gain from multi-task learning. A similar trend is seen at inference, supplying the true graph G\mathcal{G} reduces interventional and counterfactual MSE by 3–12% while leaving observational MSE unchanged at 0.0% (Proposition 2, Table 1).

New References

Kusner et al. Counterfactual Fairness. NeurIPS, 2017. Paper

Zhou et al. Counterfactual Fairness by Combining Factual and Counterfactual Predictions. NeurIPS, 2024. Paper

Robertson et al. FairPFN: A Tabular Foundation Model for Causal Fairness. ICML, 2025. Paper

Fernandes Machado et al. Sequential Conditional Transport on Probabilistic Graphs for Interpretable Counterfactual Fairness. AAAI, 2025. Paper

Almodóvar et al. DeCaFlow: A Deconfounding Causal Generative Model. NeurIPS, 2025. Paper

Guldogan et al. Counterfactually Fair Conformal Prediction. AISTATS, 2026. Paper

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.