acceptodds
Under review as a conference paper at ICLR 2027

What You Observe Determines How You Identify Causal Effects: Evaluating Causal Models across Observational Views

Abstract

Causal foundation models (CFMs) pre-trained on data generated from various structural causal models (SCMs) have been proposed for estimating causal effects from observational data. However, differences in pre-training environments and evaluation protocols make it difficult to assess how their performance depends on the information available for causal identification. To enable controlled comparisons, we introduce *CausalIDView*, a multi-view benchmark that holds fixed SCM realization and target estimand while varying only the observational view available to the estimator. Each observational view corresponds to a distinct identification regime under the benchmark's maintained causal assumptions. Across these matched views, no CFM consistently performs best and model rankings vary substantially. Under controlled structural changes, CFMs exhibit model-specific failures to maintain stable estimates when true effects are unchanged and to track genuine effect changes. We also examine whether combining explicit identification with strong predictive estimation is effective. A modular approach that pairs a predictive tabular foundation model with regime-specific identification procedures is competitive with CFMs and outperforms several of them. These findings motivate cross-regime comparisons to assess the empirical value of CFMs.

Primary area:
datasets and benchmarks
Keywords:
Causal Effect Estimation, Causal Benchmarking, Causal Identification, Causal Foundation Models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_DLE8OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes CausalIDView, a causal effect estimation from observational data benchmark to evaluate causal estimators on different causal identification regimes (back-door, front-door, instrumental variable, proximal). It is motivated by recent developments on causal foundation models (CFMs) and their differences in pre-training that depend on different identification conditions. CausalIDView’s different identification regimes address this gap, shedding light on shortcomings of current CFMs and on their differences as compared to standard estimation approaches.

Critical strengths

Causal effect estimation from different identification conditions is an important problem. This is especially important in the context of CFMs as their pre-training data may use different identification conditions, which can ultimately bias their predictions at deployment. Thus, comparing CFMs fairly across different identification regimes is a good step towards a better understanding of them and their limitations.

Strong experimental design. On top of multi-regime data, the paper also covers additional semi-synthetic and RCT datasets. It also covers causal estimators very comprehensively, including established and state-of-the-art estimation methods, as well as the modular approaches that combine TabPFN with identification specific techniques.

Critical weaknesses

Mostly presentation issues (all addressable). The paper relies too much on the appendix in some places in the main content of the paper. My understanding is that information in the appendix is optional for the reader and should be treated as such. Some examples. RQ3 in section 6 references Figure 6 that is available only in the appendix (it is the main evidence source for this paragraph). Tables 1 and 2 report results for methods never mentioned in the main text (e.g. Causal Tree, TARNet, GRF, CFRNet). I know these are described in the appendix, but some description should also appear in the main text.

Questions for authors
  1. I would suggest to either move RQ3 from section 6 to the appendix, or move Figure 6 to the main body of the paper.
  2. I would suggest to add missing descirptions of the methods included in Tables 1 and 2 in the main body of the paper.
3 comments
AuthorsOpenReview ↗

Clarifications to Baseline Descriptions and RQ3 Findings

We sincerely thank the reviewer for the careful assessment of our manuscript and the constructive suggestions. We agree that the original manuscript did not sufficiently present the supporting evidence for RQ3 and the descriptions of certain baselines in the main text.

To address these concerns, we have revised Section 5.2 to include brief descriptions of the previously omitted baselines and refined the RQ3 discussion in Section 6 to more clearly present its supporting evidence and findings in the main text. These revisions are highlighted in blue in the revised PDF for ease of reference. We appreciate the reviewer's valuable feedback, which has helped us improve the clarity and readability of our manuscript.

Reviewer_DLE8OpenReview ↗

Thanks for the revision

Thank you for addressing my concerns. I have also read the issues raised by the other reviewers, all the authors' responses and the updated manuscript. In my view, the new additions and results, as summarised in the global response, make this paper much stronger. I am overall impressed by the revision and will update my score to 4 to reflect the now much stronger state of the paper.

Thank you very much for your thoughtful reassessment and for raising your score. We are delighted that you find the revised manuscript much stronger. Your insightful comments played an important role in shaping these improvements, and we sincerely appreciate your time and engagement throughout the review process.

Reviewer_xS2EOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces a benchmark for evaluating causal foundation models across different identification regimes. It generates multiple observational views from the same data generating process (SCMs) while keeping the target CATE fixed. The authors compare existing CFMs with traditional causal estimators and TabPFN-based estimators. They find that no CFM consistently performs best, while TabPFN combined with explicit identification methods often performs better.

Critical strengths
  1. The benchmark evaluates different identification methods on the same underlying SCM and changes only which variables are observed. This provides a well controlled environment for evaluating different Id methods.
  2. The paper evaluates several causal models across synthetic, semi-synthetic, and real-world datasets.
  3. The finding is very interesting. The results show that strong predictive models combined with standard causal identification methods can outperform specialized CFMs.
Critical weaknesses
  1. The benchmark assumes complete mediation and treatment effects that do not vary with unobserved confounders. This limits how well the results generalize.

  2. Some CFMs were not designed for all identification regimes, for example, CausalPFN was trained for conditional ignorability (backdoor criteria). Their poor performance on certain ID methods is expected.

  3. The main multi-view experiments are synthetic. The real world experiments only evaluate zero treatment effects and do not test the full multi-view framework.

Questions for authors

See the weakneses section.

4 comments
AuthorsOpenReview ↗

[W1] Extensions Beyond the Original SCM Restrictions

Thank you for highlighting this limitation. The original SCM enables controlled comparisons across identification regimes by maintaining a common target CATE, but its structural restrictions limit the range of causal mechanisms covered. We therefore conducted additional experiments that separately relax treatment-gain homogeneity conditional on observed covariates and complete mediation (Appendix F.3, Figures 8 and 10).

Treatment gains that depend on the latent confounder. We modify the mediator–outcome coefficient so that the treatment effect becomes Y(1)−Y(0)=τ0(X)+αUY(1)-Y(0)=\tau_0(X)+\alpha U, allowing individuals with the same XX to have different treatment effects. Because E[U∣X]=0E[U\mid X]=0, the population XX-CATE remains τ0(X)\tau_0(X). For BD, predictions conditional on UU are marginalized over its empirical distribution estimated from the context data. For IV, we modify treatment assignment to maintain a constant instrument-induced treatment probability difference of 0.30 and preserve monotonicity, so that the conditional Wald functional continues to identify the common XX-CATE. Under this extension, the TabPFN-based DR-Learner and FD plug-in achieve the lowest mean sPEHE² in BD and FD, respectively, while CausalFM achieves the lowest mean error in IV.

A direct treatment–outcome pathway. We also introduce a direct effect while reducing the mediated component, preserving the total treatment effect. With ρ=0.5\rho=0.5, the direct and mediated pathways each account for half of the total effect. BD, IV, and PX retain their identification conditions, and their lowest-error configurations remain the same as in Figure 2. In FD, however, the direct pathway violates complete mediation: the original front-door functional recovers the mediated component rather than the total CATE. We therefore interpret these FD results as diagnostics under an identification-assumption violation, not as evidence of valid total-effect identification.

Table W1: Lowest-error configurations under relaxed SCM restrictions. Each cell reports the estimator with the lowest observed mean sPEHE² (↓) and its value across 40 SCM worlds. DR = TabPFN + DR-Learner; FD = TabPFN + FD Plug-in; P = TabPFN + P-Learner. Under U-dependent gains, proxy-augmented CFM predictions in PX are treated separately as diagnostics because they may correspond to different conditional effects. † The FD result under partial mediation is an assumption-violation diagnostic. Full model-wise results and error decompositions are provided in Appendix F.3.

SCM extensionBDFDIVPX
U-dependent gains (α = 0.5)DR 0.1484FD 0.1665CausalFM 0.2778P 0.1995
Partial mediation (ρ = 0.5)DR 0.1336FD 0.2779†CausalFM 0.2599P 0.1699

These experiments show that the leading configurations largely persist under the evaluated relaxations where identification remains valid for the common target. They also illustrate why changes in causal structure must be distinguished from violations of an identifying assumption. We do not claim generalization to arbitrary SCMs. Rather, these results extend the empirical evaluation beyond two restrictive features of the original construction.

AuthorsOpenReview ↗

[W2] Interpreting Off-Regime Performance and Native-Regime Comparisons

We agree that performance degradation can be expected when a CFM is applied outside the identification regimes considered during pre-training. In particular, poor performance from applying the released CausalPFN checkpoint to an IV view does not, by itself, establish a general inability to perform causal identification. Our objective is to characterize how the evaluated model configurations respond to changes in identifying information within the same causal world, rather than to assume that every CFM natively supports every regime.

To address this concern, we distinguish native-regime comparisons from cross-regime transfer diagnostics. For CausalFM, we use the publicly released BD and IV checkpoints. For CausalPFN, we use the released BD checkpoint and an additionally trained IV-specific checkpoint based on a binary-IV SCM prior. All configurations are evaluated on the same 40 SCM worlds as Figure 2, with the same context and query units and target CATEs.

Table W2a: Native-regime comparisons. Each checkpoint is evaluated under its corresponding pretraining identification regime.

ModelEval. viewMean sPEHE ↓Pearson r ↑Pred. CATE SD
CausalPFN-BDBD0.450980.526220.40965
CausalFM-BDBD0.481150.283930.07098
CausalPFN-IVIV0.678670.410110.06900
CausalFM-IVIV0.506350.183110.04628

Table W2b: Cross-regime transfer diagnostics. Each checkpoint is evaluated outside its corresponding pretraining identification regime.

ModelEval. viewMean sPEHE ↓Pearson r ↑Pred. CATE SD
CausalPFN-BDIV0.712760.475280.41389
CausalFM-BDIV0.548920.224900.09882
CausalPFN-IVBD0.672990.409710.06823
CausalFM-IVBD0.527740.196090.03840

The native comparisons show a change in relative rankings by mean sPEHE. CausalPFN-BD has lower error than CausalFM-BD in BD (0.451 vs. 0.481), whereas CausalFM-IV has lower error than CausalPFN-IV in IV (0.506 vs. 0.679). The IV-trained CausalPFN also has lower sPEHE on the IV view than the released BD checkpoint (0.679 vs. 0.713). However, because these checkpoints have different training histories, this comparison does not isolate the contribution of the pre-training regime.

Native-regime evaluation also reveals limitations in recovering CATE variation. The average standard deviation of the true CATE is 0.489, compared with predicted standard deviations of 0.069 for CausalPFN-IV and 0.046 for CausalFM-IV. This observation concerns the evaluated checkpoints and does not establish a failure of causal identification: training adequacy, pre-training distribution mismatch, and finite-sample estimation difficulty may also contribute.

Thus, the observed ranking change is not confined to off-regime evaluations. At the same time, we explicitly restrict the interpretation of transfer results and avoid treating their performance gaps as evidence of an inherent deficiency in CFMs. The experimental settings and results are provided in Appendix F.8.

AuthorsOpenReview ↗

[W3] Real-World Non-Null Effects Across BD and FD Strategies

We agree that the original real-world evaluation addresses null effects and does not test the full matched multi-view framework. To provide complementary evidence on a real non-null causal problem, we conducted an additional comparison using the National Job Training Partnership Act (JTPA) Study [1]. Glynn and Kashin (2018) [2] applied back-door (BD) and front-door (FD) adjustments to this problem and compared their ATT estimates with experimental references.

Data and target estimand. Following this setup, treatment TT denotes program access following application, mediator MM denotes actual training-service utilization, and outcome YY is earnings over the subsequent 18 months. The target is the ATT of program access, rather than the effect of actual training receipt. The Baseline covariates are age, race, and study site. We additionally consider a specification including assignment-month earnings (earn_0) as a sensitivity analysis.

BD estimates counterfactual earnings using external eligible nonparticipants (ENPs), whereas FD uses noncompliers within the program-access group. We reconstructed an analysis cohort of 5,620 individuals using the original data and the authors' replication materials. The experimental references computed for this cohort are USD 708.42 for men and USD 702.17 for women. The male reference differs from the USD 699.60 reported in [2]. We therefore distinguish our reconstructed-cohort reference from the published value.

In addition to averaging estimated CATEs over the treated evaluation units, we evaluate procedures that directly target the ATT. For BD, we compare DR-Learner, Direct ATT, and AIPW-ATT; for FD, we compare FD Plug-in and Augmented FD-ATT. Comparisons use the same target evaluation units and split protocol, while the observations used to construct counterfactual estimates differ between BD and FD.

Table W3: JTPA ATT estimates relative to reconstructed experimental references. Entries are estimated ATT minus the corresponding experimental reference, in USD. Smaller absolute deviations indicate closer agreement, not necessarily smaller causal estimation error. The experimental references are USD 708.42 for men and USD 702.17 for women. Because earn_0 is not guaranteed to precede treatment, this specification is included only as a sensitivity analysis.

StrategyEstimatorMen: BaseMen: +earn_0Women: BaseWomen: +earn_0
BDDR-Learner (CATE avg.)−7,217+2,566−65+3,347
BDDirect ATT−7,351+3,104+27+3,296
BDAIPW-ATT−6,822+3,105−43+3,340
FDFD Plug-in+726+479+45−91
FDAugmented FD-ATT+670+511+109−67

The results reveal substantial covariate sensitivity in BD. For men, the Baseline BD estimates are approximately USD 6,800–7,400 below the reference. Adding earn_0 increases the estimated ATTs by approximately USD 9,800–10,500, changing them from negative to positive. This pattern also appears in the ATT-specific estimators and therefore cannot be explained solely by averaging CATE predictions. In contrast, the corresponding FD estimates change by only USD 159–247 for men, and their deviations for women remain approximately within USD 110 of the reference. These are descriptive comparisons of point estimates rather than claims of statistically significant differences in causal accuracy.

Scope of the real-world evidence. This experiment compares two identification strategies for a real non-null ATT problem, but it is not equivalent to the synthetic matched-view benchmark. BD relies on conditional exchangeability, while FD relies on an exclusion restriction and additional assumptions concerning service utilization. Agreement with an experimental reference does not establish that these assumptions hold. Moreover, BD uses external ENPs and FD uses noncompliers within the program-access group, so the comparison does not vary only auxiliary variables in an otherwise identical observational sample.

Although this experiment does not evaluate pointwise CATE accuracy or CFM rankings, it provides a complementary comparison of BD and FD strategies for the same real-world ATT problem. CausalIDView supplies the controlled counterpart: a fixed SCM realization, shared query units and target CATE, and explicitly maintained regime-specific identification conditions.

[1] Upjohn Institute. National Job Training Partnership Act (JTPA) Study. W.E. Upjohn Institute for Employment Research.

[2] Glynn, A. N., and Kashin, K. (2018). Front-Door Versus Back-Door Adjustment With Unmeasured Confounding: Bias Formulas for Front-Door and Hybrid Adjustments With Application to a Job Training Program. Journal of the American Statistical Association, 113(523), 1040–1049.

Thank you again for your valuable comments on our paper. We wanted to kindly follow up to check whether our responses have adequately addressed your concerns. If there are any remaining questions or points that need further clarification, we would be happy to address them. We truly appreciate your time and effort.

Reviewer_fhM7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CausalIDView to evaluate causal estimators across observational views of the same underlying system. The benchmark fixes the realized data and target treatment effects while varying the available identifying variables. It covers back-door, front-door, instrumental-variable, and proximal identification, alongside hidden-confounding evaluations. Additional tests examine responses to structural changes, semi-synthetic data, and real-world null effects. Results show changing model rankings and demonstrate that TabPFN combined with explicit identification procedures can compete with causal foundation models.

Critical strengths
  1. The paired benchmark design is valuable. Holding the causal world and target fixed makes cross-regime comparisons more informative than evaluations on separately generated datasets. The diagnostics reveal more than average estimation error. Stay/Move tests distinguish spurious responses from failures to track genuine effect changes. The error decomposition also separates average-effect bias from errors in treatment heterogeneity.
Critical weaknesses
  1. The shared-target construction restricts the benchmark’s scope. Treatment gains cannot vary with the latent confounder at fixed covariates. The main generator also uses a binary confounder and complete mediation. These choices enable clean comparisons but leave broader applicability uncertain.
  2. The real-world evaluation only tests null effects. Predicting zero is optimal by construction. These tests assess suppression of spurious effects, but provide little evidence about recovering nonzero treatment heterogeneity.
Questions for authors

Could native-regime comparisons be summarized separately from transfer evaluations? This would clarify which conclusions hold when the compared models can use the relevant identifying variables. Are the main rankings consistent across the five treatment-effect families? A breakdown of the existing results would help show whether the aggregate findings depend on particular mechanisms.

7 comments
AuthorsOpenReview ↗

[W1] Robustness to Relaxed SCM Assumptions

Thank you for this valuable observation. We agree that the structural restrictions introduced to enable controlled comparisons across identification regimes with a shared target CATE limit the scope of our benchmark. To examine these limitations, we conducted additional experiments that relax three key assumptions of the original full SCM (Appendix F.3, Figures 8–10).

First, we allow treatment-effect heterogeneity with respect to the latent confounder. While treatment effects in the original SCM depend only on XX, we extend the outcome mechanism to Y(1)−Y(0)=τ0(X)+αUY(1)-Y(0)=\tau_0(X)+\alpha U, allowing treatment effects to vary with UU even at fixed XX. We preserve the same XX-CATE and modify the treatment assignment mechanism so that the IV identifying functional continues to recover the shared target. Under this extension, the TabPFN-based DR-Learner (0.1484) and FD plug-in (0.1665) achieve the lowest mean sPEHE² in BD and FD, respectively, while CausalFM (0.2778) remains the best-performing model in IV.

Second, we extend the binary latent confounder to a continuous variable. We replace the binary UU with U∼N(0,1)U\sim\mathcal{N}(0,1) and adapt the proxy generation and estimation procedures accordingly. In this setting, TabPFN-based modular estimators achieve the lowest mean errors in BD and FD, while CausalFM remains the best-performing model in IV. The TabPFN-based P-Learner also achieves the lowest mean error in PX. These results indicate that the main performance patterns are not limited to a binary latent confounder.

Third, we relax the complete mediation assumption by introducing a direct treatment–outcome pathway. We allow part of the treatment effect to act directly on the outcome rather than entirely through the mediator. Under this extension, the same models as in Figure 2 achieve the lowest mean errors in BD, IV, and PX, where the identification assumptions remain satisfied. In contrast, complete mediation is violated in FD. We therefore do not interpret the corresponding results as evidence of valid front-door identification, but instead report them as a separate diagnostic of model behavior under an identification-assumption violation.

Table W1: Best-performing configurations under relaxed SCM assumptions. Each cell reports the best-performing estimator and its mean sPEHE² (↓) across 40 SCM worlds. DR = TabPFN + DR-Learner; FD = TabPFN + FD Plug-in; P = TabPFN + P-Learner. In the UU-dependent-gain PX setting, proxy-augmented CFM predictions are treated separately as diagnostics because they may target different conditional effects. The continuous-UU PX evaluation uses a continuous-bridge P-Learner. † The FD result under partial mediation is an assumption-violation diagnostic, not evidence of valid front-door identification.

SCM ExtensionBDFDIVPX
UU-dependent gains (α=0.5\alpha=0.5)DR 0.1484FD 0.1665CausalFM 0.2778P 0.1995
Continuous UUDR 0.1513FD 0.1543CausalFM 0.2546P 0.2244
Partial mediation (ρ=0.5\rho=0.5)DR 0.1336FD 0.2779†CausalFM 0.2599P 0.1699

Full model-wise results and error decompositions are provided in Appendix F.3 (Figures 8–10).

Across these additional experiments, the main regime-specific performance patterns observed in Figure 2 remain largely consistent under the evaluated structural extensions. These findings suggest that our main empirical observations are not solely driven by the original assumptions of treatment-gain homogeneity conditional on XX, a binary confounder, or complete mediation. Nevertheless, these extensions do not guarantee generalization to arbitrary SCMs, and extending the benchmark to broader causal structures and identification assumptions remains an important direction for future work.

AuthorsOpenReview ↗

[W2] Real-World Non-Null Effects and Subgroup Heterogeneity

We agree that the existing real-world null-effect evaluation assesses the ability of causal estimators to suppress spurious treatment effects but provides limited evidence of their ability to recover non-null treatment effects and treatment-effect heterogeneity. To address this limitation, we conducted an additional experiment using data from the Infant Health and Development Program (IHDP) randomized controlled trial, which reported non-null treatment effects (Appendix F.2, Figure 7).

Unlike our previous semi-synthetic IHDP evaluation, this experiment uses actual randomized treatment assignments and IQ outcomes measured at 36 months of corrected age. The original IHDP study [1] reported a treatment effect of 6.6 IQ points for infants with birthweights of at most 2,000 g (Low birthweight) and 13.2 IQ points for those weighing 2,001–2,500 g (High birthweight). We use these experimentally reported subgroup effects as empirical references and compare them with the subgroup-averaged CATE predictions of each model.

The results show that all evaluated models predict positive treatment effects in both subgroups. For the Low birthweight group, CausalFM achieves the smallest deviation from the reference effect (+0.88 IQ points), while TARNet achieves the smallest deviation in the High birthweight group (−1.90 IQ points). However, all models underestimate the reported 6.6-point difference in treatment effects between the two subgroups. Thus, the models reproduce the positive direction of the effects in both groups but do not fully reproduce the experimentally reported between-subgroup contrast.

Table W2: Comparison with experimentally reported IHDP subgroup effects (IQ points). The published reference effects are 6.60 points for the Low-birthweight group and 13.20 points for the High-birthweight group, giving a between-group contrast of 6.60 points. The table reports deviations between subgroup-averaged CATE predictions and published experimental reference effects, along with the predicted between-subgroup effect difference (High − Low). Low and High columns report predicted minus reference subgroup effects, not the predicted effects themselves. Positive and negative values indicate overestimation and underestimation relative to the published reference, respectively. The Pred. gap column is calculated from the rounded deviations in Figure 7. Bold values indicate the closest model to the published reference for each measure.

ModelLow (≤2,000 g)High (2,001–2,500 g)Pred. gap
CausalPFN+1.83−3.041.73
Do-PFN−3.20−9.510.29
CausalFM+0.88−5.420.30
Causal Tree+2.25−3.620.73
TARNet+0.95−1.903.75
GRF / Causal Forest+1.58−2.572.45
CFRNet+1.85−2.931.82
TabPFN-v3.5 + X-Learner+1.82−2.592.19
TabPFN-v3.5 + S-Learner+1.45−4.390.76
TabPFN-v3.5 + DR-Learner+2.03−3.121.45

This experiment extends our real-world evaluation beyond null effects by assessing model predictions against non-null treatment effects and subgroup-level heterogeneity. However, the subgroup effects reported in the RCT are not individual-level oracle CATEs and therefore cannot establish individual-level CATE estimation accuracy. CausalIDView complements this empirical evaluation with a synthetic multi-view benchmark that preserves the same SCM and ground-truth CATE across different identification regimes. This enables quantitative evaluation of query-level CATE estimation errors against known ground truth while allowing controlled comparisons of model performance across identification regimes.

[1] The Infant Health and Development Program. Enhancing the Outcomes of Low-Birth-Weight, Premature Infants: A Multisite, Randomized Trial. JAMA, 263(22):3035–3042, 1990.

AuthorsOpenReview ↗

[Q1] Native-Regime Comparisons and Cross-Regime Transfer Evaluations

Following your recommendation, we present the native-regime and cross-regime transfer results separately below and distinguish the conclusions supported by each comparison. In Figure 2, we compared CFMs across different observational views of the same causal world, although not all models were pre-trained specifically for their evaluation regimes. We therefore recognize the need to examine whether our main conclusions hold when models are evaluated under regimes aligned with their pre-training and have access to the relevant identifying variables.

To address this concern, we conducted additional native-regime and cross-regime transfer evaluations under Back-Door (BD) and Instrumental Variable (IV) identification. For CausalFM, we used the publicly released BD and IV checkpoints. For CausalPFN, we used the released BD checkpoint and additionally trained an IV-specific checkpoint using a binary-IV SCM prior. We define Native as evaluation under the same identification regime used during pre-training and Transfer as evaluation under a different regime. All models were evaluated on the same 40 SCM worlds as in Figure 2, preserving 1,024 context units, 100 query units, and the target CATE within each world. The BD and IV views observe (X,U,T,Y)(X,U,T,Y) and (X,I,T,Y)(X,I,T,Y), respectively.

Table Q1a: Native-regime evaluation. Models are evaluated under identification regimes aligned with their pre-training. Lower sPEHE indicates better CATE estimation accuracy.

ModelEval. viewsPEHE ↓Pearson r ↑Pred. CATE SD
CausalPFN-BDBD0.450980.526220.40965
CausalFM-BDBD0.481150.283930.07098
CausalPFN-IVIV0.678670.410110.06900
CausalFM-IVIV0.506350.183110.04628

Table Q1b: Cross-regime transfer evaluation. Models are evaluated under identification regimes different from those used during pre-training. For IV-trained checkpoints evaluated on BD, the instrument input is fixed to I=0I=0. These transfer results are treated as off-regime diagnostics rather than valid IV identification estimates.

ModelEval. viewsPEHE ↓Pearson r ↑Pred. CATE SD
CausalPFN-BDIV0.712760.475280.41389
CausalFM-BDIV0.548920.224900.09882
CausalPFN-IVBD0.672990.409710.06823
CausalFM-IVBD0.527740.196090.03840

The native-regime comparisons reveal changes in model rankings by sPEHE across identification regimes. In BD, CausalPFN-BD achieves lower sPEHE than CausalFM-BD (0.451 vs. 0.481), whereas in IV, CausalFM-IV outperforms the additionally trained CausalPFN-IV (0.506 vs. 0.679). Moreover, within each evaluation view, regime-aligned checkpoints achieve lower sPEHE than their off-regime counterparts for both model families. However, a checkpoint does not necessarily perform better on its native view than on a transfer view. For instance, CausalPFN-IV exhibits similar sPEHE in BD and IV (0.673 vs. 0.679).

We also observe limitations in recovering CATE heterogeneity under native-regime evaluation. While the average standard deviation of the oracle CATE is 0.489, CausalPFN-IV and CausalFM-IV produce substantially smaller predicted CATE standard deviations of 0.069 and 0.046, respectively. This suggests that regime-specific pre-training and access to the relevant identifying variables do not necessarily guarantee accurate recovery of CATE variation.

Overall, these additional experiments demonstrate that the regime-dependent ranking of CFMs by sPEHE is not solely an artifact of cross-regime transfer. Even when comparisons are restricted to native-regime configurations, the relative performance of CFMs varies between BD and IV, and limitations in estimation accuracy remain despite access to the relevant identifying information. These comparisons concern the evaluated checkpoints and do not isolate the effect of pre-training regime from differences in training history. We have incorporated the experimental settings and results into the revised manuscript (Appendix F.8).

Reviewer_hYu5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a benchmark to evaluate the performance of causal foundation models. It also included some modular causal effect identification approaches like S-learner with the prediction model TabPFN for comparison. The benchmark includes 4 settings (they call them views), backdoor, frontdoor, IV, proximal. For each causal world, each setting uses the same causal graph, same underlying data-generating process and target treatment effects.

Critical strengths

This paper provides a benchmark to evaluate different causal foundation models. And the benchmark includes different causal effect identification settings. This is interesting. And the results appear to be surprising because even for the simulation backdoor settings in Fig2 (all causal foundation models support this setting), causal foundation models can not beat some more direct causal effect identification approaches like X-learner. This can motivate the community to rethink whether the current causal pretraining of causal foundation models is really necessary?

Critical weaknesses

I like the comparison for backdoor settings because all the methods natively support this setting. But I doubt whether it is reasonable to compare them in Front-door, IV, proximal settings in the way that the author used? Take causalPFN as an example; causalPFN assumes that the covariate X can control all the confounders, which means it can satisfy the backdoor adjustment conditions, and the pretraining is designed for using backdoor adjustment to estimate the causal effect. But the authors directly use the backdoor pretrained checkpoint of causalPFN for front-door, IV, proximal settings. And the authors compared causalPFN with the approaches that directly use the correct identification formula for the settings. Take the front-door setting as an example: NN + FD Plug-in, XGBoost + FD Plug-in, TabPFN-v3.5 + FD Plug-in explicitly using the correct front-door adjustment formula. Therefore it is not surprising that causalPFN has a worse performance compared to those specialized methods for front-door settings.

Questions for authors

From the results of this paper, even in the back-door setting, cfm performs worse than other methods. Do you think there is any unique advantage that causal pretraining of the causal foundation model can provide?

4 comments
AuthorsOpenReview ↗

[W1] Native-Regime Comparisons and Cross-Regime Transfer

We agree with the reviewer that CFMs evaluated outside their pretraining identification regimes should not be regarded as being compared on equal footing with estimators that explicitly implement the appropriate identifying functionals. In particular, CausalPFN was pre-trained under conditional ignorability, so performance degradation when directly applying its released BD checkpoint to FD, IV, and PX settings is not unexpected. Accordingly, the corresponding comparisons in Figure 2 should be interpreted as cross-regime diagnostics examining how existing CFMs perform when the available identifying information changes within the same causal world, rather than as comparisons in which all models natively support each regime. Furthermore, our comparisons with modular estimators that explicitly implement regime-specific identifying functionals aim to assess the practical value of these identification procedures, and we do not attribute performance gaps under transfer settings to an inherent inability of CFMs to perform causal identification.

The original Figure 2 also includes direct comparisons between CFMs and modular estimators under native regimes. In BD, CausalPFN-BD achieves a higher mean sPEHE$^2$ than TabPFN + DR-Learner (0.2083 vs. 0.1434). Similarly, in FD, CausalFM-FD shows a higher error than TabPFN + FD Plug-in (0.5513 vs. 0.1534). In contrast, CausalFM-IV achieves a lower error than TabPFN + Wald in IV (0.2579 vs. 0.3357). These results show that modular estimators outperform native CFMs in some regimes, but this advantage does not hold consistently across all regimes.

To further address this concern, we conducted additional native-regime evaluations under BD and IV, aligning the pre-training and evaluation regimes. For CausalFM, we used the publicly released BD and IV checkpoints. For CausalPFN, we used the existing BD checkpoint and a newly trained IV checkpoint. Specifically, CausalPFN-IV was additionally pre-trained on a binary-IV SCM prior starting from the publicly released predictive TabDPT-based initialization. The prior includes observed covariates XX, a binary instrument II, and latent confounders UU, with conditional expected potential outcomes (CEPOs) approximated via conditional sequential Monte Carlo as training targets. We retained the original CausalPFN architecture and histogram cross-entropy loss to examine the effect of additional pretraining for the IV setting. All models were evaluated on the same SCM worlds, context/query units, and target CATEs as in Figure 2.

The additional training process also revealed notable characteristics. The CEPO cross-entropy training loss of CausalPFN-IV decreased only slightly, from an average of 4.8730 over the first 512 updates to 4.8607 over the most recent 512 updates. The sPEHE$^2$ measured on a separate fixed source-validation task also decreased from approximately 2.67 to 2.61, but improvements became limited during later training. These results indicate that additional pre-training with the IV prior was performed, but do not establish whether the model adequately learned the corresponding identification relationships. We have added the training curves and source-validation results to Appendix F.8 of the revised manuscript.

ModelEval. viewSettingsPEHE ↓Pearson r ↑Pred. CATE SD
CausalPFN-BDBDNative0.450980.526220.40965
CausalPFN-BDIVTransfer0.712760.475280.41389
CausalPFN-IVBDTransfer0.672990.409710.06823
CausalPFN-IVIVNative0.678670.410110.06900
CausalFM-BDBDNative0.481150.283930.07098
CausalFM-BDIVTransfer0.548920.224900.09882
CausalFM-IVBDTransfer0.527740.196090.03840
CausalFM-IVIVNative0.506350.183110.04628

Even when comparisons are restricted to native regimes, the relative performance of the models varies across identification regimes. In BD, CausalPFN-BD achieves a lower sPEHE than CausalFM-BD (0.451 vs. 0.481), whereas in IV, CausalFM-IV achieves a lower error than CausalPFN-IV (0.506 vs. 0.679). Furthermore, the additionally trained CausalPFN-IV achieves a lower sPEHE than the original CausalPFN-BD checkpoint evaluated on the IV view (0.679 vs. 0.713). Nevertheless, limitations remain in recovering CATE heterogeneity under IV. While the average standard deviation of the true CATE is 0.489, the predicted standard deviations of CausalPFN-IV and CausalFM-IV are only 0.069 and 0.046, respectively. This suggests that regime-specific pre-training and access to relevant identifying information do not necessarily guarantee accurate recovery of CATE variation.

AuthorsOpenReview ↗

[Q1] Practical Benefits of Causal Pre-training: Accuracy and Efficiency

Thank you for this insightful question. We acknowledge that CFMs do not consistently outperform conventional causal estimators in terms of accuracy, including in the back-door setting. We consider this one of the important findings of our study. However, we believe that the potential benefits of causal pre-training are not limited to achieving superior accuracy across all identification regimes.

Through pre-training on causal data-generating processes and causal estimands, CFMs can amortize causal effect estimation across datasets. This allows them to directly estimate causal effects on new datasets without repeatedly performing complex dataset-specific estimation procedures, potentially providing competitive estimates at lower computational cost.

To empirically investigate this benefit, we conducted an additional experiment comparing estimation accuracy and runtime on the benchmark used in Figure 2 (Appendix G). Most notably, in the IV setting, CausalFM achieves a 5.51$\times$ speedup over TabPFN--Wald while reducing mean sPEHE$^2$ by 23.2% (0.362 s vs. 1.995 s; 0.2579 vs. 0.3357). In the proximal setting, CausalFM is 40.4$\times$ faster than ExtraTrees--P-Learner, with only a 3.4% higher mean error. These results demonstrate that some CFMs can substantially reduce estimation costs without necessarily sacrificing estimation accuracy.

We acknowledge that these results do not isolate the contribution of causal pre-training from other model design factors. Moreover, our comparison focuses on deployment-time estimation and excludes pre-training costs. Nevertheless, our findings suggest that amortizing causal effect estimation through pre-training can offer practical accuracy--efficiency advantages in specific identification regimes. We therefore view preserving these benefits while improving accuracy and robustness across diverse identification regimes as a promising direction for future CFM research.

RegimeMethodsPEHE2sPEHE^2 ↓Time (s) ↓
BDCausalPFN0.20832.319
BDDo-PFN0.35950.939
BDCausalFM0.23260.363
BDTabPFN + X-Learner0.15286.688
BDTabPFN + S-Learner0.19690.502
BDTabPFN + DR-Learner0.14347.360
FDCausalPFN0.48542.334
FDDo-PFN0.36800.238
FDCausalFM0.55130.357
FDNN + FD Plug-in0.22723.970
FDXGBoost + FD Plug-in0.23346.192
FDTabPFN + FD Plug-in0.1534162.800
IVCausalPFN0.51792.335
IVDo-PFN0.37140.938
IVCausalFM0.25790.362
IVForestDRIV0.460369.086
IVKIV †0.42020.249
IVTabPFN + Wald0.33571.995
PROXCausalPFN0.24252.326
PROXDo-PFN0.34720.942
PROXCausalFM0.24030.372
PROXExtraTrees + P-Learner0.232515.031
PROXNN + P-Learner0.662410.946
PROXTabPFN + P-Learner0.181120.211
RegimeCFM vs. TabPFN with modular approachSpeed Up ↑relative error change ↓
IVCausalFM vs. TabPFN + Wald5.51× [5.44, 5.55]−23.2% [−29.4, −16.0]
PROXCausalFM vs. ExtraTrees + P-Learner40.41× [40.10, 40.61]+3.4% [−2.5, +9.5]
BDCausalFM vs. TabPFN + S-Learner1.39× [1.38, 1.39]+18.1% [10.5, 27.8]
FDDo-PFN vs. NN + FD Plug-in16.69× [16.18, 17.09]+62.0% [52.7, 71.6]

Thanks for your additional experimental results. My concerns are well addressed. I raise my score from 2 to 3.

Thank you very much for taking the time to review our additional results and for raising your score. We truly appreciate your constructive feedback, which has helped us improve the paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We sincerely thank Reviewers hYu5, fhM7, xS2E, and DLE8 for their thoughtful and constructive feedback. We summarize the strengths highlighted by the reviewers and the main additional results and revisions below. Detailed responses are provided in the individual comments.

  1. Strengths Highlighted by the Reviewers
  • [DLE8] An important evaluation problem. Comparing causal estimators across identification regimes helps clarify how pre-training assumptions affect CFM performance and limitations.

  • [fhM7, xS2E] Controlled matched-view design. Holding the causal world and target effect fixed while changing the observed identifying variables enables more informative cross-regime comparisons.

  • [fhM7] Diagnostics beyond average error. Stay/Move evaluations distinguish spurious responses from failures to track genuine effect changes, while the error decomposition separates mean-effect and heterogeneity errors.

  • [xS2E, DLE8] Strong Experimental Design and Broad Coverage. The benchmark evaluates diverse causal estimators across synthetic, semi-synthetic, and real-world datasets, including CFMs, conventional methods, and TabPFN-based modular approaches.

  • [hYu5, xS2E] Informative comparisons with modular estimators. Strong predictive models combined with explicit causal estimation procedures can outperform specialized CFMs, even in the native back-door setting, motivating a closer examination of the empirical value of causal pre-training.

  1. Summary of New Results and Revisions
  • [hYu5, fhM7, xS2E] Separate native-regime and transfer evaluations. We compare released CausalFM-BD/IV checkpoints and CausalPFN-BD with an additionally trained CausalPFN-IV. Relative rankings by mean sPEHE differ between BD and IV even among the evaluated native checkpoints. Transfer results are explicitly treated as diagnostics, not evidence of an inherent identification deficiency. See Appendix F.8, Table 11 and Figure 13.

  • [fhM7, xS2E] Extensions beyond the original SCM restrictions. We separately introduce latent-confounder-dependent treatment gains, a continuous confounder, and partial mediation. The leading performance patterns largely persist in comparisons where identification remains valid for the shared target. FD results under partial mediation are explicitly labeled as assumption-violation diagnostics. See Appendix F.3, Figures 8–10.

  • [fhM7] Real-world non-null effects and subgroup heterogeneity. Using actual IHDP treatment assignments and IQ outcomes, we compare subgroup-averaged predictions with published experimental effects. All evaluated models predict positive effects but produce smaller between-subgroup contrasts than the reported 6.6-point difference. These are comparisons with empirical references, not oracle-CATE evaluations. See Appendix F.2, Figure 7.

  • [xS2E] Real-world comparison of identification strategies. The additional JTPA experiment compares BD and FD estimates of a non-null ATT against experimental references. BD estimates show greater sensitivity to covariate specification than FD estimates. We explicitly distinguish this complementary real-world comparison from the fully matched synthetic multi-view CATE benchmark. See our response to xS2E, W3.

  • [fhM7] Evaluation consistency across treatment-effect families. We disaggregate the original 40-world results into five families, with eight worlds each. TabPFN-based modular estimators lead in BD, FD, and PX across all families, while CausalFM leads in IV. Extending evaluation to 100 worlds preserves the best-performing configuration in 19 of 20 regime–family combinations. See Appendix F.4, Tables 7–8.

  • [hYu5] Practical accuracy–efficiency benefits. In native IV evaluation, CausalFM-IV is 5.51× faster than TabPFN + Wald while reducing mean sPEHE² by 23.2%. In PX, CausalFM is 40.4× faster than ExtraTrees + P-Learner with 3.4% higher mean sPEHE². These deployment-time comparisons exclude pretraining costs and do not isolate causal pretraining from other design factors. See Appendix G, Figure 14.

  • [DLE8] A more self-contained main text. We moved the hidden-confounding and partial-identification figure into the main text as Figure 4 and added descriptions of Causal Tree, GRF, TARNet, and CFRNet in Section 5.2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.