acceptodds
Under review as a conference paper at ICLR 2027

Learning Outcomes of Joint Interventions via Sim2Real Causal Foundation Models

Abstract

Real-world decisions rarely involve a single lever. Drug combinations, multiplexed gene perturbations, and multi-component policies require predicting outcomes when several variables are manipulated at once, often with limited or no interventional data. Motivated by these demands, we propose a new paradigm: learning outcomes of joint interventions (OJIs) with a single pre-trained model that infers them from observational data alone and flexibly exploits interventional data when available. In their original implementations, causal foundation models (CFMs) such as CausalPFN and Do-PFN address single-variable interventions using observational-only pre-training contexts. We realize this paradigm with JOINT (Joint Outcome Inference via Neural Transfer), a CFM pre-trained entirely on diverse causal systems simulated via next-token generation. JOINT is pre-trained on contexts that randomly include interventional samples to align pre-training with deployment, reducing mean average treatment effect (ATE) error by approximately 39% relative to observational-only pre-training when interventional examples are available. For interventions on up to ten variables, JOINT estimates individual treatment effects more accurately and achieves lower ATE error than competing causal and tabular foundation models. It remains robust under partial observability and generalizes to graphs twice the size of those seen during pre-training. Without fine-tuning, JOINT attains the lowest ATE error on real-world genetic and chemical perturbation datasets, improving on the strongest baseline by approximately 13% and demonstrating effective Sim2Real transfer.

Primary area:
foundation or frontier models, including LLMs
Keywords:
joint interventions, causal foundation models, sim-to-real transfer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1G75OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes JOINT, a causal foundation model built on TabICL and pre-trained on synthetic data generated from structural causal models (SCMs). At inference time, predictions for new datasets and queries are made in context, so no (re-)training is required. The model uses both observational and interventional data: a dedicated positional encoding marks variables that were directly intervened on. From the synthetic data, the model learns to tell causal relationships apart from confounded ones. In evaluations on synthetic and real-world data, JOINT consistently outperforms the competing methods.

Critical strengths
  • The Sim2Real paradigm is a promising direction for biomedical applications, where data are often scarce and mostly observational.
  • Both the data-generating process and the prediction architecture are well thought out.
Critical weaknesses
  • Limited real-world evaluation.
    • The real-world evaluation covers only two datasets. The baselines are restricted to other foundation models. Simple linear models are known to be strong competitors in this setting. To judge practical relevance, the method should be compared against the actual state of the art, not only against methods from the same paradigm.
    • The experimental task on the real-world data does not reflect a realistic use case. Predicting a single gene while covariate genes are measured is not a practical problem: if that gene were of interest, it would simply be measured as well. A more convincing setting would be, for example, treatment effect estimation for patients from observational health data.
  • Limited ablation study. It is unsurprising that pre-training without interventional data performs worse. Interventional data are exactly what lets the model separate spurious correlations from causal effects. More informative ablations would target the design choices themselves:
    • the data-generating process (e.g., linear models instead of transformers),
    • the choice of base predictive model,
    • the intervention embedding and the topological role embedding.
  • Limited out-of-distribution experiments. The number of variables is varied, but no model shows a notable drop in performance. The regime with fewer variables even appears harder, likely because of a lower signal-to-noise ratio. In these settings, a variable’s local neighborhood in the SCM is barely distinguishable across conditions, so the experiment does little to probe generalization. It would be informative to also vary the average in-degree of the causal graphs.
Questions for authors
  1. How does JOINT compare against established state-of-the-art methods on the real-world datasets? This includes simple linear models and other methods outside the foundation-model paradigm.
  2. Could the authors evaluate JOINT on additional real-world datasets? Could they also include a more realistic task, such as treatment effect estimation from observational patient data?
  3. How sensitive is performance to the choice of data-generating process during pre-training? For example, what happens with linear SCMs instead of transformer-based ones?
  4. How much does the choice of TabICL as the base model contribute to the results? Have other base architectures been tried?
  5. What are the individual contributions of the intervention embedding and the topological role embedding? Could the authors ablate each separately?
  6. How does performance change when the average in-degree of the causal graphs is varied, in addition to the number of variables?
  7. Do the authors have an explanation for why the regime with fewer variables appears harder? Is this driven by the signal-to-noise ratio?
3 comments

We thank the reviewer for the positive assessment of the design and the concrete suggestions.

Real-world baselines and task realism (Q1, Q2). We add linear and additive baselines, and JIL. The task now predicts all 48 evaluation genes from the perturbation indicators and context alone, which is the decision-relevant question: which combination to run before measuring the query’s gene responses. JOINT achieves the lowest mean held-out-combination ATE error among the seven methods on both datasets (0.02983 on Norman and 0.05720 on ComboSciPlex, Table 3). The evaluation remains limited to these two datasets and the few-shot setting. Regarding patient treatment-effect estimation, Appendix F.3 already evaluates IHDP, ACIC and Lalonde with pre-treatment covariates. JOINT is competitive on Lalonde and on IHDP ATE relative error, but is weaker on IHDP PEHE and on ACIC.

Ablations (Q3, Q4, Q5). The retained ablation compares full JOINT with its observational-only pre-training variant under the revised query protocol (Figure 5). JOINT achieves lower mean PEHE and ATE error in all 12 settings, reducing mean ATE error by 48.2% across the nine few-shot settings and by 43.3–54.1% across the three zero-shot settings. This comparison evaluates the overall dual-regime design, including its intervention representation. It does not separately isolate the generator, intervention embedding or role embedding. We have not included separate ablations of those components in this revision. We consider the backbone orthogonal to the contribution and leave comparisons with other backbones to future work.

In-degree (Q6). We vary expected average in-degree at fixed d=50d=50, using edge probabilities 0.025, 0.05, and 0.10, corresponding to expected average in-degrees 0.6125, 1.225, and 2.45. JOINT’s PEHE and ATE error increase with graph density, but remain the lowest across all nine combinations of density and intervention-set size ∣I∣∈1,2,5|I|\in\\{1,2,5\\} (Appendix F.2, Figure 7).

Why smaller graphs looked harder (Q7). Under the revised setting, JOINT’s PEHE and ATE error remain higher at d=25d=25 than at d=50d=50 for all three intervention-set sizes. However, the relationship is not monotonic: errors increase again at d=75d=75 and decrease at d=100d=100 (Figure 4). All non-intervened query measurements are now masked, so the remaining pattern cannot be attributed to having fewer observed post-intervention query variables. These experiments do not isolate whether the pattern is driven by signal-to-noise ratio or another property of the generated systems.

I commend the authors on the comprehensive revision they have made to the paper. Such a large conceptual deviation from the initial submission may, however, warrant an entirely new review.

The authors did not address my concerns on the limited number of real-world datasets, ablation of the architecture, or the OOD setting.

I maintain my score.

Thank you for your further feedback and for acknowledging the substantial revision. We would like to clarify the evidence supporting our contributions and the remaining limitations.

1. Architecture ablations. Our comparisons demonstrate benefits of the complete JOINT framework, while individual component attribution remains open.

Our core evaluations include pretrained TabICL, and Figure 5 additionally supports the overall dual-regime training design. These comparisons do not isolate individual architectural changes. Separate component ablations require substantial retraining and computational resources, so we cannot add them within this revision. This limits attribution of the gains, but does not negate the demonstrated framework-level benefits.

2. OOD evaluation. The revised experiments provide concrete evidence of transfer beyond the default training setting, alongside robustness across generator configurations.

Figure 4 evaluates graphs with up to 100 variables, twice the pre-training maximum. At this size, JOINT achieves the lowest mean ATE error across all three intervention sizes, although TabICL has lower PEHE. Figures 7 and 8 show JOINT’s advantage across different expected in-degrees and initialization scales.

New tests on independent Linear-Gaussian and Do-PFN-ANM generators appear in Table 2. JOINT leads on both metrics in all Linear-Gaussian settings and on zero-shot PEHE for Do-PFN-ANM, with mixed results elsewhere. Table 3 further demonstrates transfer to real perturbation data. To conclude, these results support meaningful transfer benefits, without claiming universal OOD superiority.

3. Real-world coverage. Our two datasets provide evidence across genetic and chemical perturbations, although broader coverage remains desirable.

On Norman and ComboSciPlex, JOINT achieves the lowest mean held-out-combination ATE error among seven methods, including the added linear, additive, and JIL baselines (Table 3). Obtaining and validating additional suitable datasets presents practical difficulties within the revision period. We acknowledge this limitation while emphasizing the evidence already provided.

Our main contribution is the AOJI paradigm and its realization through JOINT. Section 3.1 and Table 1 distinguish this setting through arbitrary intervention subsets, prediction of all remaining non-intervened variables, prospective query information, and flexible use of context without per-system updates. This formulation offers a distinct perspective on joint-intervention prediction, with benefits demonstrated across multiple settings.

Reviewer_mC6mOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to predict the outcomes of joint interventions on several variables from observational data and, when available, interventional data. The authors first build a simulator in which a randomly initialized Transformer generates variables in causal (topological) order, with attention masks restricting each variable to its parents. They then propose JOINT, a causal foundation model built on the TabICL backbone and pre-trained on these simulated systems, with contexts that randomly include interventional samples and with embeddings that mark intervened variables and the role of the target. On simulated systems with up to ten intervened variables, JOINT achieves lower PEHE and ATE error than baselines, and it remains accurate under partial observation and on graphs larger than those seen in pre-training. On two real perturbation datasets, Norman and ComboSciPlex, JOINT performs comparably to or slightly better than the baselines in a few-shot setting.

Critical strengths
  1. Predicting the effects of joint interventions with one pre-trained model is an important problem, and the Transformer-based simulator is a simple way to separate the causal graph from the functional mechanisms.

  2. The paper is clearly written, and the experiments cover many settings (intervention-set size, graph size, partial observation, graph density).

Critical weaknesses
  1. The technical novelty is limited, and the task is easier than it appears. Based on my understanding, Do-PFN already samples explicit causal graphs, so allowing several intervened variables, marking them with an embedding, and adding interventional samples to the context are natural extensions. More importantly, at query time JOINT sees all other variables measured after the intervention, including those downstream of the target. The target can then be predicted largely from its observed neighbors, which may explain why JOINT's error barely changes with the number of intervened variables. In practice, for example, when choosing a drug combination, these post-intervention values are not available before acting. This setting also puts CausalPFN and Do-PFN at a disadvantage, since they were designed for a single treatment and pre-treatment covariates.

  2. The experiments do not show that JOINT works beyond its own simulator. The synthetic test systems come from the same simulator used for pre-training. On the semi-synthetic benchmarks in Appendix F.3, which use pre-treatment covariates, JOINT's PEHE on IHDP and ACIC is several times higher than CausalPFN's. The real-data experiments are few-shot, with both single perturbations in the context, so they do not test observational-only inference. On the held-out combination (Table 2), the gain over the best baseline is only marginal, and simple baselines such as an additive baseline that sums the two single effects are missing.

Questions for authors
  1. In the motivating applications, post-intervention values of other variables are not available before acting. How does JOINT compare with the baselines when the query contains only pre-intervention variables or only the intervention indicators?

  2. On the held-out combinations in Norman and ComboSciPlex, how does JOINT compare with JIL and with an additive baseline that sums the two single-perturbation effects?

  3. How does JOINT perform on test data from a simulator other than its own, for example, with additive-noise mechanisms (as used to pre-train Do-PFN)? Also, both the pre-training and test graphs sample each edge independently (Erdős–Rényi). How does JOINT perform on scale-free graphs, which are often used to model gene regulatory networks?

2 comments

We thank the reviewer for the precise and actionable questions.

Post-intervention conditioning (Q1). Agreed; see General Response. The reviewer’s hypothesis that this explains the flat error across ∣I∣|I| was a useful diagnostic: under the revised setting, JOINT’s errors remain comparatively flat over ∣I∣=1,…,8|I|=1,\ldots,8, with non-monotonic fluctuations (Figure 3). Because all non-intervened query measurements are now masked, the persistent pattern cannot be explained by access to those measurements. JOINT retains the lowest mean PEHE and ATE error in every evaluated intervention-size/context setting. CausalPFN and Do-PFN use the same available query information.

Held-out combinations (Q2). We add JIL and the additive baseline (sum of the two single-perturbation effects) on Norman and ComboSciPlex, and report the held-out combination separately. The following results use intervention indicators only and few-shot contexts, with mean ± SE over 100 tasks and errors averaged over 48 genes per task (Table 3):

MethodNorman ATE errorComboSciPlex ATE error
JOINT0.02983 ± 0.001480.05720 ± 0.00150
Additive0.03129 ± 0.001220.06798 ± 0.00155
Linear0.03148 ± 0.001220.06841 ± 0.00157
JIL0.09927 ± 0.004520.29389 ± 0.01445

JOINT’s mean held-out-combination ATE error is 4.7% lower than Additive on Norman and 15.9% lower on ComboSciPlex. The Norman improvement is modest. For ATE error averaged over all non-control assignments, Additive performs slightly better on Norman (0.02643 versus JOINT’s 0.02693), while JOINT performs best on ComboSciPlex (0.05401 versus Additive’s 0.05990).

Other simulators and scale-free graphs (Q3). We evaluate on additive-noise systems following Do-PFN’s prior and on linear-Gaussian SCMs (Table 2). JOINT achieves the lowest mean error in most comparisons across these two priors and remains competitive in the others. In particular, it achieves the lowest mean ATE error and PEHE in all evaluated linear-Gaussian settings. These experiments change the mechanism family. Scale-free graphs are not included in the current evaluation, so generalization to that topology remains untested.

Novelty vs Do-PFN. Beyond multiple treatments, the revised setting differs in what is predicted (all remaining non-intervened variables, no designated outcome) and what is in context (mixed interventional data from arbitrary intervention sets). Our ablation compares full JOINT with its observational-only pre-training variant under the revised setting: JOINT achieves lower mean ATE error in all 12 settings, reducing mean ATE error by 48.2% across the nine few-shot settings (Figure 5).

IHDP and ACIC. These benchmarks are a special case with a single treatment and pre-treatment covariates. In the existing Appendix F.3 results, JOINT’s PEHE is higher than CausalPFN’s on IHDP (3.73 versus 0.58) and ACIC (3.26 versus 0.92). Its ATE relative error is comparable on IHDP (0.19 versus 0.20), but worse on ACIC (0.26 versus 0.05). We report these limitations without claiming uniformly better performance on conventional single-treatment tasks.

Reviewer_mC6mOpenReview ↗

Official Comment by Reviewer mC6m

Thank you for the detailed response and the effort. The rebuttal has addressed my concern, so I have raised my score to 3.

Reviewer_vinLOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes JOINT, a pretrained foundation model for estimating outcomes under joint interventions. The approach uses a randomly initialized Transformer with causal attention masks to synthesize structural causal models and generate observational and interventional training data. JOINT is then trained to predict masked outcomes from observational contexts, optionally augmented with interventional examples. The model supports different intervention sets and prediction targets without test-time parameter updates.

The experiments evaluate JOINT on synthetic causal systems with varying intervention-set sizes, graph sizes, and partially observed inputs. The authors also evaluate Sim2Real transfer on two real-world genetic and chemical perturbation datasets. JOINT generally outperforms the evaluated foundation-model baselines, and the ablation study shows benefits from including interventional examples during pretraining.

Critical strengths
  1. Joint-intervention outcome prediction is an important problem, and extending causal foundation models beyond single-variable interventions is a meaningful direction. The proposed framework accommodates different intervention subsets and allows the use of interventional examples at inference time.

  2. The paper attempts to evaluate Sim2Real transfer rather than relying entirely on synthetic benchmarks. The results on measured genetic and chemical perturbations are encouraging, although their interpretation depends on the evaluation setting discussed below.

Critical weaknesses

1. The evaluation does not convincingly establish the claimed causal effect estimation capability.

My main concern is the formulation of the prediction task.

The paper motivates JOINT as a model for predicting the outcomes of joint interventions, which is particularly important when deciding which interventions to perform. However, the actual prediction objective in Eq. (6) conditions on other variables already realized under the queried intervention.

This is a substantially different problem. If these variables are downstream of the intervention, or even of the target outcome, predicting the missing target can become a conventional conditional prediction problem rather than estimating the outcome of an intervention that has not yet been performed.

This distinction is particularly important in the real-world experiments, where the model predicts a target gene's expression using 47 other gene-expression measurements from the perturbed cells. These measurements would not ordinarily be available when deciding which perturbation to perform.

2. The methodological contribution and generalization evidence are insufficient.

The proposed approach follows an established supervised causal learning paradigm: synthesize causal systems with known intervention outcomes and train a model to predict these outcomes. The main differences are the use of a Transformer to parameterize the structural mechanisms and the inclusion of interventional examples during pretraining.

It is not clear what fundamental advantage the proposed next-token data synthesis offers over existing SCM simulators. Traditional simulation approaches can already independently control graph structures, functional mechanisms, and noise distributions, while supporting joint interventions. These choices also provide a natural way to define and evaluate generalization across causal assumptions.

In contrast, the main synthetic experiments use the same Transformer-based SCM generation framework for both training and testing. Varying graph size, density, or generator parameters does not establish generalization to unseen causal mechanism families. The reported results therefore provide limited evidence that JOINT has learned a transferable causal inference capability rather than characteristics of its synthetic training distribution.

The real-world evaluation is potentially more convincing, but covers only two perturbation datasets, with two intervention variables in each case, and is affected by the prediction-setting concern above. The comparisons also rely heavily on models originally designed for single-variable interventions or general tabular prediction.

Overall, the paper demonstrates promising predictive performance under its evaluation protocol. However, I do not find sufficient evidence that it establishes a general-purpose causal foundation model for joint-intervention effect estimation. The distinction between learning to predict data generated by SCMs and learning a transferable causal inference capability remains insufficiently examined.

Questions for authors
  1. Prediction task. What causal estimation setting is JOINT intended to address? In particular, how should we interpret the use of non-target variables realized under the same intervention at query time? Can the authors evaluate the model using only the intervention assignments and information available before the intervention? This would help establish whether JOINT can predict intervention outcomes prospectively, rather than imputing missing measurements from partially observed interventional data.

  2. Generalization. Can the authors evaluate JOINT on SCMs generated by independent simulation frameworks, with graph families, functional mechanisms, and noise distributions different from those used during pretraining? Such experiments would provide more direct evidence of generalization beyond the synthetic training prior.

  3. Methodological contribution. What is the essential advantage of Transformer-based causal data synthesis over existing SCM generators? Does it provide better coverage of causal mechanisms or better transfer to unseen systems? More broadly, what evidence distinguishes JOINT's causal inference capability from ordinary outcome prediction using intervention annotations and partially observed post-intervention variables?

1 comment

We thank the reviewer for pinpointing the most important issue in the submission.

Prediction task (W1, Q1). The reviewer is right that conditioning on non-target variables realized under the same intervention turns the task into imputation for variables downstream of II. As described in the General Response, JOINT now addresses amortized OJI inference (AOJI): the query contains only the intervention assignment and, optionally, covariates declared to be measured before acting. All other non-intervened variables are inferred using a mean-field approximation in Eq. (6). This is exactly the prospective evaluation the reviewer requested. Under this setting, JOINT achieves the lowest mean PEHE and ATE error across all 32 intervention-size/context settings in Figure 3. On Norman and ComboSciPlex with intervention indicators only, its held-out-combination ATE errors are 0.02983 ± 0.00148 and 0.05720 ± 0.00150, respectively, the lowest means among the compared methods.

Generalization (W2, Q2). We test on SCMs from independent generators not used in pre-training: linear-Gaussian SCMs and additive-noise models following Do-PFN’s prior (Table 2). JOINT achieves the lowest mean error in most comparisons across these two priors and remains competitive in the others. In particular, it achieves the lowest mean ATE error and PEHE in all evaluated linear-Gaussian settings. These tests assess transfer across mechanism families.

Methodological contribution (Q3). We see the contribution at two levels. First, the problem setting: interventions on arbitrary subsets, inference over all remaining non-intervened variables without a designated outcome, and adaptive use of interventional context (Table 1, Sec. 3.1). Second, the Transformer generator: masked attention fixes the graph independently of the mechanism, and attention can represent non-additive interactions between parents and noise. We agree that standard SCM generators can also control graphs and support joint interventions. The independent-generator tests assess transfer, but do not isolate the benefit of the Transformer generator over alternative training priors. We also clarify that, as with all PFN-style models, JOINT approximates the posterior predictive under its prior; we now state this scope explicitly rather than claim general causal identification.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful and constructive reviews. All three raised the same central issue: at query time, JOINT conditioned on non-target variables realized under the queried intervention, which does not match the prospective decision setting that motivates the paper. We agree. We have revised the formulation, retrained JOINT, and re-run the experiments. The main changes are:

  1. A clearly stated paradigm with a prospective, joint objective. A new Section 3.1 defines amortized OJI inference (AOJI): given only an intervention do(VI=vI)\mathrm{do}(V_I=v_I) on any subset, an optional set of covariates declared to be measured before acting, and an observational (optionally interventional) context, infer the joint distribution of all remaining non-intervened variables. No post-intervention measurement of any remaining non-intervened variable is used in the query, and the model is not told which variables are outcomes. Eq. (6) represents this predictive distribution as a conditionally independent product of scalar predictive distributions sharing the same conditioning information. The first training stage randomly samples a target column with all non-target query variables observed (Eq. (7)); the second continues training with a randomly sampled target column and visible-variable set (Eq. (8)). All compared methods use the same available query information.

  2. Re-run results under AOJI. With all non-intervened query measurements masked, JOINT achieves the lowest mean PEHE and ATE error across all 32 combinations of intervention-set sizes 1–8 and interventional context sizes 0, 32, 64, and 128 (Figure 3). Both metrics are averaged over all remaining non-intervened variables, with 100 causal systems per setting. JOINT’s zero-shot mean PEHE and ATE error are also lower than those of the other methods in their few-shot settings. The graph-size, edge-probability and initialization-scale evaluations use the same query protocol (Figures 4, 7 and 8).

  3. Generalization beyond our simulator. We evaluate on systems from independent generators: linear-Gaussian SCMs and additive-noise models following Do-PFN’s prior. These tests use two intervened variables and either 0 or 128 interventional context samples (Table 2). JOINT achieves the lowest mean error in most comparisons across these two priors and remains competitive in the others. In particular, it achieves the lowest mean ATE error and PEHE in all evaluated linear-Gaussian settings.

  4. Stronger real-world evaluation. On Norman and ComboSciPlex the query now contains only intervention indicators. We add additive, linear and JIL baselines, and report the held-out combination separately in the few-shot setting. Across 100 tasks per dataset, with errors averaged over 48 genes per task, JOINT achieves held-out-combination ATE errors of 0.02983 ± 0.00148 and 0.05720 ± 0.00150, respectively (mean ± SE, Table 3). These are the lowest means among the seven methods, improving on Additive, the best baseline, by 4.7% and 15.9%. When averaging over all non-control assignments, Additive has the lowest mean ATE error on Norman, while JOINT has the lowest on ComboSciPlex (Appendix F.1).

  5. Dual-regime pre-training ablation. We compare full JOINT with its observational-only pre-training variant under the revised query protocol (Figure 5). JOINT achieves lower mean PEHE and ATE error in all 12 evaluated settings. Relative to this variant, JOINT reduces mean ATE error by 48.2% across the nine few-shot settings and by 43.3–54.1% across the three zero-shot settings. The 48.2% reduction is computed from the ATE means averaged over the nine few-shot settings. This comparison evaluates the overall dual-regime design, including the different intervention representations used by the two variants.

We have updated the PDF; changes are marked in blue. Per-reviewer responses follow.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.