- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes JOINT, a causal foundation model built on TabICL and pre-trained on synthetic data generated from structural causal models (SCMs). At inference time, predictions for new datasets and queries are made in context, so no (re-)training is required. The model uses both observational and interventional data: a dedicated positional encoding marks variables that were directly intervened on. From the synthetic data, the model learns to tell causal relationships apart from confounded ones. In evaluations on synthetic and real-world data, JOINT consistently outperforms the competing methods.
Critical strengths
- The Sim2Real paradigm is a promising direction for biomedical applications, where data are often scarce and mostly observational.
- Both the data-generating process and the prediction architecture are well thought out.
Critical weaknesses
- Limited real-world evaluation.
- The real-world evaluation covers only two datasets. The baselines are restricted to other foundation models. Simple linear models are known to be strong competitors in this setting. To judge practical relevance, the method should be compared against the actual state of the art, not only against methods from the same paradigm.
- The experimental task on the real-world data does not reflect a realistic use case. Predicting a single gene while covariate genes are measured is not a practical problem: if that gene were of interest, it would simply be measured as well. A more convincing setting would be, for example, treatment effect estimation for patients from observational health data.
- Limited ablation study.
It is unsurprising that pre-training without interventional data performs worse. Interventional data are exactly what lets the model separate spurious correlations from causal effects.
More informative ablations would target the design choices themselves:
- the data-generating process (e.g., linear models instead of transformers),
- the choice of base predictive model,
- the intervention embedding and the topological role embedding.
- Limited out-of-distribution experiments. The number of variables is varied, but no model shows a notable drop in performance. The regime with fewer variables even appears harder, likely because of a lower signal-to-noise ratio. In these settings, a variable’s local neighborhood in the SCM is barely distinguishable across conditions, so the experiment does little to probe generalization. It would be informative to also vary the average in-degree of the causal graphs.
Questions for authors
- How does JOINT compare against established state-of-the-art methods on the real-world datasets? This includes simple linear models and other methods outside the foundation-model paradigm.
- Could the authors evaluate JOINT on additional real-world datasets? Could they also include a more realistic task, such as treatment effect estimation from observational patient data?
- How sensitive is performance to the choice of data-generating process during pre-training? For example, what happens with linear SCMs instead of transformer-based ones?
- How much does the choice of TabICL as the base model contribute to the results? Have other base architectures been tried?
- What are the individual contributions of the intervention embedding and the topological role embedding? Could the authors ablate each separately?
- How does performance change when the average in-degree of the causal graphs is varied, in addition to the number of variables?
- Do the authors have an explanation for why the regime with fewer variables appears harder? Is this driven by the signal-to-noise ratio?
3 comments
I commend the authors on the comprehensive revision they have made to the paper. Such a large conceptual deviation from the initial submission may, however, warrant an entirely new review.
The authors did not address my concerns on the limited number of real-world datasets, ablation of the architecture, or the OOD setting.
I maintain my score.
Thank you for your further feedback and for acknowledging the substantial revision. We would like to clarify the evidence supporting our contributions and the remaining limitations.
1. Architecture ablations. Our comparisons demonstrate benefits of the complete JOINT framework, while individual component attribution remains open.
Our core evaluations include pretrained TabICL, and Figure 5 additionally supports the overall dual-regime training design. These comparisons do not isolate individual architectural changes. Separate component ablations require substantial retraining and computational resources, so we cannot add them within this revision. This limits attribution of the gains, but does not negate the demonstrated framework-level benefits.
2. OOD evaluation. The revised experiments provide concrete evidence of transfer beyond the default training setting, alongside robustness across generator configurations.
Figure 4 evaluates graphs with up to 100 variables, twice the pre-training maximum. At this size, JOINT achieves the lowest mean ATE error across all three intervention sizes, although TabICL has lower PEHE. Figures 7 and 8 show JOINT’s advantage across different expected in-degrees and initialization scales.
New tests on independent Linear-Gaussian and Do-PFN-ANM generators appear in Table 2. JOINT leads on both metrics in all Linear-Gaussian settings and on zero-shot PEHE for Do-PFN-ANM, with mixed results elsewhere. Table 3 further demonstrates transfer to real perturbation data. To conclude, these results support meaningful transfer benefits, without claiming universal OOD superiority.
3. Real-world coverage. Our two datasets provide evidence across genetic and chemical perturbations, although broader coverage remains desirable.
On Norman and ComboSciPlex, JOINT achieves the lowest mean held-out-combination ATE error among seven methods, including the added linear, additive, and JIL baselines (Table 3). Obtaining and validating additional suitable datasets presents practical difficulties within the revision period. We acknowledge this limitation while emphasizing the evidence already provided.
Our main contribution is the AOJI paradigm and its realization through JOINT. Section 3.1 and Table 1 distinguish this setting through arbitrary intervention subsets, prediction of all remaining non-intervened variables, prospective query information, and flexible use of context without per-system updates. This formulation offers a distinct perspective on joint-intervention prediction, with benefits demonstrated across multiple settings.
We thank the reviewer for the positive assessment of the design and the concrete suggestions.
Real-world baselines and task realism (Q1, Q2). We add linear and additive baselines, and JIL. The task now predicts all 48 evaluation genes from the perturbation indicators and context alone, which is the decision-relevant question: which combination to run before measuring the query’s gene responses. JOINT achieves the lowest mean held-out-combination ATE error among the seven methods on both datasets (0.02983 on Norman and 0.05720 on ComboSciPlex, Table 3). The evaluation remains limited to these two datasets and the few-shot setting. Regarding patient treatment-effect estimation, Appendix F.3 already evaluates IHDP, ACIC and Lalonde with pre-treatment covariates. JOINT is competitive on Lalonde and on IHDP ATE relative error, but is weaker on IHDP PEHE and on ACIC.
Ablations (Q3, Q4, Q5). The retained ablation compares full JOINT with its observational-only pre-training variant under the revised query protocol (Figure 5). JOINT achieves lower mean PEHE and ATE error in all 12 settings, reducing mean ATE error by 48.2% across the nine few-shot settings and by 43.3–54.1% across the three zero-shot settings. This comparison evaluates the overall dual-regime design, including its intervention representation. It does not separately isolate the generator, intervention embedding or role embedding. We have not included separate ablations of those components in this revision. We consider the backbone orthogonal to the contribution and leave comparisons with other backbones to future work.
In-degree (Q6). We vary expected average in-degree at fixed d=50, using edge probabilities 0.025, 0.05, and 0.10, corresponding to expected average in-degrees 0.6125, 1.225, and 2.45. JOINT’s PEHE and ATE error increase with graph density, but remain the lowest across all nine combinations of density and intervention-set size ∣I∣∈1,2,5 (Appendix F.2, Figure 7).
Why smaller graphs looked harder (Q7). Under the revised setting, JOINT’s PEHE and ATE error remain higher at d=25 than at d=50 for all three intervention-set sizes. However, the relationship is not monotonic: errors increase again at d=75 and decrease at d=100 (Figure 4). All non-intervened query measurements are now masked, so the remaining pattern cannot be attributed to having fewer observed post-intervention query variables. These experiments do not isolate whether the pattern is driven by signal-to-noise ratio or another property of the generated systems.