acceptodds
Under review as a conference paper at ICLR 2027

CIDER-FM: Foundation Models for Causal Inference from Diverse Experimental Regimes

11 views

Abstract

Causal foundation models (CFMs) amortise causal inference over priors of synthetic structural causal models (SCMs), predicting the effect of an experiment on a specific variable. However, observational data alone may leave multiple causal models compatible with available evidence, while experimental data with interventions on exactly the variable of interest might be unavailable. This work studies CFMs as a method to combine finite observational and surrogate-interventional datasets in order to predict a target conditional interventional distribution (CID) more accurately than with observational data alone. We first formalise the conceptual benefits of surrogate experiments. Building on this analysis, we introduce Foundation Models for Causal Inference from Diverse Experimental Regimes (CIDER-FM), a causal foundation model that uses an intervention-aware representation and hierarchical three-axis attention to exchange information across variables, samples, and experimental regimes. We evaluate CIDER-FM against a wide range of baselines across diverse synthetic graph and mechanism families, as well as on both simulated and real-world data from Causal Chambers. Our results demonstrate strong CID prediction performance and show that incorporating experimental context can improve predictions over observational data alone.

Primary area:
causal reasoning
Keywords:
Causal Foundation Models, Causal Inference, Experimental Design, Tabular Foundation Models, In-Context Learning, Synthetic Data

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_duDLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a new foundation model for causal inference. These foundation models answer causal queries about unknown causal systems given observational and interventional data from that system. The key novelty is that they include interventional data as context at test time.

Critical strengths
  • Overall Excellent presentation
  • Use of the real-world Chambers dataset.
  • Natural and relevant extension of causal foundation models.
  • Figure 3 and the discussion of identifiability are particularly useful.
Critical weaknesses
  • The biggest weakness is the external CFM comparisons. They are difficult to interpret because, it seems, the baselines are pretrained under different priors and information regimes. CIDER-FM-Obs appears to be the only closely matched baseline.
  • The novelty relative to existing foundation models using interventional context could be explained more clearly.
  • It remains unclear how much of the gain comes from the proposed three-way attention and if this is really the best way of using the interventional context.
  • The repository has no README explaining how to reproduce results of the manuscript.
Questions for authors
  • Which parts of the theoretical results are specific to CIDER-FM?
  • How does CIDER-FM relate to MapPFN (not cited), TabCausal (not cited), and PerturbPFN (cited but not discussed in detail)?
  • Given some evidence that tabular foundation models can behave like implicit retrieval methods, how do you assess whether CIDER-FM learns transferable causal reasoning or mainly exploits similarity to its training prior and context?
  • Could you include a comparison to a foundation model that can be considered fair?
  • How do I reproduce the paper's results using the repo?
4 comments
AuthorsOpenReview ↗

Response to Reviewer duDL (1/3)

Thank you for your comprehensive review!

We are especially happy that you find the presentation of the paper to be excellent.

We adress your concerns and questions below and have updated the manuscript accordingly.

Weaknesses

The biggest weakness is the external CFM comparisons.

Thank you for highlighting this distinction. In general, we are not aware of any directly applicable CFM that we can directly compare to for this task.

In repsonse to your comment, we trained a new Graph4CFM model [4] using CIDER-FM’s ComplexMech prior. The retraining is needed to make sure both CIDER-FM and the Graphs4CFM are trained on the same prior. Please note that we always provide the same number of context samples for both models. While CIDER-FM uses some interventional samples, the Graphs4CFM model uses observational data together with ground-truth graphical information.

ComplexMech: original 512 test SCMs

MethodMean NLL ↓Median NLL ↓MSE ↓Mean R2R^2 ↑Median R2R^2 ↑Pooled R2R^2 ↑
CIDER-FM−0.4369−0.5234267.660.26680.12940.9400
CIDER-FM-Obs−0.4578−0.5049435.190.20770.12320.9024
Graph4CFM, no graph3811.88−0.4510356.630.12990.07210.9200
Graph4CFM, Y ancestors3811.86−0.4570364.150.08690.06210.9183
Graph4CFM, full ancestry3811.86−0.4921324.640.21250.07520.9272

While, as expected, Graphs4CFM improves when we provide more graph information (full ancestry), CIDER-FM obtains lower MSE and higher mean, median, and pooled R2R^2 than Graph4CFM (even with the full graph) in all three information settings.

We also evaluated the new Graphs4CFM checkpoint on CausalChambers. All methods receive 192 context rows: 192 observational rows for Graph4CFM and CIDER-FM-Obs, and 96 observational plus 96 surrogate-interventional rows for CIDER-FM.

CausalChambers: nine acquisition blocks, three context draws per block

MethodMean NLL ↓Median NLL ↓MSE ↓Mean R2R^2 ↑Median R2R^2 ↑Pooled R2R^2 ↑
CIDER-FM5.76945.77866819.010.36580.37080.3734
CIDER-FM-Obs5.75935.73766954.170.35110.33570.3609
Graph4CFM, no graph5.84065.88606958.650.35040.36020.3605

CIDER-FM improves on Graph4CFM across the reported metrics in this real-world setting.

Accordingly, our principal CFM comparisons are CIDER-FM-Obs, which provides the matched architecture and training-budget control for the contribution of experimental context, and the newly trained Graph4CFM, which provides an external graph-conditioned CFM comparison under a matched natural prior. Tabular foundation models such as TabPFN serve as supplementary predictive baselines.

The novelty relative to existing foundation models using interventional context could be explained more clearly.

We added a new paragraph ("CFMs with experimental context") to related works (Section 2, pp. 3-4) where we explain that CIDER-FM is, to our knowledge, the first CFM that predicts conditional interventional distributions p(y∣do(t),x)p(y \mid do(t), \mathbf{x}) from a context that combines observational data with data from several experimental regimes, including surrogate experiments on variables other than the treatment, in tabular systems with hidden confounding.

More specifically:

  • Do-PFN, CausalPFN, CausalFM, ACTIVA, Graph4CFM, TabPFN-CFM and CCPFN condition on observational data only. MACE-TNP is trained on observational contexts. One two-node linear–Gaussian experiment additionally provides five samples from the queried intervention itself, which is a direct experiment on the treatment.
  • MapPFN and PerturbPFN, developed for single-cell perturbation prediction, do condition on several experimental regimes. They predict population-level expression responses to a held-out perturbation under priors in which all variables are observed.
  • Amortised causal discovery models such as TabCausal also take mixed observational–interventional tables as input, but output a graph.
AuthorsOpenReview ↗

Response to Reviewer duDL (2/3)

It remains unclear how much of the gain comes from the proposed three-way attention and if this is really the best way of using the interventional context.

We now report a controlled two-axis versus three-axis ablation in Appendix M (pp. 38-41), with closely matched parameter counts and the same pretraining and larger-context adaptation schedule. At 20 variables and 512 samples per regime, three-axis attention provides 1.41× and 3.75× higher training throughput with 5 and 19 regimes, respectively, although it offers no speedup with only 2 regimes. After adaptation, it also achieves a modest NLL improvement of 0.0130 (95% paired CI: [0.0037, 0.0229]), while differences in MSE and R2R^2 remain inconclusive. These results support the architecture’s computational advantage in larger multi-regime contexts, alongside a modest predictive benefit, but do not establish that it is universally optimal.

The repository has no README explaining how to reproduce results of the manuscript.

We apologize for forgetting to add the README and added a README with detailed instructions for reproducing our experiments.

Questions

Which parts of the theoretical results are specific to CIDER-FM?

To the very best of our knowledge, all theoretical results fo Causal Foundation models are with respect to a posterior (predictive), p(y∣do(t),x,D)p(y|do(t), \mathbf{x}, D) in our case, induced by the prior for the respective data-generating process, which the Causal Foundation Model learns to approximate. This includes the results in [1], [2], [3], and [4].

Our results are not different in that they are about the ''ground-truth'' causal posterior predictive p(y∣do(t),x,D)p(y|do(t), \mathbf{x}, D), and are thus independent of and do not account for the inference method used to approximate it.

We also moved the details on our theoretical arguments to the Appendix B to D (pp. 15-18) since we see them not as novel theoretical constributions, but rather as simple formalisations of the general principles behind what makes CIDER-FM work and included them mainly for the purpose of building intuition.

How does CIDER-FM relate to MapPFN (not cited), TabCausal (not cited), and PerturbPFN (cited but not discussed in detail)?

MapPFN focuses on generating multivariate post-perturbation distributions for biological systems under the assumption of no unobserved confounders. CIDER-FM instead directly predicts conditional interventional densities p(y∣do(T=t),x,D)p(y\mid do(T=t),\mathbf{x},\mathcal D), allowing hidden confounding and query-specific conditioning values. PerturbPFN addresses a complementary setting in which perturbation targets and strengths are unknown. It uses perturbation descriptors and observed responses to estimate a graph, targets, and strengths, then predicts responses through an SCM decoder with additive mechanism shifts. CIDER-FM assumes known intervention targets and assigned values, and directly approximates the predictive density without explicitly estimating a graph as an intermediate step.

TabCausal is a model that performs amortised causal discovery. Analogously to CIDER-FM, it is trained on mixed observational-interventional data, but outputs edge probabilities for causal graph instead of predicting a causal effect. TabCausal is also not trained with hidden confounding.

  • Could you include a comparison to a foundation model that can be considered fair?

As detailed in our response to W1, we use three complementary foundation-model comparisons. CIDER-FM-Obs provides the most closely controlled comparison, sharing CIDER-FM’s architecture, model size, and training budget, while using observational context only. One can see CIDER-FM-Obs as version of DoPFN [1] with continuous treatment that matches exactly our training setup and architecture.

We also train a custion Graph4CFM model [4] to provides an external CFM comparison under CIDER-FM’s ComplexMech natural SCM prior, evaluated on the same queries with matched total context sizes. Its model size and training budget differ from ours. TabPFN provides a supplementary comparison to a general-purpose predictive foundation model.

How do I reproduce the paper's results using the repo?

Please refer to our updated code-base together with the new README.

AuthorsOpenReview ↗

Response to Reviewer duDL (3/3)

Given some evidence that tabular foundation models can behave like implicit retrieval methods, how do you assess whether CIDER-FM learns transferable causal reasoning or mainly exploits similarity to its training prior and context?

We see, for example, the gains of CIDER-FM compared to the purely predictive TabPFN (which was trained with much more compute) on our benchmarks that pose carefully controlled causal challenges as strong evidence that CIDER-FM performs causal reasoning. We see the faxct that the model generalizes to the real-world Causal Chambers data as evidence that those capabilities are transferrable, even to real-world scenarios.

We believe that obtaining explicit mechanistic insights into CIDER-FM's behaviour, similar to emerging work in the context of predictive Tabular Foundation Models [5], is challenging yet interesting direction for future work.

References

[1] Robertson, Jake, et al. "Do-pfn: In-context learning for causal effect estimation." Neurips 2025.

[2] Balazadeh Meresht, Vahid, et al. "Causalpfn: Amortized causal effect estimation via in-context learning." Neurips 2025.

[3] Ma, Yuchen, et al. "Foundation models for causal inference via prior-data fitted networks." ICLR 2026.

[4] Reuter, Arik, et al. "Use what you know: Causal foundation models with partial graphs." ICML 2026.

[5] Balef, Amir, et al. "Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models". ICML 2026.

AuthorsOpenReview ↗

The Rebuttal Period Will Be Over Soon (Less Than 48 Hours Left)

Thank you again for your review!

We address your comments and questions in our answers above.

If you have further questions, we are very happy to answer them.

If everything is addressed, we would greatly appreciate you increasing your score.

Reviewer_Ve4NOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes CIDER-FM, a prior-fitted causal foundation model. It receives an observational table and tables from single-target surrogate experiments, where the intervened value is randomised row by row. It outputs a predictive density for p(y | do(T=t), x) without a causal graph.

Critical strengths
  1. The problem is closedly relevant with a field of causal AI.

  2. Clearly written.

Critical weaknesses

W1. The method doesn't distinguish non-identifiability. CIDER-FM outputs a single density for Y, and its spread mixes two sources: outcome noise, which remains even when the query is identified, and disagreement among compatible SCMs, which remains only when it is not (Var = E[Var(Y | ψ)] + Var(E[Y | ψ])). Under non-identification, this disagreement is weighted by the pretraining prior, so even with infinite data the prediction is set by the prior. A narrow output does not certify identification, and a wide output does not signal non-identification.

W2. The evidence that experimental context helps rests on a weak control, and the natural baselines and literature are excluded. (a) In Table 1 (linear-Gaussian, Y_mean), CIDER-FM-Obs has mean R² = -0.016 and median R² = -0.016, which corresponds to near-constant predictions. Graph4CFM without experiments reaches 0.474 and 0.811, and TabPFN reaches median R² = 0.897. The headline gap (MSE 95 versus 472) may therefore reflect an under-trained observational model. Where the control is competitive, the gains are small. In ComplexMech the control has better NLL (-0.416 versus -0.395). In fusion_non_id the Y_q R² values are nearly identical (Fig. 4a). On Causal Chambers the MSE is 6810 ± 399 versus 6861 ± 340, and the NLL is slightly worse.

(b) Graph-informed estimators are excluded from the rankings, and where they are shown they dominate in the identified case. In Table 8 (linear-Gaussian, fusion_id), c-gID with linear regression on the true ADMG attains Y_mean MSE 0.081 (R² 0.961), against 0.675 (R² 0.709) for CIDER-FM.

(c) The data-fusion literature is not cited beyond z-identification, g-identification and c-gID. For the identified case, estimators for effects identified from combinations of observational and experimental data exist (Jung et al., NeurIPS 2023; Jung et al., ICML 2023; Jung and Bellot, NeurIPS 2024) and are the appropriate graph-informed baselines. For the non-identified case, partial identification from observational and experimental data (Xia et al., NeurIPS 2021; Xia et al., ICLR 2023; Zhang et al., ICML 2022) gives the identified set against which W1 can be checked. The general data-fusion framework is also uncited (Bareinboim and Pearl, PNAS 2016; Lee et al., AAAI 2020; Correa et al., NeurIPS 2021).

Questions for authors
  1. On fusion_non_id, how does the prediction compare with the identified set, and how does it change under a different pretraining prior?
  2. Why is CIDER-FM-Obs near-constant on linear-Gaussian tasks (Table 1)?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Ve4N (1/3)

Thank you for your very helpful review!

We are happy to hear that you find the paper clearly written and relevant for the field of causal AI.

We adress your concerns and questions below and have updated the manuscript accordingly.

Weaknesses

W1: The method doesn't distinguish non-identifiability.

You have correctly pointed out that CIDER-FM does not allow for interpreting sources of uncertainty that may come from outcome noise, or uncertainty due to non-identification. An alternate approach that does do this is to first perform (Bayesian) causal discovery and then propagate the uncertainty about the graph and the mechanisms to the final prediction. However, [1] find that a neural network that directly amortises causal discovery and inference from observational data outperforms such two-stage Bayesian baselines. (The reason most likely is that the end-to-end model never has to represent the posterior over high-dimensional graphs explicitly).

Our aim in this work is to quantify this uncertainty regardless of if it comes from graph uncertainty or functional uncertainty / outcome noise. We would like to point out that one of the advantages of using neural networks is that multiple prediction heads read out quantities of interest. For example, it is possible to jointly output a posterior over causal graphs using the output head in [2]. Doing so, would output a posterior interventional distribution along with a posterior over graph, which can be used to see the uncertainty over causal graphs. We leave this for future work.

We now make clear in the conclusion of our paper, that, in case a causal graph is available, an estimator that uses it should be utilised.If any graph information is available.

More generally, we also see CIDER-FM a causal AI method using amortised inference for effect estimation. Instead of explicit graphical assumptions, it relies on implicit, regularising assumptions encoded in its prior (for example, independent causal mechanisms [3]), building upon ideas from the empirically very successful tabular foundation models [4]. As a consequence, CIDER-FM does not enjoy the theoretical guarantees of estimators that are carefully constructed for a specific identification setting. Instead, we rely on a broad synthetic evaluation and on real-world data, similar to [5].

W2 (a) The headline gap (MSE 95 versus 472) may therefore reflect an under-trained observational model.

Thank you for raising this concern. CIDER-FM and CIDER-FM-Obs use the same architecture, pretraining prior, optimization settings, and 100k-update training budget.

To investigate under-training, we continued both original linear–Gaussian models to 200k updates, keeping the architecture and prior fixed.

ModelCheckpointNLL ↓Causal-mean MSE ↓Mean R2R^2 ↑Median R2R^2 ↑
CIDER-FMOriginal 95k2.058394.580.67410.9469
CIDER-FMValidation-selected 150k2.0527101.250.68670.9464
CIDER-FM200k endpoint2.049178.050.70610.9465
CIDER-FM-ObsOriginal 100k2.4449470.93−0.0160−0.0164
CIDER-FM-ObsValidation-selected 140k2.4414483.73−0.0389−0.0162
CIDER-FM-Obs200k endpoint2.4392470.41−0.0127−0.0154

Even after 200k training steps, CIDER-FM improves, while CIDER-FM-Obs remains largely unchanged, so the fusion advantage persists after doubling the training budget. At the same time, the observational models keep improving on the ComplexMech priors.

We hypothesize that the failure of CIDER-FM-Obs on the linear-Gaussian setup is caused by a form of mode-collapse. Since with just observational data neither the causal graph nor the effect are identifiable, the model learns to always just predict values very close to zero.

AuthorsOpenReview ↗

Response to Reviewer Ve4N (2/3)

W2 (b) Graph-informed estimators are excluded from the rankings

The aim of this work is to propose a foundation model for causal effect estimation from diverse experimental regimes without access to a specific causal graph.

We exclude graph-based estimators from our ranking, because graph-informed estimators utilise a different type of information, namely the correct causal graph. In contrast, the posterior over the graph needs to be (implicitly) learned by CIDER-FM. We therefore moved the comparison to the graph-informed estimators to the Appendix.

As they use an additional form of rich causal infromation, we think that the graph-informed estimators outperforming CIDER-FM is very much expected. We see that this happens as a confirmation of the soundness of our benchmark and not as indicating any issue with our model. In general, we now make clear in the discussion section of our paper that in case reliable graphical information is available, it should be used.

However, in response to your comment, we report results for the subset of cases where graph-based baselines are applicable and also for cases where a two-stage method of causal discovery followed by effect estimation reports success.

Orcale-Graph baselines

To make this comparison feasible, we need to rescrict the SCMs we evaluate on to a subset of cases from where CIDER-FM is applicable; namely to cases where c-gID yields point-identifiable effects. Here, we find that CIDER-FM (even though it does not use the oracel graphical method, and even though we explicitly selected for cases wher c-gID yields point-identification) remains competetive:

Linear-Gaussian — Oracle Graph, 316 out of 560 original SCMs

MethodNLLMSEMean R2R^2Median R2R^2Pooled R2R^2
CIDER-FM1.764 ± 0.08158.34 ± 14.330.587 ± 0.0250.763 ± 0.0490.821 ± 0.054
CIDER-FM-Obs2.197 ± 0.075328.21 ± 168.780.004 ± 0.020−0.022 ± 0.004−0.005 ± 0.214
Bayesian LR16.677 ± 2.505918.77 ± 355.48−9.037 ± 2.7650.309 ± 0.101−1.815 ± 1.468
TabPFN v2∞280.92 ± 157.84−0.365 ± 0.1570.558 ± 0.0900.139 ± 0.259
c-gID–BLR + Oracle graph2.007 ± 0.47350.81 ± 12.460.411 ± 0.2110.822 ± 0.0390.844 ± 0.061

ComplexMech — Oracle Graph, 246 out of 512 original SCMs

MethodNLLMSEMean R2R^2Median R2R^2Pooled R2R^2
CIDER-FM−0.487 ± 0.116280.44 ± 136.020.271 ± 0.0260.135 ± 0.0400.798 ± 0.159
CIDER-FM-Obs−0.517 ± 0.117293.20 ± 132.570.247 ± 0.0330.133 ± 0.0330.788 ± 0.198
Bayesian LR2.293 ± 0.5851557.55 ± 981.51−1001.895 ± 987.929−0.056 ± 0.013−0.124 ± 0.905
TabPFN v2∞728.23 ± 535.720.050 ± 0.0670.047 ± 0.0360.474 ± 0.152
Graph4CFM 95k + Oracle graph−0.640 ± 0.125267.88 ± 128.960.208 ± 0.0300.085 ± 0.0260.807 ± 0.176
c-gID–TabPFN v2 + Oracle graph−0.780 ± 0.127328.79 ± 154.190.241 ± 0.0260.068 ± 0.0300.763 ± 0.154

Two-stage pipeline comparison

Alongside oracle-graph comparisons, we evaluate two-stage pipelines that first estimate graphical information using FoundCause [6] or FCI-JCI [7] and then apply Graph4CFM or c-gID-based estimators.

For this comparison, we also need to restrict the evaluation cases to something smaller than where CIDER-FM is applicable. More specifically, we only report those cases where the used causal discovery methods report a succesfully found graph and downstream estimation complete successfully. Even though we condition on the overall success of the baseline-methods, we find CIDER-FM to be competetive:

Linear-Gaussian — Causal Discovery, 159 out of 560 original SCMs

MethodNLLMSEMean R2R^2Median R2R^2Pooled R2R^2
CIDER-FM1.824 ± 0.11360.38 ± 18.020.579 ± 0.0330.716 ± 0.0790.790 ± 0.102
c-gID–BLR + FoundCause29.461 ± 27.68254.90 ± 15.480.569 ± 0.0350.773 ± 0.0580.809 ± 0.087
c-gID–BLR + FCI-JCI52.479 ± 36.024101.05 ± 22.75−0.002 ± 0.2850.279 ± 0.0950.648 ± 0.160

ComplexMech — Causal Discovery, 130 out of 512 original SCMs

MethodNLLMSEMean R2R^2Median R2R^2Pooled R2R^2
CIDER-FM−0.359 ± 0.168234.96 ± 127.150.253 ± 0.0290.106 ± 0.0480.756 ± 0.207
Graph4CFM 95k + FoundCause−0.515 ± 0.178341.84 ± 178.490.122 ± 0.0480.022 ± 0.0200.645 ± 0.326
Graph4CFM 95k + FCI-JCI−0.523 ± 0.178280.29 ± 155.300.169 ± 0.0290.022 ± 0.0190.709 ± 0.307
c-gID–TabPFN v2 + FoundCause−0.653 ± 0.185354.54 ± 197.920.224 ± 0.0300.051 ± 0.0360.632 ± 0.200
c-gID–TabPFN v2 + FCI-JCI−0.631 ± 0.186354.59 ± 197.920.205 ± 0.0300.036 ± 0.0300.632 ± 0.200
AuthorsOpenReview ↗

Response to Reviewer Ve4N (3/3)

W2. (c) The data-fusion literature is not cited beyond z-identification, g-identification and c-gID.

Thank you very much for these pointers. They are very helpful for positioning our work! We now discuss all of these papers in the revised manuscript, briefly in Section 2 ("Data fusion with known graphs") and in detail in Appendix F.

The major difference between these methods and CIDER-FM is that they take the causal graph as input, whereas CIDER-FM receives no graph and has to infer the causal structure from the data by using an amortised inference approach. The two approaches are therefore complementary. When a reliable graph is available, graph-informed methods use more information and should be preferred, which we now state in the limitations (Section 7).

A second difference concerns the data type. The estimator of Jung et al. (2023a) and the algorithms of Zhang et al. (2022) and Xia et al. (2021, 2023) are formulated or evaluated for discrete variables, while our benchmarks consist of continuous variables with continuous treatments. For this reason we use c-gID with regression-based estimators and the ground-truth graph as the graph-informed reference (see our response to W2(b)).

Questions

On fusion_non_id, how does the prediction compare with the identified set, and how does it change under a different pretraining prior?

In a fusion_non_id setting, CIDER-FM returns a single effect distribution. Alternatively, one can take a look at the set of effect distributions for different SCMs compatible with the data. We conducted a controlled case study to compare these objects and examine sensitivity to the pretrained model.

We extended Figure 2’s linear–Gaussian setup with hidden confounding. The resulting SCMs produce identical observational and do(Z)do(Z) distributions but different target distributions. Within this specified SCM class, the identified set at the selected intervention query is Q={N(0.512245+0.189984δ,  0.19+0.5775δ2):δ∈(−∞,+∞)}.\mathcal{Q} = \{\mathcal{N}(0.512245 + 0.189984\delta,\; 0.19 + 0.5775\delta^2) : \delta \in (-\infty,+\infty)\}. Thus, ambiguity remains even with population-level knowledge of both data sources. In this particular construction, do(Z)do(Z) does not further shrink the population identified set.

Here, we can analytically derived sharp bounds on the target distribution’s 5th and 95th percentiles. We then evaluated frozen LG- and ComplexMech-pretrained models on identical LG contexts, using 64 independent context draws, each containing 64 observational and 64 do(Z)do(Z) samples. We repeatedly draw those contexts and average the 0.05 and 0.95 quantiles those models output.

Model / ReferenceLower endpoint q0.05q_{0.05}Upper endpoint q0.95q_{0.95}Mean maximum CDF-bound violation ↓
Theoretical identified range(−∞,−0.1964](-\infty,-0.1964][1.2209,+∞)[1.2209,+\infty)—
Linear-Gaussian−0.53571.36360.00472
ComplexMech−0.71931.42190.02636

We find that the mean predicted interval endpoints fall within the theoretical identified ranges, although some individual predictive CDFs can cross the pointwise bounds.

Given identical LG contexts, the LG-pretrained model agrees more closely with the theoretical bounds, whereas the ComplexMech-pretrained model produces wider intervals.

A visualisation of the predictive distributions and theoretical CDF bounds is available here.

Why is CIDER-FM-Obs near-constant on linear-Gaussian tasks (Table 1)?

We find that this model suffers from a form of mode-collapse, indicated by the model producing predictions that are not very correlated with the true conditional mean and exhibit a high median prediction-to-target variance ratio of approximately 0.10.

Please also kindly refer to our repsonse to W2(a).

References

[1] Dhir, A. et al. "Estimating interventional distributions with uncertain causal graphs through meta-learning." NeurIPS 2025.

[2] Dhir, A., et al. "A meta-learning approach to bayesian causal discovery." ICML 2025.

[3] Schölkopf, B., et al. "On causal and anticausal learning." ICML 2012.

[4] Hollmann, N., et al. "Accurate predictions on small data with a tabular foundation model." Nature 637, 2025.

[5] Robertson, J., et al. "Do-PFN: In-context learning for causal effect estimation." NeurIPS 2025.

[6] Blöbaum, P., et al. “FoundCause: Causal Discovery with Latent Confounders from Observational Data.” arXiv:2606.17516, 2026.

[7] Mooij, J. M., et al. “Joint Causal Inference from Multiple Contexts.” JMLR 2020.

AuthorsOpenReview ↗

The Rebuttal Period Will Be Over Soon (Less Than 48 Hours Left)

Thank you again for your review!

We address your comments and questions in our answers above.

If you have further questions, we are very happy to answer them.

If everything is addressed, we would greatly appreciate you increasing your score.

Reviewer_yXQKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper explores an extension to existing work in causal foundation models (CFMs). These models are trained on synthetic data to predict distributions of interventional effects (CIDs) from observational data. Observational data alone, however, is often not enough to uniquely identify the causal model, leading to multiple candidates being considered within the CFM and producing a much wider CID distribution due to this ambiguity. One way to get additional information to help narrow down the causal structure options is to use experimental data. The propose CIDER-FM, a new CFM that is able to incorporate a mix of observational and interventional data from the same system, providing more specific CID estimates than is possible with observational data alone. CIDER-FM is able to benefit not just from interventional data that aligns with the target CID but also interventional data with interventions on other variables, since they can help reduce the space of possible causal structures. The authors demonstrate the effectiveness of CIDER-FM on multiple synthetic datasets and a single empirical one and find that CIDER-FM consistently outperforms other CFM or FM methods that aren't able to make use of interventional data.

Critical strengths

The core idea of this work is sound and well-presented. Combining limited experiments with observational data is a common and well-justified method to improve causal estimates, and the way the authors apply it here to the existing CFM literature seems well-justified and compelling. The experimental results cover a nice range of comparison methods, evaluation metrics, and dataset styles. The focus on surrogate experiments is especially interesting - opening up useable experiments beyond just interventions on the target of interest seems very useful in practice and isn't something I usually see discussed.

Critical weaknesses

I think the distribution of what was included in the main paper and what was relegated the appendix needs to be reworked.

  • The formalism on lines 207-232 seems like the kind of thing that would be perfect for an appendix. The idea that adding new information and constraints reduces the space of possible compatible SCMs (and that getting more information from the target domain can't increase error) is very intuitive, and I don't see many readers feeling skeptical enough at this point that they need a detailed proof in the main body of the paper.
  • In contrast, Section 4 is the formalism of our proposed CFM, and much of the details are brushed over and relegated to the appendix. This leads to the body of Section 4 referencing aspects of CIDER-FM that aren't explained in the text. For example, the second sentence refers to "each query row" and line 252 references "every table cell". This section continues to reference "individual cell value"s and "rows", concepts that are only meaningful if you've read Appendix D. If you want to leave these details for the appendix, you can't be relying on an understanding of the things covered there in order to make sense of the main paper's text.
  • Another part of the paper that could benefit from more space is the training method for CIDER-FM. The effectiveness of CIDER-FM seems to hinge significantly on the specifics of the synthetic data used for pretraining. However, nearly all of the details about this synthetic data are in the appendix. In particular, when looking at the results in Table 1, it would be nice to know how many interventional vs observational samples are provided to CIDER-FM. The description in Section 5.2 for how you set up your real-world experiments is also overly condensed, making it hard to follow and to understand the intent of the design.

You reference Figure 1 for the first time on line 78, and the caption of Figure 1 uses the acronym 'TFM'. You never explicitly define TFM, but I assume it's "Tabular Foundation Model", which first appears as a term on line 143.

On line 90, you state that CIDER-FM is "following the prior-data fitted network (PFN) paradigm of synthetic pretraining and in-context learning (Hollmann et al., 2025)". Without describing what that means, though, this statement doesn't really add any value where it is. This reads more like something that should be in the "Background and Related Work" section.

Questions for authors

I'm not following the second paragraph of Section 5.2, where you describe your real-world evaluation method. I understand that you have a simulator and set up mixes of observational and interventional datasets. And then you run them through a frozen-weights version of CIDER-FM to find out which combination has the best likelihood, which ended up being "observational data + interventions on green"? What is the purpose of doing that, vs just picking an arbitrary combination of observation and interventional data?

Did you try any experiments where the specifics of the mechanism in the evaluation data doesn't correspond to any of the mechanisms seen during pretraining?

What assumptions do you make about how the range of mechanisms, number of variables, observational and interventional mix, etc. relate to the domain of interest?

2 comments
AuthorsOpenReview ↗

Response to Reviewer yXQK

Thank you for your detailed review! We are happy to hear that you find that "Combining limited experiments with observational data is a common and well-justified method to improve causal estimates, and the way the authors apply it here to the existing CFM literature seems well-justified and compelling."

We adress your concerns and questions below and have updated the manuscript accordingly.

Weaknesses

The formalism on lines 207-232 seems like the kind of thing that would be perfect for an appendix.

We agree and following your suggestion, and we have now shortened this section and moved most of it to the Appendix B and C (p. 15). Our motivation behind this part is to provide intuition, especially for readers less familar with data fusion and the Bayesian ideas underlying Causal Foundation Models.

[...] Section 4 is the formalism of our proposed CFM, and much of the details are brushed over [...]

Thank you for pointing this out. We have revised Section 4 (p. 5) in the udpated version of the manuscript to be more understandable and contain the important details, especially regarding the points referred to later.

Another part of the paper that could benefit from more space is the training method for CIDER-FM.

Thanks! This is another good idea and we have now included a desription of our prior in Section 5 (p.6), including its assumptions in the main part of the paper. We also provide full details in Appendix E.5 (p. 24).

You never explicitly define TFM

Thanks! We have fixed writing that TFM stands for "Tabular Foundation Model".

On line 90, you state that CIDER-FM is "following the prior-data fitted network (PFN) paradigm [...] This reads more like something that should be in the "Background and Related Work" section.

This is a good idea; we moved this sentence to the related work section (p. 3).

Questions

I'm not following the second paragraph of Section 5.2, where you describe your real-world evaluation method.

The purpose of this simulation-based experimental design is to demonstrate how a simulator can guide the choice of surrogate experiments under a limited data budget. Different interventions may provide different amounts of useful information for the target query, so an arbitrary mixture may allocate samples to less informative regimes. We compare candidate mixtures with the same total context size using predictive NLL on simulated query outcomes, then fix the selected mixture before evaluating it on real measurements. The model remains frozen throughout.

We have slightly improved the wording of this section in the paper (pp. 9-10) and now also refer to Appendix K (p. 36) where the procedure is explained in more detail.

Did you try any experiments where the specifics of the mechanism in the evaluation data doesn't correspond to any of the mechanisms seen during pretraining?

Yes. We now report out-of-distribution evaluations in Appendix J (pp. 34-36), using a frozen ComplexMech-pretrained CIDER-FM without further training on the shifted distributions. We test Softsign activations and additive spline mechanisms absent from pretraining, alongside unseen uniform and exponential noise distributions.

The results show that CIDER-FM can generalise beyond the specific mechanisms and noise distributions used during pretraining. The model achieves, for instance, a pooled R2R^2 of 0.702 on unseen Softsign mechanisms and 0.849 under unseen exponential noise. It also retains useful point-prediction performance on the Linear–Gaussian case.

What assumptions do you make about how the range of mechanisms, number of variables, observational and interventional mix, etc. relate to the domain of interest?

Thank you for this question. We have added Section 5 (p. 6) to describe the synthetic prior and its assumptions. The general-graph models cover 4–10 observed continuous variables and possible latent confounding, with separate models for linear–Gaussian and ComplexMech nonlinear MLP mechanisms. For ComplexMech, the number of single-target interventional regimes follows a truncated geometric distribution, with an optional observational regime and 64 samples per active regime.

We also describe an extended model in Appendix I, obtained through 20k additional training updates under a prior covering 4–20 variables and 64–512 samples per regime.

AuthorsOpenReview ↗

The Rebuttal Period Will Be Over Soon (Less Than 48 Hours Left)

Thank you again for your review!

We address your comments and questions in our answers above.

If you have further questions, we are very happy to answer them.

If everything is addressed, we would greatly appreciate you increasing your score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Our Responses

We sincerely thank all reviewers for their helpful comments and questions! We summarise below the main additions and changes made in response to the reviews. Changes to the manuscript are marked in blue.

1. Stronger Baseline Comparisons (Reviewers Ve4N, duDL)

  • We trained a custom version of Graph4CFM, a recently proposed Causal Foundation Model that takes graphical information as input, on our ComplexMech prior. On the full 512 ComplexMech test SCMs, CIDER-FM outperforms this retrained Graph4CFM on all reported metrics in all three graph-information settings (for example, MSE 267.66 vs. 324.64 with full ancestry).
  • On CausalChambers, CIDER-FM is slightly better than the retrained Graph4CFM.
  • We added further c-gID-based baselines, using Bayesian linear regression for Linear–Gaussian tasks and TabPFN v2 for ComplexMech tasks, and we evaluate graph-based methods (c-gID and Graph4CFM) with both oracle graphs and graphs discovered using FoundCause or FCI-JCI. On the subsets where these methods apply, CIDER-FM remains competitive without access to a graph.

2. Generalisation under Prior Mismatch and Larger Contexts (Reviewers yXQK, duDL)

We now evaluate the frozen ComplexMech-pretrained model on mechanisms and noise distributions absent from pretraining (Appendix J, pp. 34-36). It reaches a pooled R2R^2 of 0.702 on Softsign mechanisms and 0.849 under exponential noise, compared to 0.940 on the in-distribution ComplexMech test set. Separately, we describe an extended model trained for 20k additional updates under a prior covering 4–20 variables and 64–512 samples per regime (Appendix I).

3. Non-identifiability and Predictive Uncertainty (Reviewer Ve4N)

We clarify that CIDER-FM predicts a posterior-predictive distribution by integrating over plausible SCMs that could explain the available data, thereby mixing uncertainty due to unidentifiability with outcome noise. To illustrate this, we provide a controlled non-identifiable case study with analytically derived identified ranges and CDF bounds.

4. Scaling Up the Observational Baseline-Version of CIDER-FM (Reviewer Ve4N)

We doubled the training budget of both Linear–Gaussian models from 100k to 200k updates, keeping their architecture and prior fixed. CIDER-FM’s mean R2R^2 rises from 0.674 (original 95k checkpoint) to 0.706 (200k endpoint), while CIDER-FM-Obs stays near zero (−0.016 and −0.013). We hypothesize that CIDER-FM-Obs suffers from a form of mode collapse on this prior.

We would also like to point out that we compare to further competetive baselines. On the same Linear–Gaussian benchmark (Table 1), CIDER-FM reaches a median R2R^2 of 0.947, compared to 0.897 for TabPFN and 0.811 for Graph4CFM without experiments. On the c-gID-identifiable subset, CIDER-FM reaches a median R2R^2 of 0.763, compared to 0.558 for TabPFN v2.

5. Architectural Ablation (Reviewer duDL)

We compare two-axis and three-axis models with closely matched parameter counts and identical training schedules (Appendix M, pp. 38-41). At 20 variables and 512 samples per regime, three-axis attention provides 1.41× and 3.75× higher training throughput with 5 and 19 regimes, respectively.

6. Presentation, Related Work, and Details on Experiments (Reviewers yXQK, Ve4N, duDL)

We expanded the method description in Section 4 (p. 5), added the synthetic prior and its assumptions in Section 5 (p. 6), and provided training details in Appendix E.5 (p. 24). We clarified the CausalChambers evaluation protocol, moved detailed theoretical arguments to Appendices B–D (pp. 15-18), and expanded discussion of data-fusion methods, MapPFN, PerturbPFN, and TabCausal. We also added a README with instructions for reproducing the experiments.

We are very happy to answer any further questions!

AuthorsOpenReview ↗

Response to Reviewer duDL's follow-up

Thank you for recognising that the new version has clearly improve!

We agree that predictive performance is similar. We see the main benefits of the 3 axis attention in terms of improved scaling to larger contexts, where we find at 20 variables and 512 samples per regime, training throughput is 1.41× and 3.75× higher with 5 and 19 regimes, respectively (Appendix M).

In general, to our very best knowledge no other perfectly matched Causal Foundation Model baseline exists. We therefore trained our own versions of Graphs4CFM and the observational-only baseline (basically DoPFN) on our prior, while acknowledging that Graph4CFM still differs in model size and training budget.

Prior dependence is a broader challenge for tabular and causal foundation models. Our frozen-model results on CausalChambers provide evidence of transfer to real experimental data beyond the synthetic prior, complemented by our evaluations on unseen mechanisms and noise distributions.

Thank you again for your thoughtful feedback; we are happy to address any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.