- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper concerns causal discovery foundation models. There are two contributions (1) a theoretical analysis of the behaviour of pre-training and posterior concentration, and (2) a new Causal Discovery Foundation Model that separates learning the skeleton of the DAG from learning the causal ordering. The proposed Causal Discovery Foundation model is evaluated on a diverse synthetic simulator and the Sachs dataset.
Critical strengths
I find the theoretical analysis interesting, meaningful and reassuring. Most stated claims are straightforward and it is not surprising that they are true, but it is still good to know and to have them formulated that way.
The proposed diffusion loss for learning the causal ordering seems meaningful and the performance of the proposed method appears to be quite good (under the caveats under "critical limitations").
Critical weaknesses
My main concern is that the paper contains two seemingly completely separate things: The theoretical analysis (contribution (1)) and (2) a new causal discovery foundation model. The two seem very disconnected. This makes the overall contribution of the paper unclear. The contributions of (1) and (2) individually are interesting, but would not be sufficient for acceptance. Just concatenating (1) and (2) does not change this fundamentally.
In terms of evaluation, of the new causal discovery foundation model, it would have been better to consider an existing causal discovery benchmark (e.g. [1]) and more real-world dataset (e.g. from CausalChambers [2]) to fully avoid leakage between pre-training and testing distribution. Alternatively, the synthetic simulator in [3] that has been at least justified in a predictive setting could be a further alternative.
This issue in the evaluation makes me less convinced by the performance of the proposed method in (2).
[1] Göbler, Konstantin, et al. "$\texttt {causalAssembly} $: Generating Realistic Production Data for Benchmarking Causal Discovery." Causal Learning and Reasoning. PMLR, 2024.
[2] Gamella, Juan L., Jonas Peters, and Peter Bühlmann. "Causal chambers as a real-world physical testbed for AI methodology." Nature Machine Intelligence 7.1 (2025): 107-118.
[3] Reuter, Arik, et al. "Use What You Know: Causal Foundation Models with Partial Graphs." ICML 2026.
Questions for authors
What is the connection between the presented theoretical analysis (contribution (1)) and the proposed model (2)?
What is the performance of the proposed model on external benchmarks/simulators/real-world data?
4 comments
Response to W2 and Q2
Thank you for suggesting additional external evaluations. Following your suggestion, we conducted experiments on all three recommended data sources: the SCM simulator released by Reuter et al. [3] (Graphs4CFM), the real-world physical dataset Causal Chambers [2], and the industrial semisynthetic benchmark causalAssembly [1].
We first describe how we constructed the test datasets from each source.
Graphs4CFM. Reuter et al. [3] primarily study how full or partial knowledge of the causal graph can improve causal effect estimation. For their synthetic evaluation, the authors use the same prior distribution as in training in place of a separately constructed causal discovery benchmark. We use their publicly available SCM simulator to evaluate our model's ability to recover causal graphs from observational data under this external prior. We retain the authors' default priors over graph structures, causal mechanisms, and noise, and set the number of variables to and the sample size to . For each of the 24 combinations, we independently sample 100 SCMs, yielding 2,400 test datasets, each with a newly sampled graph and causal mechanisms. Our model receives only observational data, without any background graph knowledge.
Causal Chambers. We use the same 20 variables as Gamella et al. [2] in their light-tunnel causal discovery experiment. The 11 control variables take discrete settings, and the 9 sensor variables record quantized measurements of continuous physical quantities. We randomly shuffle the 10,000 measurements from the light-tunnel observational experiment once. We then take the first 100, 500, 1,000, and 2,000 measurements in that order to form four nested test sets.
causalAssembly. We use the three stations in this benchmark with at most 20 variables. Stations 1, 3, and 5 contain 6, 16, and 16 variables, respectively. Following the official station-level procedure, we fit distributional random forests (DRFs) once per station using the publicly released semisynthetic reference data and the corresponding station DAG. We then generate new observational datasets at .
The SHD results for the additional experiments are as follows.
| Method | Graphs4CFM | Causal Chambers (discrete controls and quantized observations) |
|---|---|---|
| Ours | 24.68 | 40.75 |
| TabCausal | 22.23 | 25.25 |
| AVICI | 23.83 | 105.50 |
| Arrow | 24.89 | 43.00 |
| FoundCause | 25.45 | 39.25 |
| CauScale | 26.38 | 35.50 |
| CDFM | 29.90 | 21.00 |
For causalAssembly, the mean SHD for each station, averaged over the six sample sizes and their replicates, is as follows.
| Method | Station 1 (6 variables) | Station 3 (16 variables) | Station 5 (16 variables) |
|---|---|---|---|
| Ours | 5.928 | 12.237 | 13.117 |
| TabCausal | 6.585 | 6.777 | 14.787 |
| AVICI | 7.047 | 18.400 | 17.230 |
| Arrow | 6.798 | 7.873 | 10.163 |
| FoundCause | 6.428 | 15.743 | 23.857 |
| CauScale | 7.028 | 14.273 | 15.403 |
| CDFM | 9.052 | 25.100 | 26.027 |
Among the seven methods, our method achieves the lowest SHD on Station 1, ranks third on Station 3, and ranks second on Station 5.
The performance of current causal discovery foundation models depends on the choice of pretraining distribution and the amount of pretraining data. In particular, discrete settings and quantized observations are prevalent in Causal Chambers. Of the 20 variables, 11 are controls whose brightness and angle settings take discrete values. The other nine are sensors that measure continuous physical quantities but produce quantized readings. These characteristics differ substantially from those of our pretraining distribution, which is based on continuous mechanisms.
Our main contribution is a statistical framework for analyzing and explaining existing causal discovery foundation models, together with an approach derived from this framework for improving model expressiveness. Multiple experiments in our paper demonstrate the value of the framework in guiding model design and improving expressiveness.
Thank you for your response!
Re: > Response to W2 and Q2
However, your reply does not fully convince me of a real connection between your theoretic analysis and the proposed model. And you also do not mention that/how you would like to make this connection clearer in the updated manuscript. This remains a major concern in terms of presentation for me.
Re: > Response to W2 and Q2
I appreciate your honest evaluation on all proposed benchmarks. (Thanks for this!), They do, however, not really address my concern because there seems to be a simple explanation for the mediocre performance of the method, which is that it was never trained on discrete variables.
I generally believe that there are interesting methodological and theoretical things in this paper in form of the theoretical analysis and the proposed method.
I just stay being confused by putting them together in this paper in the current form and stay concerned by the empirical evaluation of the proposed method.
(In my mind, there is potential for two meaningful papers in this one submission: one regarding the theory (which would then also need bespoke experiments); and one paper regarding the diffusion approach for learning the causal order, which would also need a more thorough empirical evaluation and different motivation).
Thank you for your response.
Regarding the external benchmarks, our goal is to provide a theoretical explanation, clarify the applicability of our theory, and illustrate the architectural advantages of our model, rather than to develop a large model pretrained on vast amounts of data covering a broad range of distributions. Such broad distributional coverage and large-scale pretraining are important factors in performance on real-world benchmarks.
Thank you also for suggesting that we explain the connection between our method and theory more clearly in the manuscript. we have added paragraphs to the Introduction, under “Implications for existing models,” and at the beginning of the method section to explain our motivation and the connection between the theory and the model. We have marked the changes in blue in the main text.
We chose to present the theory and the method in a single paper because, while it is important to use the theory to explain existing models and to test the theory itself, we also wanted to study how it can guide model design and application in practice. We therefore developed our model based on the assumptions underlying our theory. Our aim is to explain, in one paper, the theoretical properties of foundation models, the requirements that follow from the theory, and how these requirements inform the design of both the model and the pretraining distribution.
We further explain below how the theory motivates our model design.
We chose to present the theory and the method in a single paper because, while it is important to use the theory to explain existing models and to test the theory itself, we also wanted to study how it can guide model design and application in practice. We therefore developed our model based on the assumptions underlying our theory. Our aim is to explain, in one paper, the theoretical properties of foundation models, the requirements that follow from the theory, and how these requirements inform the design of both the model and the pretraining distribution.
First, the model needs sufficient expressive capacity. Existing models make simplifying choices in how they represent graphs. For example, TabCausal does not directly enforce acyclicity, so its predictions may contain cycles. Other models, such as CSIvA, generate edges sequentially. This approach is expressive, but as the number of nodes grows, errors may accumulate during generation and training becomes more difficult, which may limit performance in practice. Another approach is to decompose a DAG into a skeleton and a topological order, using a relatively simple model for the distribution over orders. For example, Arrow uses a single Plackett–Luce (PL) distribution to model topological orders. A single PL distribution has limited capacity to approximate complex posteriors over topological orders.
To increase expressive capacity while keeping the model practical to train, we propose using a diffusion model to learn the distribution over topological orders.
Second, the pretraining distribution must also satisfy certain conditions. The unbounded parameter priors used in existing causal discovery foundation methods do not guarantee large-sample convergence, even with sufficient pretraining. We therefore specify the relevant conditions and restrict the parameters to closed and bounded sets when constructing our pretraining distribution. At our current training scale, restricting the parameters to closed and bounded sets have little effect in practice. However, this condition is important when training at a substantially larger scale.
We hope that these clarifications and revisions address your concerns. We will further revise the manuscript to address any additional comments.
Response to W1and Q1
Thank you for raising this question. Our theoretical analysis directly informs the design of both the proposed model architecture and its pretraining distribution. Our main contribution is a unified statistical framework that explains why existing causal discovery foundation models can learn from synthetic tasks and what limits their ability to recover causal structure. The analysis of posterior learning guides how we represent uncertainty over topological orders, and the analysis of posterior concentration guides our construction of the pretraining prior.
Posterior learning and model architecture. For a fixed number of observations per dataset, we decompose the expected posterior KL error into approximation error and fitting error. Under the stated conditions, as the number of pretraining tasks grows and model capacity increases at a suitable rate, the learned graph distribution approaches the graph posterior induced by the pretraining prior. If the chosen family of order distributions cannot represent well-separated modes, increasing the number of pretraining tasks alone cannot eliminate the resulting approximation error. We therefore use conditional diffusion to model topological orders and evaluate its ability to fit multimodal order distributions in Section 5.1. We have added a controlled experiment to show that our proposed method improves the model’s ability to approximate the graph posterior (Section 5.4 and Appendix J.4).
Posterior concentration and prior design. Under the stated conditions, as the number of observations grows, the prior-induced graph posterior concentrates on the KL-best-fitting graph set. Under correct specification and graph identifiability within the support of the pretraining prior, this set contains only the true DAG. The regularity conditions for posterior concentration motivate our choice to restrict the simulator parameters to compact sets. When the parameter-to-density map is Hellinger continuous, compact parameter support ensures that the density family for each graph is compact in Hellinger distance, as required by the concentration theorem. The controlled We have added a LiNGAM experiment in Section 5.3 and Appendix J.3.2 to provide empirical support for the posterior concentration predicted by our theory.