acceptodds
Under review as a conference paper at ICLR 2027

What Do Causal Discovery Foundation Models Learn?

Abstract

Recovering a unique causal structure from observational data is difficult due to finite data and non-identifiability. Recent causal discovery foundation models pretrain on synthetic causal data and directly infer structures for new datasets, avoiding repeated structure search. Yet when the causal structure is not identifiable or the true data-generating distribution lies outside the distribution family used for pretraining, what these models learn and which structures they can recover remain unclear. To answer these questions, we develop a statistical framework that characterizes both the graph distribution learned through pretraining and its large-sample recovery target. We prove that, as the number of pretraining tasks and model capacity increase, the learned graph distribution approaches the posterior over graphs induced by the pretraining prior. As the sample size increases, this posterior concentrates on the KL-best-fitting graph set under model misspecification. Under correct specification and the Markov and faithfulness assumptions, it concentrates on the Markov equivalence class of the true DAG; under additional identifiability assumptions, it concentrates on the true DAG. These results provide a unified perspective for understanding existing models and guide the design of pretraining distributions and model architectures. As a concrete instantiation, we factorize DAGs into skeletons and topological orders, use conditional diffusion to model multimodal distributions over topological orders, and orient skeletons by sampled orders to guarantee acyclicity. Across diverse synthetic benchmarks, our method achieves stronger overall structure recovery than traditional methods and other causal discovery foundation models.

Primary area:
causal reasoning
Keywords:
Causal Discovery, Foundation Models, Posterior Concentration, Model Misspecification, Diffusion Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4qfkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper concerns causal discovery foundation models. There are two contributions (1) a theoretical analysis of the behaviour of pre-training and posterior concentration, and (2) a new Causal Discovery Foundation Model that separates learning the skeleton of the DAG from learning the causal ordering. The proposed Causal Discovery Foundation model is evaluated on a diverse synthetic simulator and the Sachs dataset.

Critical strengths

I find the theoretical analysis interesting, meaningful and reassuring. Most stated claims are straightforward and it is not surprising that they are true, but it is still good to know and to have them formulated that way.

The proposed diffusion loss for learning the causal ordering seems meaningful and the performance of the proposed method appears to be quite good (under the caveats under "critical limitations").

Critical weaknesses

My main concern is that the paper contains two seemingly completely separate things: The theoretical analysis (contribution (1)) and (2) a new causal discovery foundation model. The two seem very disconnected. This makes the overall contribution of the paper unclear. The contributions of (1) and (2) individually are interesting, but would not be sufficient for acceptance. Just concatenating (1) and (2) does not change this fundamentally.

In terms of evaluation, of the new causal discovery foundation model, it would have been better to consider an existing causal discovery benchmark (e.g. [1]) and more real-world dataset (e.g. from CausalChambers [2]) to fully avoid leakage between pre-training and testing distribution. Alternatively, the synthetic simulator in [3] that has been at least justified in a predictive setting could be a further alternative.

This issue in the evaluation makes me less convinced by the performance of the proposed method in (2).

[1] Göbler, Konstantin, et al. "$\texttt {causalAssembly} $: Generating Realistic Production Data for Benchmarking Causal Discovery." Causal Learning and Reasoning. PMLR, 2024.

[2] Gamella, Juan L., Jonas Peters, and Peter Bühlmann. "Causal chambers as a real-world physical testbed for AI methodology." Nature Machine Intelligence 7.1 (2025): 107-118.

[3] Reuter, Arik, et al. "Use What You Know: Causal Foundation Models with Partial Graphs." ICML 2026.

Questions for authors

What is the connection between the presented theoretical analysis (contribution (1)) and the proposed model (2)?

What is the performance of the proposed model on external benchmarks/simulators/real-world data?

4 comments

Response to W1and Q1

Thank you for raising this question. Our theoretical analysis directly informs the design of both the proposed model architecture and its pretraining distribution. Our main contribution is a unified statistical framework that explains why existing causal discovery foundation models can learn from synthetic tasks and what limits their ability to recover causal structure. The analysis of posterior learning guides how we represent uncertainty over topological orders, and the analysis of posterior concentration guides our construction of the pretraining prior.

Posterior learning and model architecture. For a fixed number of observations per dataset, we decompose the expected posterior KL error into approximation error and fitting error. Under the stated conditions, as the number of pretraining tasks grows and model capacity increases at a suitable rate, the learned graph distribution approaches the graph posterior induced by the pretraining prior. If the chosen family of order distributions cannot represent well-separated modes, increasing the number of pretraining tasks alone cannot eliminate the resulting approximation error. We therefore use conditional diffusion to model topological orders and evaluate its ability to fit multimodal order distributions in Section 5.1. We have added a controlled experiment to show that our proposed method improves the model’s ability to approximate the graph posterior (Section 5.4 and Appendix J.4).

Posterior concentration and prior design. Under the stated conditions, as the number of observations grows, the prior-induced graph posterior concentrates on the KL-best-fitting graph set. Under correct specification and graph identifiability within the support of the pretraining prior, this set contains only the true DAG. The regularity conditions for posterior concentration motivate our choice to restrict the simulator parameters to compact sets. When the parameter-to-density map is Hellinger continuous, compact parameter support ensures that the density family for each graph is compact in Hellinger distance, as required by the concentration theorem. The controlled We have added a LiNGAM experiment in Section 5.3 and Appendix J.3.2 to provide empirical support for the posterior concentration predicted by our theory.

Response to W2 and Q2

Thank you for suggesting additional external evaluations. Following your suggestion, we conducted experiments on all three recommended data sources: the SCM simulator released by Reuter et al. [3] (Graphs4CFM), the real-world physical dataset Causal Chambers [2], and the industrial semisynthetic benchmark causalAssembly [1].

We first describe how we constructed the test datasets from each source.

Graphs4CFM. Reuter et al. [3] primarily study how full or partial knowledge of the causal graph can improve causal effect estimation. For their synthetic evaluation, the authors use the same prior distribution as in training in place of a separately constructed causal discovery benchmark. We use their publicly available SCM simulator to evaluate our model's ability to recover causal graphs from observational data under this external prior. We retain the authors' default priors over graph structures, causal mechanisms, and noise, and set the number of variables to d∈{3,5,10,20}d \in \{3,5,10,20\} and the sample size to n∈{20,50,100,300,1000,2000}n \in \{20,50,100,300,1000,2000\}. For each of the 24 combinations, we independently sample 100 SCMs, yielding 2,400 test datasets, each with a newly sampled graph and causal mechanisms. Our model receives only observational data, without any background graph knowledge.

Causal Chambers. We use the same 20 variables as Gamella et al. [2] in their light-tunnel causal discovery experiment. The 11 control variables take discrete settings, and the 9 sensor variables record quantized measurements of continuous physical quantities. We randomly shuffle the 10,000 measurements from the light-tunnel observational experiment once. We then take the first 100, 500, 1,000, and 2,000 measurements in that order to form four nested test sets.

causalAssembly. We use the three stations in this benchmark with at most 20 variables. Stations 1, 3, and 5 contain 6, 16, and 16 variables, respectively. Following the official station-level procedure, we fit distributional random forests (DRFs) once per station using the publicly released semisynthetic reference data and the corresponding station DAG. We then generate new observational datasets at n∈{20,50,100,300,1000,2000}n \in \{20,50,100,300,1000,2000\}.

The SHD results for the additional experiments are as follows.

MethodGraphs4CFMCausal Chambers (discrete controls and quantized observations)
Ours24.6840.75
TabCausal22.2325.25
AVICI23.83105.50
Arrow24.8943.00
FoundCause25.4539.25
CauScale26.3835.50
CDFM29.9021.00

For causalAssembly, the mean SHD for each station, averaged over the six sample sizes and their replicates, is as follows.

MethodStation 1 (6 variables)Station 3 (16 variables)Station 5 (16 variables)
Ours5.92812.23713.117
TabCausal6.5856.77714.787
AVICI7.04718.40017.230
Arrow6.7987.87310.163
FoundCause6.42815.74323.857
CauScale7.02814.27315.403
CDFM9.05225.10026.027

Among the seven methods, our method achieves the lowest SHD on Station 1, ranks third on Station 3, and ranks second on Station 5.

The performance of current causal discovery foundation models depends on the choice of pretraining distribution and the amount of pretraining data. In particular, discrete settings and quantized observations are prevalent in Causal Chambers. Of the 20 variables, 11 are controls whose brightness and angle settings take discrete values. The other nine are sensors that measure continuous physical quantities but produce quantized readings. These characteristics differ substantially from those of our pretraining distribution, which is based on continuous mechanisms.

Our main contribution is a statistical framework for analyzing and explaining existing causal discovery foundation models, together with an approach derived from this framework for improving model expressiveness. Multiple experiments in our paper demonstrate the value of the framework in guiding model design and improving expressiveness.

Thank you for your response!

Re: > Response to W2 and Q2

However, your reply does not fully convince me of a real connection between your theoretic analysis and the proposed model. And you also do not mention that/how you would like to make this connection clearer in the updated manuscript. This remains a major concern in terms of presentation for me.

Re: > Response to W2 and Q2

I appreciate your honest evaluation on all proposed benchmarks. (Thanks for this!), They do, however, not really address my concern because there seems to be a simple explanation for the mediocre performance of the method, which is that it was never trained on discrete variables.

I generally believe that there are interesting methodological and theoretical things in this paper in form of the theoretical analysis and the proposed method.

I just stay being confused by putting them together in this paper in the current form and stay concerned by the empirical evaluation of the proposed method.

(In my mind, there is potential for two meaningful papers in this one submission: one regarding the theory (which would then also need bespoke experiments); and one paper regarding the diffusion approach for learning the causal order, which would also need a more thorough empirical evaluation and different motivation).

Thank you for your response.

Regarding the external benchmarks, our goal is to provide a theoretical explanation, clarify the applicability of our theory, and illustrate the architectural advantages of our model, rather than to develop a large model pretrained on vast amounts of data covering a broad range of distributions. Such broad distributional coverage and large-scale pretraining are important factors in performance on real-world benchmarks.

Thank you also for suggesting that we explain the connection between our method and theory more clearly in the manuscript. we have added paragraphs to the Introduction, under “Implications for existing models,” and at the beginning of the method section to explain our motivation and the connection between the theory and the model. We have marked the changes in blue in the main text.

We chose to present the theory and the method in a single paper because, while it is important to use the theory to explain existing models and to test the theory itself, we also wanted to study how it can guide model design and application in practice. We therefore developed our model based on the assumptions underlying our theory. Our aim is to explain, in one paper, the theoretical properties of foundation models, the requirements that follow from the theory, and how these requirements inform the design of both the model and the pretraining distribution.

We further explain below how the theory motivates our model design.

We chose to present the theory and the method in a single paper because, while it is important to use the theory to explain existing models and to test the theory itself, we also wanted to study how it can guide model design and application in practice. We therefore developed our model based on the assumptions underlying our theory. Our aim is to explain, in one paper, the theoretical properties of foundation models, the requirements that follow from the theory, and how these requirements inform the design of both the model and the pretraining distribution.

First, the model needs sufficient expressive capacity. Existing models make simplifying choices in how they represent graphs. For example, TabCausal does not directly enforce acyclicity, so its predictions may contain cycles. Other models, such as CSIvA, generate edges sequentially. This approach is expressive, but as the number of nodes grows, errors may accumulate during generation and training becomes more difficult, which may limit performance in practice. Another approach is to decompose a DAG into a skeleton and a topological order, using a relatively simple model for the distribution over orders. For example, Arrow uses a single Plackett–Luce (PL) distribution to model topological orders. A single PL distribution has limited capacity to approximate complex posteriors over topological orders.

To increase expressive capacity while keeping the model practical to train, we propose using a diffusion model to learn the distribution over topological orders.

Second, the pretraining distribution must also satisfy certain conditions. The unbounded parameter priors used in existing causal discovery foundation methods do not guarantee large-sample convergence, even with sufficient pretraining. We therefore specify the relevant conditions and restrict the parameters to closed and bounded sets when constructing our pretraining distribution. At our current training scale, restricting the parameters to closed and bounded sets have little effect in practice. However, this condition is important when training at a substantially larger scale.

We hope that these clarifications and revisions address your concerns. We will further revise the manuscript to address any additional comments.

Reviewer_XzrqOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper develops a statistical framework for causal discovery foundation models that separates approximation of the graph posterior induced by a synthetic pretraining prior from the large-sample causal target of that posterior. Under stated assumptions, it establishes posterior approximation as pretraining data and model capacity increase, and concentration on a KL-best-fitting graph set as the observational sample size increases, with specializations to Markov equivalence classes and uniquely identifiable DAGs under additional conditions. The paper also proposes a model combining an independent-Bernoulli skeleton distribution with conditional diffusion over topological orders, guaranteeing acyclic graph samples. The authors test proposed method against baselines on comprehensive synthetic datasets across different settings. They also include an identifiable LiNGAM experiment. The results show improved graph recovery performance.

Critical strengths
  1. The statistical framework distinguishes simulator assumptions from causal identification. Separating posterior approximation from posterior concentration is useful. The framework explains why increasing pretraining scale cannot repair an unsuitable simulator family, and why greater encoder capacity cannot overcome a restrictive output distribution.

  2. The proposed architecture has plausible benefits and empirical support. It guarantees acyclicity and provides a flexible distribution over orders. The controlled permutation experiment demonstrates that multiple GPL diffusion steps can represent distributions that the tested simpler parameterizations struggle to fit. The main benchmark is also comprehensive with diverse settings.

Critical weaknesses
  1. The posterior-learning guarantees are not established for the proposed model and training objective. The authors acknowledge the factorization limitation in the conclusion, but still the applicability is not established or tested.

  2. The experiments establish graph-recovery performance more convincingly than the claimed posterior-modeling contribution. The learned posterior is never evaluated. There is no ablation under a shared prior. I think some results would helpful at least for small graphs and conjugate priors where the exact posterior can be enumerated. This can also support Thm 3.1 and Thm 3.2.

  3. The LiNGAM experiment does not test the theory's prediction. Any consistent estimator would show F1 rising with nn, while q(G0∣X)q(G_0|X) should be measured when it goes beyond training nn.

Questions for authors
  1. Can the authors provide a controlled comparison against the simulator-induced graph posterior, including a non-identifiable example on small graphs? This can help support whether improved order expressiveness produces more accurate graph uncertainty, beyond improved point-estimate F1.

  2. What will happen when the proposed order diffusion is replaced with PL or one-step GPL while other configurations remain the same? The existing permutation-fitting experiment does not isolate its contribution to causal discovery.

  3. Which posterior-consistency guarantees apply to the implemented architecture and training objective? Appendix C.3 shows that independent-edge parameterizations cannot approximate certain correlated graph posteriors. How is Assumption A.4 addressed for the proposed independent-Bernoulli skeleton and skeleton/order factorization under the actual pretraining prior? Additionally, what distribution is targeted by the class-balanced skeleton loss, and is the combined training objective a consistent surrogate for the graph log-loss analyzed in Section 3?

  4. Appendix F truncates the noise distributions and restricts simulator parameters to compact sets, while several benchmark families use unbounded Gaussian, Laplace, or exponential noise. Under the literal bounded-support construction, how are finite optimal KL divergence in A.8 and test-distribution absolute continuity in Theorem 3.7 satisfied for those cases? Are these experiments intended to lie outside the theorem’s scope?

  5. How is the single graph used for F1/SHD evaluation obtained from the stochastic model?

  6. What is the "auxiliary precedence head" in App. E.5, and is it part of the loss?

6 comments

Response to W1 Thank you for raising this concern. Our main contribution is to provide a statistical framework for analyzing causal discovery foundation models. Guided by the theory, we propose a model that can represent posterior distributions over DAGs. In response to the feedback, we have added two experiments to demonstrate the significance of the theory and the improvement in our model’s expressive capacity. We will discuss these experiments in detail in our responses to the corresponding questions.

Response to W2,Q1 and Q2 Thank you for suggesting a direct evaluation of the learned graph posterior. Theorems 3.1 and 3.2 characterize the convergence of the learned posterior with sufficient training. We added a controlled experiment on three-node graphs, described in detail in Section 5.4 and Appendix J.4. Under the prior used in this experiment, observational data cannot distinguish DAGs within the same Markov equivalence class (MEC), and we can compute the exact graph posterior p(G∣X)p(G \mid X) for any given dataset XX.

We trained three models under this shared prior, using the same preprocessing and encoder architecture. The first is a categorical model that directly predicts posterior probabilities over all 25 DAGs. The second is our order diffusion model. The third replaces only the order diffusion module with a Plackett–Luce (PL) distribution. We evaluated all three on the same 5,000 test datasets, each containing n=100n=100 observations. The mean forward KL divergence from the exact graph posterior to the predicted distribution was 0.00492 for the categorical model, 0.05258 for order diffusion, and 0.15847 for PL. Order diffusion thus reduced the mean KL divergence by 66.8% relative to PL, indicating that a more expressive distribution over topological orders improves graph-posterior approximation.

Appendix J.4 also compares two test datasets generated by the same chain, A→B→CA \to B \to C. Although both contain 100 observations, their exact posteriors differ substantially. For X2X_2, the exact posterior assigns 99.99% of its mass to the generating MEC, with approximately 1/31/3 on each of its three DAGs. Order diffusion assigns 99.45% to this MEC and closely reproduces the uniform allocation within it. PL assigns approximately 58% to the fork A←B→CA \leftarrow B \to C.

For X1X_1, the exact posterior assigns 10.37% of its mass outside the generating MEC and 29.88% to each of the three DAGs within it. Order diffusion assigns 16.56% outside the MEC and gives the forward chain, the fork, and the reverse chain probabilities of 26.63%, 30.34%, and 26.48%, respectively. It therefore largely preserves their relative weights. PL again favors the fork, assigning it 48.64% and each chain only about 16.8%. In this case, a more expressive order model improves the approximation, with a forward KL divergence on X1X_1 of 0.01738 for order diffusion versus 0.14242 for PL.

Response to Q3 Thank you for raising this point. We agree that our skeleton–order factorization with an independent Bernoulli skeleton does not satisfy Assumption A.4. Our model aims to provide a more expressive representation of distributions over DAGs. We examine the theorem's predictions empirically on graphs with three or five nodes (see Appendices J.3 and J.4). How to efficiently represent general posteriors over DAGs remains an open question. Methods such as TabCausal predict edges directly, and does not guarantee acyclicity. DAG-FM guarantees acyclicity by successively predicting leaf nodes and their parents, but this sequential procedure can make learning more difficult and allow early prediction errors to propagate. Following BCNP and Arrow, we use a node order to guarantee acyclicity and introduce diffusion over orders to better capture complex order distributions. Developing more expressive models of DAG posteriors is a direction for future work.

Thank you for pointing out the issue with the class-balanced loss. We introduced class balancing to improve edge recall under a limited training budget because non-edges can dominate ordinary binary cross-entropy (BCE) on sparse causal graphs. The two objectives target different edge probabilities, and we agree that the class weights should be removed in the later stages of training. We therefore continued training with ordinary BCE to target the original posterior edge marginals. Performance improved across all reported metrics. We have updated the corresponding results in the main text and appendix and marked the changes in blue.

Response to Q4 Thank you for raising this point. We would first like to clarify that our compactness assumption on the distribution family does not require noise truncation. A sufficient condition is that the pretraining parameter space is closed and bounded and that the parameter-to-distribution map is continuous in Hellinger distance. In this setting, Assumption A.8 is easy to satisfy. Neither the LiNGAM experiment nor the additional experiment reported in the main text uses truncated noise (see Appendices J.3.2 and J.4, respectively). We used truncated noise only when pretraining the model evaluated in Section 5.2. We agree that your concern is valid. The correct approach is to restrict the parameter ranges. At the scale of our training runs, however, this truncation has no practical effect.

Effect of noise truncation. In the bounded-noise compact pretraining simulator, we truncate the base random variables before applying node-specific noise scales. Specifically, we condition Z∼N(0,1)Z\sim\mathcal{N}(0,1) on ∣Z∣≤8|Z|\leq 8, L∼Laplace(0,1/2)L\sim\mathrm{Laplace}(0,1/\sqrt{2}) on ∣L∣≤24|L|\leq 24, and E∼Exp(1)E\sim\mathrm{Exp}(1) on E≤36E\leq 36. The tail probabilities removed by these cutoffs are approximately 1.24×10−151.24\times10^{-15}, 1.82×10−151.82\times10^{-15}, and 2.32×10−162.32\times10^{-16}, respectively. For a single base-noise array of at most 25×200025\times2000 draws, the probability that at least one untruncated draw exceeds its cutoff is at most 9.09×10−119.09\times10^{-11}.

Response to Q5 Thank you for asking us to clarify how we obtain the graph used for evaluation. We have added Appendix E.5 (Graph Prediction) to make this procedure explicit. For each test dataset, we draw 500 independent priority vectors and decode one candidate node order per vector using reverse-diffusion beam search. We select the Kendall medoid, i.e., the candidate that minimizes the total number of pairwise ordering disagreements across all candidates. We then retain skeleton edges with predicted probability above 0.5 and orient them according to this order. This yields a single acyclic graph per dataset, which we use for both directed-edge F1 and normalized SHD.

We chose the Kendall medoid because a DAG may admit several valid topological orders. Even when no complete order occurs twice, the medoid uses the pairwise ordering information shared across candidates. Because the selected order is itself a candidate, it also preserves every precedence relation on which all candidates agree.

Response to Q6 Thank you for the question. We clarify its definition and auxiliary loss in Appendix E.6. The auxiliary precedence head is attached to the shared encoder and predicts three relations for each node pair (i,j)(i,j): ii is an ancestor of jj, jj is an ancestor of ii, or the two nodes are incomparable. Labels are obtained from the transitive closure of the training DAG and therefore include indirect directed paths. The head contributes a mean three-class cross-entropy loss only during the initial training stage. This weight is subsequently set to zero, and later training optimizes only the skeleton and order-diffusion losses.

The purpose of this head is to provide direct structural guidance for early representation learning. It supplies ancestral reachability information beyond direct adjacency, with labels that do not depend on the sampled topological order or diffusion noise. This supervision is intended to help the shared encoder learn structural features relevant to topological ordering. Disabling it later allows optimization to focus on the main tasks.

The head is not used at inference. It therefore serves as an early-stage optimization aid without altering the final skeleton–order parameterization, inference procedure, or acyclicity guarantee.

response to W3 Thank you for pointing this out. We agree that DAG F1 measures structural recovery, while the probability assigned to the true DAG, q(G0∣X)q(G_0 \mid X), more directly assesses concentration of the learned graph distribution.

We added a three-node LiNGAM experiment in Section 5.3 and Appendix J.3.2, where we compare our diffusion model with a classifier over all 25 labeled DAGs. We pretrained each model on 5,000,000 datasets with 20 to 500 observations each. From n=20n=20 to n=500n=500, the mean probability assigned to the true DAG rose from 0.4251 to 0.9770 for our diffusion model and from 0.4456 to 0.9970 for the classifier, averaged over 50 test datasets per sample size. With model parameters held fixed, our model improved further beyond the pretraining range, assigning mean probabilities of 0.9922 at n=1,000n=1{,}000 and 0.9961 at n=2,000n=2{,}000.

Thank you for the detailed comments and the revision. The exact-posterior comparison and the new LiNGAM results address my main empirical requests. I have two remaining questions:

  1. Section 5.4 clarifies the benefit of order diffusion over PL, but I am still unclear what the theory implies for the full model. Given that the implemented model does not satisfy A.4, what does the framework tell us about the errors we should expect from the full model? For example, could the exact posteriors in section 5.4 help quantify how much error remains under the current factorization even with an unrestricted order distribution?

  2. understand why noise truncation is unlikely to affect the reported finite-sample results. Could you state explicitly in the revised manuscript whether the comparisons in section 5.2 with unbounded test noise satisfy A.8 and the test-support condition in Theorem 3.7 under the actual pretraining prior, or whether these comparisons should be interpreted as empirical results outside the guarantees?

Reviewer_qjNCOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proves that pretraining drives the model to learn the prior-induced graph posterior, which concentrates on the KL-best-fitting graph set, and narrows to the true DAG or Markov Equivalence Class under identifiable conditions, as sample size grows. Guided by this theory, authors introduce a foundation model that factorizes DAGs into an undirected skeleton and a topological order, using priority-conditioned diffusion to capture multimodal order uncertainty while strictly guaranteeing acyclic predictions. Across synthetic benchmark settings and the real-world Sachs dataset, their approach achieves the top overall structure recovery scores, outperforming existing foundation models and classical baselines

Critical strengths
  1. The paper establishes a statistical framework that formalises what causal discovery foundation models learn during pretraining and recover as sample size grows, providing much-needed interpretability for pretraining dynamics.
  2. To my knowledge, this is the first work to apply discrete diffusion over permutations to topological order generation, elegantly capturing multimodal order uncertainty while guaranteeing graph acyclicity by construction.
  3. The method shows superior zero-shot transfer compared to existing baselines across mechanism families excluded from pretraining (Gaussian Process and ANM-Gaussian) as well as the real-world Sachs biological dataset.
Critical weaknesses
  1. While the model excels on smaller graphs (d10), its performance drops noticeably as dimensionality grows, falling behind baselines like TabCausal.

  2. The paper does not answer whether the proposed foundation model genuinely learns causal mechanisms or exploits known sorting artifacts present in synthetic pretraining distributions. Prior work has demonstrated that standard synthetic SCM benchmarks suffer from severe variance sortability (Reisach et al., NeurIPS 2021) and R2-sortability (Ormaniec et al., ICLR 2025), allowing models to infer topological ordering directly from marginal covariance scales rather than structural causal dependencies. Since both the pretraining corpus and the 192 synthetic test settings rely on standard, non-standardized SCMs, it remains unclear how much of the model's order-diffusion accuracy relies on these artifacts. Evaluating the proposed method on naively-standardized SCMs and internally-standardized SCMs (iSCMs) would clarify whether priority-conditioned order diffusion generalizes beyond scale-based sorting heuristics.

Minor: In Table 6, are the nSHD and Exact column headers accidentally swapped, given that reported CPDAG nSHD values (e.g., 0.115 for CauScale) are lower than their underlying skeleton nSHD values in Table 5 (0.368)?

Questions for authors
  1. Given that the paper’s central question from the title, how much does the priority-conditioned order diffusion rely on marginal scale shortcuts versus genuine structural causal dependencies? Could the authors discuss, how the proposed model and key baselines perform on naively standardized SCMs, and iSCMs where sortability heuristics are explicitly eliminated?
  2. How does the pretrained priority diffusion model perform when evaluated zero-shot on graphs that exceed its pretraining dimension bound (e.g., d=30 or d=35)?
1 comment

Response to W2 and Q1

Thank you for raising this concern about sorting artifacts in synthetic causal discovery benchmarks. In the manuscript, we did not explain in sufficient detail how we standardize the model inputs or how we generate the synthetic test data. We would like to clarify two implementation details that directly address this concern. First, during both pretraining and inference, we standardize each variable within each dataset before passing the observations to the model. Second, five of our eight test distributions also use internal standardization during data generation. We have added descriptions of these procedures to Appendices E.1 and G.1.

Standardization of model inputs. For each dataset, we compute the sample mean and sample standard deviation of each variable across observations and use them to center and scale that variable. The model receives only the standardized observations. It therefore cannot infer a topological order by directly ranking variables according to their original marginal variances.

Standardization during data generation. The eight test distributions in Section 5.2 use the following procedures.

  1. Empirical internal standardization (ID-Mix, Linear, MLP-Additive, MLP-Nonadditive, and RFF). We generate variables in topological order. Immediately after generating each variable, including each root variable, we standardize it using its empirical mean and standard deviation across observations in the current dataset. Its children therefore receive standardized parent values.

  2. Standardization of parent inputs (GP). Before evaluating the Gaussian-process mechanism at each non-root node, we standardize each parent variable across observations. We also standardize each column of the completed dataset before storage.

  3. Post-hoc standardization (LiNGAM and ANM-Gaussian). For these two distributions, we store the observations on their original scales.

The model therefore receives standardized observations in all 192 benchmark settings. Of these, 5×4×6=1205 \times 4 \times 6 = 120 settings use empirical internal standardization during data generation, covering the mechanism mixture and each of its four component families.

Response to W1 and Q2

Thank you for pointing out the decline in recovery accuracy on larger graphs. To evaluate generalization beyond the pretraining range, we tested our pretrained model zero-shot on 30-node graphs. Its mean SHD was 56.211, ranking third among the seven models. Only TabCausal (43.755) and Arrow (51.502) achieved a lower mean SHD.

Both the range of graph sizes covered during pretraining and the amount of pretraining data are important for interpreting this result. Given our computational budget, we pretrained our model on graphs with 3–25 nodes, so it never encountered a 30-node graph during training. According to our theory, the model learns to approximate the posterior induced by the pretraining prior, and it received no training signal for learning the posterior over 30-node graphs. The pretraining distributions of both TabCausal and Arrow explicitly include 30-node graphs. Thus, although all models are evaluated zero-shot on new 30-node graphs, our model must also extrapolate to a graph size beyond its pretraining limit. Despite this limitation, our theory provides a statistical foundation for causal discovery foundation models, and our results on smaller graphs give confidence in this approach.

Response to the minor comment The column headers are correct. For CauScale, Table 6 reports a CPDAG nSHD of 0.404 and an exact-match rate of 0.115. The CPDAG nSHD is therefore higher than the skeleton nSHD of 0.368 in Table 5.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their constructive feedback. We appreciate their recognition of the value of our statistical framework and our proposed model.

During the rebuttal, we made four main improvements:

  1. Added posterior-fitting experiments that provide empirical support for our theoretical results and show that our model improves posterior approximation.
  2. Used a three-node example to show how our theory can analyze our model’s error and establish large-sample MEC recovery for an idealized version of our model.
  3. Added details about model implementation and evaluation.
  4. Added content to clarify the connection between our theory and method, making our motivation and the overall structure of the paper clearer.

The main points of our rebuttal are summarized below.

TopicNew in this rebuttalReviewer
Sorting artifactsClarified input standardization and test-data generation.qjNC
Larger graphsEvaluated on 30-node graphs.qjNC
Posterior modelingAdded exact-posterior and LiNGAM experiments; reduced mean KL divergence by 66.8% versus PL.Xzrq
Approximation errorAnalyzed approximation error and proved convergence to the true MEC under the three-node prior.Xzrq
Theoretical scopeClarified noise truncation and the empirical purpose of Section 5.2.Xzrq
Implementation detailsAdded descriptions of graph prediction, the auxiliary precedence head, and the class-balanced loss.Xzrq
Theory–model connectionClarified the connection and explained why we present them together.4qfk
External evaluationEvaluated on three external benchmarks and explained the results.4qfk

We have revised the manuscript accordingly and marked the changes in blue. Sections J.3.2, J.4, and J.4.1 are new. Because these sections are long, we marked only their headings in blue.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.