acceptodds
Under review as a conference paper at ICLR 2027

CDFM: Towards a General-Purpose Causal Discovery Foundation Model

Abstract

Causal discovery, the process of recovering underlying causal structures from observational data, is a fundamental pursuit across scientific disciplines. Over the past decades, numerous algorithms have been developed to tackle this challenge through workflows tailored to the specific causal mechanisms underlying each type of dataset, demonstrating effectiveness across a wide range of applications. However, as the volume and heterogeneity of real-world data continue to grow, this dataset-specific approach inevitably leads to a fragmented, test-driven paradigm that struggles to scale to the demands of modern scientific discovery. To address this, we formulate the Causal Discovery Foundation Model (CDFM) as a unified, general-purpose framework for zero-shot structural inference. To ensure reliable generalization across unknown domains, we first investigate the theoretical boundaries of causal identifiability, revealing the indispensable role of causal prior mechanisms in this process. Building on these insights, we formulate a principled variational framework that treats unknown causal mechanisms as latent variables and mathematically decomposes the intractable marginal likelihood into distinct, tractable learning modules. The variational decomposition provides a conceptual design principle for the CDFM architecture, while comprehensive causal knowledge guides the large-scale synthesis of our pretraining data. By pretraining on a massive, highly diverse space of synthetic structural causal models, CDFM successfully internalizes complex statistical asymmetries. Extensive experiments demonstrate that CDFM achieves strong and consistently competitive performance across heterogeneous synthetic and real-world benchmarks.

Primary area:
causal reasoning
Keywords:
Causality, Causal Discovery, Causal Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VqRDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CDFM, a foundation model for causal discovery. Taking a theory-motivated approach based on identification, the resulting intractable exact objective is handled via a variational approximation. Rigorous empirical evaluation against both amortized and non-amortized causal discovery baselines is conducted, showing sota performance.

Critical strengths

I believe the development of foundation models for causal discovery to be of great importance. I especially appreciate the identification motivated approach, relatively clear writing and decent empirical validation of proposed architecture.

Critical weaknesses
  • Novelty: The contribution is insufficiently differentiated from existing amortized causal discovery methods. Considering there are already existing causal discovery "foundation models", this needs to be explained carefully and in detail.
  • Ablations: Missing ablations make it impossible to assess the individual contributions of the proposed architectural components.
  • Variational formulation: The variational family Q(M∣X)Q(M\mid X) is not explicitly specified, and the actual training objective does not optimize the proposed ELBO. This creates a gap between the theoretical motivation and actual implementation.
  • Theoretical guarantees: Corollary 1 appears unsupported, as the graph inference objective omits the data-likelihood term and the proof relies on a generally invalid posterior identity (Appendix A.5).
  • Generalization: The synthetic benchmarks largely cover mechanism families represented during pretraining, leaving generalization to genuinely unseen mechanisms untested.

Overall, I lean towards weak rejection, primarily due to novelty and ablation concerns, but am open to raising my score if these issues are convincingly addressed during rebuttal.

Questions for authors

See Weaknesses section for my main points. Could the authors additionally provide:

  • a link to an anonymized code repository?
  • a Limitations section?
  • identification diagnostic experiments for the original frozen model (without the fine tuning) and competing amortized causal discovery methods?
  • their thoughts of how CDFM handles uncertainty under observational non-identifiability, particularly when the posterior over causal graphs is multimodal?
4 comments
AuthorsOpenReview ↗

Reply to Reviewer VqRD (1/3)

W1: Novelty.

Thanks for asking us to clarify CDFM's contribution. We start from identifiability across unknown mechanisms and use it to design a learning procedure that couples mechanism inference, reconstruction, and graph prediction.

W2: Ablations.

Thanks for asking for the ablation study. To evaluate the contribution of CDFM's three main components, including test-time adaptation, auxiliary reconstruction loss, and the mixture-of-experts (MoE) module, we conduct controlled ablations. Specifically, we retrained all variants from scratch for 10,000 steps on the same 15-mechanism prior, using identical initialization. All variants were evaluated on the same 3,600-task panel (15 mechanisms × 4 dimensions × 2 types of sample sizes × 30 datasets). Details are provided in Appendix G.2.

VariantAUROCΔAUROC vs. baseline
Baseline (adaptation off, reconstruction off)0.7386—
+ Test-time adaptation0.7858+0.0472 (+6.4%)
+ Auxiliary reconstruction0.7437+0.0051 (+0.7%)
+ Both0.7943+0.0557 (+7.5%)
Auxiliary reconstruction, w/o expert bank0.7424-0.0038 (-0.5%)

(1) Test-time adaptation. This is the main source of improvement. It increases AUROC by 0.0472 without auxiliary reconstruction and by 0.0506 with it. The similar gains show that adaptation is effective.

(2) Auxiliary reconstruction loss. Its contribution is smaller but consistent. The results show that the performance improves by +0.0051 without adaptation and +0.0085 with adaptation.

(3) Mixture of experts. We test it by removing the expert bank. The exclusion of the expert bank decreases the performance from 0.7437 to 0.7424, indicating a modest contribution from the MoE module. To further examine its role, we added a routing visualization to the appendix showing the evidence that the expert router indeed captures mechanism-specific structure.

W3 & W4: Variational formulation; Theoretical guarantees.

Thanks for asking what the encoder represents and how it enters the identification argument. In ELBO, Q(M∣X)Q(M\mid\mathbf X) motivates the learned mechanism representation, P(X∣G,M)P(\mathbf X\mid G,M) supplies the reconstruction term, and P(G∣M)P(G\mid M) supplies the graph term. In CDFM, the implenmentation is indeed optimize the variational formulation above. To see this, first, the reconstruction term is implemented by the structure-aware data reconstruction component. Second, the graph inference are learned by direct supervised by the BCE loss. The remaining KL term in Eq. (6) usually serves as the regularization term, and we would like to omit it if we consider the uniform prior over the causal mechanism. We have revised this part of the paper to make it clearer.

For identification, Corollary 1 compares the full ELBO, including the reconstruction term. Appendix A.5 makes explicit that a sufficiently small variational gap at the true graph preserves its likelihood advantage. The structural decoder inherits this conclusion when its scores preserve the same separation. The argument compares the full bounds directly, which addresses the concern about the posterior identity.

AuthorsOpenReview ↗

Reply to Reviewer VqRD (2/3)

W5: Generalization.

Thanks for suggesting a mechanism-exclusion test. We have completed the 15-family exclusion study, with a separate retraining run for each omitted generator. Specifically, to test transfer to held-out mechanism families, we trained 16 CDFM models from scratch with a 10k-step training budget: one control using all 15 synthetic families and 15 leave-one-family-out models, each excluding one family. The results are detailed as follows:

Excluded familyFull priorFamily excludedΔ own family
CPT0.6100.487-0.123
discrete-ANM0.6600.562-0.098
PNL0.8290.764-0.065
RFF-hetero0.8100.788-0.022
physical-hetero0.6870.665-0.022
linear0.8450.832-0.013
physical0.6890.676-0.013
linear-hetero0.8030.790-0.012
time-lag0.7560.744-0.012
RFF0.8270.815-0.012
linear-ordinal0.6300.620-0.011
rounded0.7700.762-0.008
CAM0.7590.755-0.004
measurement-error0.7770.774-0.002
RFF-ordinal0.6430.650+0.008

The results suggest that other generators provide sufficient coverage for most of the tested families, while CPT, discrete ANM, and PNL benefit more from explicit representation. We also note that some retained generators share lower-level functions. The results and the training setting are detailed in Appendix G.1.

Q1: a link to an anonymized code repository?

Thanks for asking about the code. The Anonymized code and reproduction instructions are provided at https://anonymous.4open.science/r/CDFM_ICLR-8FB7/.

Q2: A Limitations section?

Thanks for the suggestion. We have added a discussion of limitations in the appendix. We view this work as a first step toward a general-purpose foundation model for causal discovery. Our main focus is to establish the identifiability conditions under which such a model can operate and empirically validate the proposed learning framework across heterogeneous causal mechanisms. However, the current pretraining prior still covers a limited set of mechanisms. Extending this coverage and further investigating generalization to unseen mechanisms and real-world causal systems are important directions for future work.

AuthorsOpenReview ↗

Reply to Reviewer VqRD (3/3)

Q3: identification diagnostic experiments for the original frozen model (without the fine tuning) and competing amortized causal discovery methods?

Thanks for this suggestion. To examine whether the identification results depend on fine-tuning, we repeated all three diagnostic experiments using the released CDFM checkpoint without diagnostic-specific fine-tuning. We also evaluated FoundCause, AVICI, and TabCausal using their official pretrained checkpoints, together with classical methods where applicable. The experimental settings follow those in our original appendix, and all methods are evaluated on the same datasets.

(1) Functional asymmetry. We consider the same functional model Y=X+bX3+ϵY=X+bX^3+\epsilon, with varying nonlinearity bb and noise parameter qq. Each setting contains 60 datasets with N=2000N=2000.

bbqqCDFMAVICITabCausalDirectLiNGAM
0148.350.051.750.0
02100.0100.0100.0100.0
0.5113.30.00.01.7
0.52100.075.0100.00.0
11100.00.00.011.7
12100.05.075.00.0

The CDFM achieves near-chance orientation accuracy in the linear Gaussian setting and correctly identifies the direction when the nonlinearity or non-Gaussianity is sufficiently strong (b=1b=1), under both noise settings.

(2) V-structure identification. We evaluate accuracy on orientation accuracy from V-structures, using 400 DAGs per setting at N=2000N=2000.

#V-structuresCDFMTabCausalAVICI
185.985.465.5
288.886.174.9
390.590.181.2

Without fine-tuning, CDFM achieves higher orientation accuracy when more v-structures are present.

(3) Latent confounding. We report AUROC and directed-edge orientation accuracy under Gaussian and heavy-tailed noise.

NoiseCDFMFoundCauseAVICITabCausal
Gaussian (q=1q=1)0.36 / 62%0.43 / 57%0.50 / 52%0.24 / 50%
Heavy-tailed (q=2q=2)0.99 / 100%0.78 / 15%1.00 / 100%1.00 / 100%

CDFM, AVICI, and TabCausal all perform substantially better under heavy-tailed noise than under Gaussian noise in this experiment.

Overall, these experiments show that several identification patterns observed in the original diagnostic experiments are already present in the pretrained CDFM without fine-tuning. Similar behaviors are also observed in competing amortized methods, although their sensitivity to different identification conditions varies. The results further highlight that theoretical identifiability remains necessary and that the foundation model satisfies it.

Q4: their thoughts of how CDFM handles uncertainty under observational non-identifiability, particularly when the posterior over causal graphs is multimodal?

Thanks for asking how the model represents competing graph explanations. In practice, CDFM can assign scores to both directions of an edge, serving as a measure of uncertainty. For example, our code provides a cycle-pruning function that prunes cycles according to confidence scores, which is one application of using ranking as confidence. We agree that explicitly modeling this uncertainty, especially the multimodal joint posterior over causal graphs, is important, and we consider it future work.

Thank you for the detailed rebuttal and additional experiments. While the new ablations and generalization tests are useful, I will maintain my original score. The gap between the variational formulation and actual training objective remains unresolved, and the proof of Corollary 1 still contains unjustified steps. I also remain unconvinced that the methodological novelty is sufficiently established or that the experiments demonstrate generalization beyond the shared synthetic framework.

Reviewer_X25UOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CDFM, a pretrained model for causal discovery from observational data. The model combines synthetic SCM pretraining, column-wise representations, alternating attention, mechanism-aware aggregation, graph prediction, and structure-conditioned reconstruction. The authors also present a theoretical formulation based on causal identifiability and variational inference.

Critical strengths

The paper addresses an important problem: amortizing causal structure learning across heterogeneous data-generating mechanisms.

The synthetic pretraining distribution is broad, covering different graph families, functional mechanisms, noise distributions, and observational imperfections.

Critical weaknesses
  1. Limited methodological novelty

Supervised causal discovery is already a well-established and relatively rich research direction. Existing methods learn from synthetic data–graph pairs and differ along several dimensions, including the construction of training data, prediction targets (skeletons, v-structures, CPDAGs, DAGs, or graph posteriors), neural architectures, and graph decoding strategies.

The proposed CDFM largely follows this established paradigm. Its main architectural components, including Set Transformer representations, alternating row-column attention, and directed-edge prediction, have close precedents in existing work.

In particular, the contribution of graph-conditioned reconstruction and test-time refinement is not independently demonstrated. The mechanism-aware experts also lack evidence that they learn meaningful mechanism-specific representations.

  1. Incomplete related work and experimental comparisons

The related-work coverage is incomplete given the extensive literature on supervised and amortized causal discovery.

Several important research directions are insufficiently represented, including supervised learning of identifiable structural components, bilinear attention for graph inference (BAM), structured graph-posterior prediction, alternative graph decoding strategies, and methods addressing generalization through test-time adaptation.

An additional concern is the use of independently scored directed edges as the primary prediction target. Prior work such as SiCL has identified structural consistency limitations of this formulation. CDFM employs a similar edge-prediction approach, but does not demonstrate whether its additional constraints or reconstruction mechanism address these limitations.

The experimental comparisons also need to be substantially strengthened. While the paper includes several established baselines, it lacks comparisons with a number of relevant supervised and pretrained causal discovery methods. At minimum, the authors should provide a more systematic comparison of methodological choices and prediction targets, with direct experimental comparisons where the input assumptions are compatible.

  1. Insufficient experimental evidence for the foundation model claim

This is my primary concern.

The motivation for supervised causal discovery is compelling: machine learning may replace carefully designed, dataset-specific causal discovery algorithms with a reusable inference model. However, the practical value of this direction must ultimately be established through successful generalization to real causal discovery problems.

The evaluation in this paper remains heavily dependent on synthetic data. Although the training and testing cover many mechanism families, the synthetic benchmarks largely operate within the same broad simulation framework and parameter ranges.

Such results demonstrate performance under the chosen simulator distribution, but provide limited evidence of generalization to genuinely unseen causal mechanisms or real-world data-generating processes.

The real-world evaluation is also not sufficiently convincing. Causal Chamber provides useful evidence, but the evaluation is limited. The newly introduced RCB benchmark relies on domain annotations and statistical dependence tests rather than independently established causal ground truth. In particular, statistical dependence does not establish a direct causal edge, and failure to reject conditional independence does not establish a non-edge.

For a paper claiming a general-purpose causal discovery foundation model, I would expect substantially stronger evidence of transfer beyond the synthetic pretraining distribution and more extensive validation using real-world causal systems with independent experimental or domain-grounded causal evidence.

Without such evidence, the paper does not convincingly demonstrate the general-purpose capability that motivates its central contribution.

  1. The following relevant works are not cited in the submitted manuscript. They cover different supervised causal discovery formulations, prediction targets, architectures, graph decoding strategies, and evaluation approaches. Not all are directly comparable under identical input assumptions. Towards a Learning Theory of Cause-Effect Inference. ICML 2015.

Supervised Whole DAG Causal Discovery. 2020.

ML4S: Learning Causal Skeleton from Vicinal Graphs. KDD 2022.

ML4C: Seeing Causality Through Latent Vicinity. SDM 2023.

Scalable Differentiable Causal Discovery in the Presence of Latent Confounders with Skeleton Posterior (SPOT). KDD 2024.

Graph Structure Inference with BAM: Neural Dependency Processing via Bilinear Attention. NeurIPS 2024.

Learning Identifiable Structures Helps Avoid Bias in DNN-based Supervised Causal Learning (SiCL). AISTATS 2025.

A Meta-Learning Approach to Bayesian Causal Discovery (BCNP). ICLR 2025.

Demystifying Amortized Causal Discovery with Transformers. TMLR 2025.

Test Time Training for Supervised Causal Learning (TTT-SCL). 2026.

A Knowledge-Informed Pretrained Model for Causal Discovery (Kode). 2026.

DAG-FM: A Foundation Model for Causal Discovery under Heterogeneous Causal Mechanisms. 2026.

Test-Time Learning of Causal Structure from Interventional Data (TICL). ICML 2026.

CausalArena: Benchmarking Causal Discovery in the Foundation Model Era. 2026.

Beware of the Simulated DAG! Causal Discovery Benchmarks May Be Easy to Game. NeurIPS 2021.

The Landscape of Causal Discovery Data: Grounding Causal Discovery in Real-World Applications. CLeaR 2025.

Questions for authors

Questions for Authors

  1. Structural consistency (SiCL). SiCL shows that independently predicting directed-edge probabilities can produce graphs outside the correct Markov equivalence class, even when the edge marginals are accurately estimated. Consider the three-variable case where X → T → Y and X ← T ← Y are observationally indistinguishable under suitable SCMs. How would CDFM behave in this setting? Can its edge prediction and graph-adaptive thresholding guarantee a valid structure within the equivalence class? A controlled experiment based on the SiCL example would be informative.

  2. Methodological novelty. What is the key methodological innovation of CDFM beyond existing supervised causal discovery approaches? In particular, can the authors isolate the contributions of mechanism-aware experts, graph-conditioned reconstruction, and test-time refinement through controlled ablations?

  3. Generalization beyond synthetic training distributions. How does CDFM perform on causal mechanism families or SCM generators entirely excluded from pretraining? Can the authors provide leave-one-mechanism-family-out or cross-simulator experiments to substantiate the general-purpose claim?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer X25U (1/2)

W1 & Q2: Limited methodological novelty; Methodological novelty and controlled ablations

Thanks for asking what CDFM adds beyond established supervised architectures. CDFM targets a setting in which a single pretrained model must infer graphs under unknown causal mechanisms. Its key component is a learned inner-loop adaptation step: the model uses masked-observation reconstruction to refine its dataset representation before graph prediction. Crucially, this loop is unrolled during pretraining and applied at test time, so the model is explicitly trained to benefit from adaptation rather than having an optimization step added only at inference. CDFM further combines a shared representation with an MoE to capture mechanism-dependent features, while an auxiliary reconstruction loss provides additional supervision. This design follows the variational decomposition in Sections 2 and 5.

Controlled ablations. To evaluate the contribution of CDFM's three main components, including test-time adaptation, auxiliary reconstruction loss, and the mixture-of-experts (MoE) module, we conduct controlled ablations. Specifically, we retrained all variants from scratch for 10,000 steps on the same 15-mechanism prior, using identical initialization. All variants were evaluated on the same 3,600-task panel (15 mechanisms × 4 dimensions × 2 types of sample sizes × 30 datasets). Details are provided in Appendix G.2.

VariantAUROCΔAUROC vs. baseline
Baseline (adaptation off, reconstruction off)0.7386—
+ Test-time adaptation0.7858+0.0472 (+6.4%)
+ Auxiliary reconstruction0.7437+0.0051 (+0.7%)
+ Both0.7943+0.0557 (+7.5%)
Auxiliary reconstruction, w/o expert bank0.7424-0.0038 (-0.5%)

(1) Test-time adaptation. This is the main source of improvement. It increases AUROC by 0.0472 without auxiliary reconstruction and by 0.0506 with it. The similar gains show that adaptation is effective.

(2) Auxiliary reconstruction loss. Its contribution is smaller but consistent. The results show that the performance improves by +0.0051 without adaptation and +0.0085 with adaptation.

(3) Mixture of experts. We test it by removing the expert bank. The exclusion of the expert bank decreases the performance from 0.7437 to 0.7424, indicating a modest contribution from the MoE module. To further examine its role, we added a routing visualization to the appendix showing the evidence that the expert router indeed captures mechanism-specific structure.

W2: Incomplete related work and experimental comparisons.

Thanks for pointing out these related works. We have expanded the related work section to discuss supervised causal discovery methods and their differences in prediction targets, input assumptions, and generalization settings. For example, ML4S focuses on skeleton prediction, SiCL predicts skeletons and v-structures, and BCNP models uncertainty over causal graphs. In contrast, CDFM aims to infer directed causal structures across heterogeneous mechanisms using a shared pretrained model with reconstruction-based adaptation.

Additional comparisons. Since the existing methods do not provide directly usable pretrained checkpoints for our evaluation setting, we retrained BAM2, SiCL, and BCNP using their official implementations and default training configurations. As these methods were originally evaluated under different data and training settings, we focus on a shared, relatively low-dimensional benchmark covering 15 mechanisms, D∈{5,10,15,20}D\in\{5,10,15,20\}, N∈{500,1000}N\in\{500,1000\}, and 30 datasets per configuration. The evaluation contains 3,600 tasks, with directed AUROC defined for 3,442 tasks.

MethodDirected AUROCOrientation accuracyF1
CDFM0.87060.81550.5609
BAM20.67050.54050.3095
SiCL0.68520.07480.2088
BCNP0.61370.49830.1721

CDFM achieves the best results across all three reported metrics on this shared evaluation panel. We also observe consistent improvements across the four tested dimensions. These comparisons provide additional evidence of CDFM's performance under heterogeneous mechanisms. We have incorporated the relevant methodological discussion into the revised Related Work section.

AuthorsOpenReview ↗

Reply to Reviewer X25U (2/2)

W2 & Q1: Structural consistency (SiCL).

Thanks for raising this important question. We would like to distinguish two issues: the identifiability of edge directions and the structural consistency of edge predictions.

Identifiability. Unlike SiCL, which focuses on identifiable structures under the canonical MEC setting, CDFM aims to exploit mechanism-specific asymmetries to identify edge directions beyond the MEC whenever possible. Nevertheless, when the underlying mechanisms are non-identifiable, such as in linear Gaussian SCMs, CDFM also exhibits ambiguity in edge orientation.

Structural consistency. Although CDFM predicts directed-edge probabilities, each prediction is conditioned on the shared representation of the entire dataset rather than being made from isolated variable pairs. This allows the model to account for global structural information when predicting individual edges.

Following the SiCL example, we conducted controlled experiments on four three-variable structures under linear Gaussian and linear non-Gaussian (Laplace noise) SCMs, with 500 datasets per configuration. We used the same pretrained checkpoint and decoding procedure throughout.

Ground-truth structureGaussian: Orient. Acc.Gaussian: In MECGaussian: V-struct. rateLaplace: Orient. Acc.Laplace: In MECLaplace: V-struct. rate
X→T→YX\to T\to Y49.2%88.2%11.8%100%100%0%
X←T←YX\leftarrow T\leftarrow Y51.3%90.0%10.0%100%100%0%
X←T→YX\leftarrow T\to Y72.1%91.2%8.8%100%100%0%
MEC average57.5%89.8%10.2%100%100%0%
X→T←YX\to T\leftarrow Y (positive control)99.7%99.6%99.6%100%100%100%

Under linear Gaussian SCMs, although edge directions within the MEC cannot be uniquely identified, 89.8% of CDFM's predictions remain within the correct MEC. The model also correctly identifies the collider in 99.6% of cases. When the noise distribution is changed to Laplace, CDFM achieves 100% orientation accuracy across all four structures, with no spurious v-structures.

These results show that CDFM can exploit identifiable mechanism-specific information when available, while largely preserving structural consistency even when edge directions are non-identifiable. However, neither edge prediction nor graph-adaptive thresholding provides a formal guarantee of MEC consistency. We acknowledge this limitation and include the experiment in the revised manuscript.

W3 & Q3: Insufficient experimental evidence for the foundation model claim; Generalization beyond synthetic training distributions.

Thanks for asking for tests that go beyond the original simulator. We have completed the 15-family exclusion study, with a separate retraining run for each omitted generator. Specifically, to test transfer to held-out mechanism families, we trained 16 CDFM models from scratch with a 10k-step training budget: one control using all 15 synthetic families and 15 leave-one-family-out models, each excluding one family. The results are detailed as follows:

Excluded familyFull priorFamily excludedΔ own family
CPT0.6100.487-0.123
discrete-ANM0.6600.562-0.098
PNL0.8290.764-0.065
RFF-hetero0.8100.788-0.022
physical-hetero0.6870.665-0.022
linear0.8450.832-0.013
physical0.6890.676-0.013
linear-hetero0.8030.790-0.012
time-lag0.7560.744-0.012
RFF0.8270.815-0.012
linear-ordinal0.6300.620-0.011
rounded0.7700.762-0.008
CAM0.7590.755-0.004
measurement-error0.7770.774-0.002
RFF-ordinal0.6430.650+0.008

The results suggest that other generators provide sufficient coverage for most of the tested families, while CPT, discrete ANM, and PNL benefit more from explicit representation. We also note that some retained generators share lower-level functions. The results and the training setting are detailed in Appendix G.1.

Reviewer_f5qbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CDFM, a foundation model that infers causal graphs from observational data without assuming a specific causal mechanism. The authors first discuss identifiability and argue that the marginal likelihood over an identifiable mechanism space can recover the true graph. The model is pretrained on a large and diverse set of synthetic SCMs, including discrete data, missing values, and latent confounding. CDFM achieves the highest AUROC on a synthetic benchmark with 15 mechanism families and shows strong results on Causal Chamber, Tübingen pairs, and a new real-world benchmark (RCB).

Critical strengths
  1. The motivation and problem definition make sense.

  2. Broad pretraining prior. The prior mixes mechanism families at the node level and mixed and discrete variables and so on. The coverage is broad than most baselines.

  3. Experimental results. CDFM achieves the highest mean AUROC on the synthetic benchmark compared with prior works. They also introduce a new real-world benchmark (RCB) with trusted causal relations. To my knowledge, real-world benchmarks with causal ground truth are rare in this field, so this is a valuable contribution.

Critical weaknesses
  1. Flaw in the identifiability condition. In Theorem 3 [Eq. (20)], any supergraph of true graph G⋆G^\star can reproduce P⋆P^\star by ignoring extra parents, so its 1N\frac{1}{N}-normalized log-marginal likelihood has the same limit as that of G⋆G^\star and the strict inequality fails. If there are any assumptions like "there are no weak causal relation", it can be right. But, I think it can not be forced.

  2. Corollary 1 does not apply to CDFM, and its proof has invalid steps. The corollary assumes the network attains the optimum of the ELBO in Eq. (6), but CDFM is trained with the supervised loss in Eq. (11) and never optimizes this ELBO.

Besides that, I feel there are some theoretical issues.

  1. Synthetic evaluation is in-prior and inconsistently reported. The pretraining generator (App. B.1) covers most of the 15 test families and the test sets differ only by held-out seeds, so Table 1 measures in-distribution fit rather than the claimed generalization to unseen mechanisms.

  2. Gap between implementation and Theory. Corollary 1's identifiability assumes that CDFM reaches the optimal value of ELBO in Eq. (6). However, in the implementation, they didn't use the ELBO as a loss function.

Questions for authors

Please see the weaknesses

3 comments
AuthorsOpenReview ↗

Reply to Reviewer f5qb

W1: Flaw in the identifiability condition: supergraphs can reproduce the true distribution by ignoring extra parents, yielding the same normalized log-marginal-likelihood limit and invalidating the strict inequality.

Thank you for pointing out this important issue. We agree that, under standard SCM, a supergraph can reproduce the distribution of a smaller graph by assigning zero effect to an extra parent. For example, consider X1→Y←X2X_1\to Y\leftarrow X_2, with Y=aX1+cX2+NY=aX_1+cX_2+N. At (c=0), the model reduces to Y=aX1+NY=aX_1+N, and thus reproduces the distribution of the smaller graph exactly.

We would like to clarify that this scenario is out of the scope of this theorem, and our strict space identifiability condition also explicitly rules out such redundant representations. Moreover, theoretically, a fixed small nonzero cc will still give X2X_2 a direct effect given the large amount of samples, and thus the normalized likelihood gap will not vanish. Thanks again for raising this issue, and we would like to revise our paper to make this point clearer.

W2 & W4: Corollary 1 does not apply to CDFM, and its proof has invalid steps; Gap between implementation and Theory.

Thanks for asking how Eq. (11) implements the variational formulation. In Eq. (6), Q(M∣X)Q(M\mid\mathbf X) motivates the learned mechanism representation, P(X∣G,M)P(\mathbf X\mid G,M) supplies the reconstruction term, and P(G∣M)P(G\mid M) supplies the graph term. In CDFM, the implenmentation is indeed optimize the variational formulation above. To see this, first, the reconstruction term is implemented by the structure-aware data reconstruction component. Second, the graph inference are learned by direct supervised by the BCE loss. The remaining KL term in Eq. (6) usually serves as the regularization term, and we would like to omit it if we consider the uniform prior over the causal mechanism. We have revised this part of the paper to make it clearer.

W3: Synthetic evaluation is in-prior and inconsistently reported.

Thanks for raising the distinction between held-out samples and held-out mechanisms. Table 1 uses fresh instances of represented families. To test transfer to held-out mechanism families, we trained 16 CDFM models from scratch with a 10k-step training budget: one control using all 15 synthetic families and 15 leave-one-family-out models, each excluding one family. The results are detailed as follows:

Excluded familyFull priorFamily excludedΔ own family
CPT0.6100.487-0.123
discrete-ANM0.6600.562-0.098
PNL0.8290.764-0.065
RFF-hetero0.8100.788-0.022
physical-hetero0.6870.665-0.022
linear0.8450.832-0.013
physical0.6890.676-0.013
linear-hetero0.8030.790-0.012
time-lag0.7560.744-0.012
RFF0.8270.815-0.012
linear-ordinal0.6300.620-0.011
rounded0.7700.762-0.008
CAM0.7590.755-0.004
measurement-error0.7770.774-0.002
RFF-ordinal0.6430.650+0.008

The results suggest that other generators provide sufficient coverage for most of the tested families, while CPT, discrete ANM, and PNL benefit more from explicit representation. We also note that some retained generators share lower-level functions. The results and the training setting are detailed in Appendix G.1.

Thank you for the rebuttal. However, my main concern about identifiability and the other theoretical issue remain. So, I will maintain my initial recommendation.

AuthorsOpenReview ↗

Reply to Reviewer f5qb

Thank you for your follow-up. Since the remaining theoretical concerns appear to be central to your recommendation, we would like to further clarify the following points.

Regarding Theorem 3, we believe it is important to distinguish between the mathematical validity of a theorem under its stated assumptions and the general applicability of these assumptions. In causal discovery, it is common to impose assumptions that exclude certain sources of observational ambiguity, with faithfulness being a well-known example. Our strict space identifiability condition serves a similar purpose, although it imposes a different requirement on the mechanism space, and thus the supergraph example violates this explicitly stated assumption.

Regarding Corollary 1, we would like to further clarify the connection between the variational formulation in Eq. (6) and the practical training objective in Eq. (11). The two likelihood terms in the ELBO directly motivate the supervised graph prediction and structure-aware data reconstruction objectives in CDFM, respectively. The remaining KL term serves as a regularizer over the latent mechanism distribution. Under a uniform mechanism prior, the KL term reduces to an entropy regularizer up to an additive constant and does not affect optimization when the variational entropy is fixed. Since CDFM employs a deterministic mechanism encoder rather than explicitly learning a stochastic posterior, we omit explicit KL regularization in the practical training objective. The theoretical formulation thus provides a principled mathematical basis for both the architectural design and the choice of learning objectives in CDFM.

We would be happy to further discuss and clarify any remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.