- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes CDFM, a foundation model for causal discovery. Taking a theory-motivated approach based on identification, the resulting intractable exact objective is handled via a variational approximation. Rigorous empirical evaluation against both amortized and non-amortized causal discovery baselines is conducted, showing sota performance.
Critical strengths
I believe the development of foundation models for causal discovery to be of great importance. I especially appreciate the identification motivated approach, relatively clear writing and decent empirical validation of proposed architecture.
Critical weaknesses
- Novelty: The contribution is insufficiently differentiated from existing amortized causal discovery methods. Considering there are already existing causal discovery "foundation models", this needs to be explained carefully and in detail.
- Ablations: Missing ablations make it impossible to assess the individual contributions of the proposed architectural components.
- Variational formulation: The variational family is not explicitly specified, and the actual training objective does not optimize the proposed ELBO. This creates a gap between the theoretical motivation and actual implementation.
- Theoretical guarantees: Corollary 1 appears unsupported, as the graph inference objective omits the data-likelihood term and the proof relies on a generally invalid posterior identity (Appendix A.5).
- Generalization: The synthetic benchmarks largely cover mechanism families represented during pretraining, leaving generalization to genuinely unseen mechanisms untested.
Overall, I lean towards weak rejection, primarily due to novelty and ablation concerns, but am open to raising my score if these issues are convincingly addressed during rebuttal.
Questions for authors
See Weaknesses section for my main points. Could the authors additionally provide:
- a link to an anonymized code repository?
- a Limitations section?
- identification diagnostic experiments for the original frozen model (without the fine tuning) and competing amortized causal discovery methods?
- their thoughts of how CDFM handles uncertainty under observational non-identifiability, particularly when the posterior over causal graphs is multimodal?
4 comments
Reply to Reviewer VqRD (2/3)
W5: Generalization.
Thanks for suggesting a mechanism-exclusion test. We have completed the 15-family exclusion study, with a separate retraining run for each omitted generator. Specifically, to test transfer to held-out mechanism families, we trained 16 CDFM models from scratch with a 10k-step training budget: one control using all 15 synthetic families and 15 leave-one-family-out models, each excluding one family. The results are detailed as follows:
| Excluded family | Full prior | Family excluded | Δ own family |
|---|---|---|---|
| CPT | 0.610 | 0.487 | -0.123 |
| discrete-ANM | 0.660 | 0.562 | -0.098 |
| PNL | 0.829 | 0.764 | -0.065 |
| RFF-hetero | 0.810 | 0.788 | -0.022 |
| physical-hetero | 0.687 | 0.665 | -0.022 |
| linear | 0.845 | 0.832 | -0.013 |
| physical | 0.689 | 0.676 | -0.013 |
| linear-hetero | 0.803 | 0.790 | -0.012 |
| time-lag | 0.756 | 0.744 | -0.012 |
| RFF | 0.827 | 0.815 | -0.012 |
| linear-ordinal | 0.630 | 0.620 | -0.011 |
| rounded | 0.770 | 0.762 | -0.008 |
| CAM | 0.759 | 0.755 | -0.004 |
| measurement-error | 0.777 | 0.774 | -0.002 |
| RFF-ordinal | 0.643 | 0.650 | +0.008 |
The results suggest that other generators provide sufficient coverage for most of the tested families, while CPT, discrete ANM, and PNL benefit more from explicit representation. We also note that some retained generators share lower-level functions. The results and the training setting are detailed in Appendix G.1.
Q1: a link to an anonymized code repository?
Thanks for asking about the code. The Anonymized code and reproduction instructions are provided at https://anonymous.4open.science/r/CDFM_ICLR-8FB7/.
Q2: A Limitations section?
Thanks for the suggestion. We have added a discussion of limitations in the appendix. We view this work as a first step toward a general-purpose foundation model for causal discovery. Our main focus is to establish the identifiability conditions under which such a model can operate and empirically validate the proposed learning framework across heterogeneous causal mechanisms. However, the current pretraining prior still covers a limited set of mechanisms. Extending this coverage and further investigating generalization to unseen mechanisms and real-world causal systems are important directions for future work.
Reply to Reviewer VqRD (3/3)
Q3: identification diagnostic experiments for the original frozen model (without the fine tuning) and competing amortized causal discovery methods?
Thanks for this suggestion. To examine whether the identification results depend on fine-tuning, we repeated all three diagnostic experiments using the released CDFM checkpoint without diagnostic-specific fine-tuning. We also evaluated FoundCause, AVICI, and TabCausal using their official pretrained checkpoints, together with classical methods where applicable. The experimental settings follow those in our original appendix, and all methods are evaluated on the same datasets.
(1) Functional asymmetry. We consider the same functional model , with varying nonlinearity and noise parameter . Each setting contains 60 datasets with .
| CDFM | AVICI | TabCausal | DirectLiNGAM | ||
|---|---|---|---|---|---|
| 0 | 1 | 48.3 | 50.0 | 51.7 | 50.0 |
| 0 | 2 | 100.0 | 100.0 | 100.0 | 100.0 |
| 0.5 | 1 | 13.3 | 0.0 | 0.0 | 1.7 |
| 0.5 | 2 | 100.0 | 75.0 | 100.0 | 0.0 |
| 1 | 1 | 100.0 | 0.0 | 0.0 | 11.7 |
| 1 | 2 | 100.0 | 5.0 | 75.0 | 0.0 |
The CDFM achieves near-chance orientation accuracy in the linear Gaussian setting and correctly identifies the direction when the nonlinearity or non-Gaussianity is sufficiently strong (), under both noise settings.
(2) V-structure identification. We evaluate accuracy on orientation accuracy from V-structures, using 400 DAGs per setting at .
| #V-structures | CDFM | TabCausal | AVICI |
|---|---|---|---|
| 1 | 85.9 | 85.4 | 65.5 |
| 2 | 88.8 | 86.1 | 74.9 |
| 3 | 90.5 | 90.1 | 81.2 |
Without fine-tuning, CDFM achieves higher orientation accuracy when more v-structures are present.
(3) Latent confounding. We report AUROC and directed-edge orientation accuracy under Gaussian and heavy-tailed noise.
| Noise | CDFM | FoundCause | AVICI | TabCausal |
|---|---|---|---|---|
| Gaussian () | 0.36 / 62% | 0.43 / 57% | 0.50 / 52% | 0.24 / 50% |
| Heavy-tailed () | 0.99 / 100% | 0.78 / 15% | 1.00 / 100% | 1.00 / 100% |
CDFM, AVICI, and TabCausal all perform substantially better under heavy-tailed noise than under Gaussian noise in this experiment.
Overall, these experiments show that several identification patterns observed in the original diagnostic experiments are already present in the pretrained CDFM without fine-tuning. Similar behaviors are also observed in competing amortized methods, although their sensitivity to different identification conditions varies. The results further highlight that theoretical identifiability remains necessary and that the foundation model satisfies it.
Q4: their thoughts of how CDFM handles uncertainty under observational non-identifiability, particularly when the posterior over causal graphs is multimodal?
Thanks for asking how the model represents competing graph explanations. In practice, CDFM can assign scores to both directions of an edge, serving as a measure of uncertainty. For example, our code provides a cycle-pruning function that prunes cycles according to confidence scores, which is one application of using ranking as confidence. We agree that explicitly modeling this uncertainty, especially the multimodal joint posterior over causal graphs, is important, and we consider it future work.
Thank you for the detailed rebuttal and additional experiments. While the new ablations and generalization tests are useful, I will maintain my original score. The gap between the variational formulation and actual training objective remains unresolved, and the proof of Corollary 1 still contains unjustified steps. I also remain unconvinced that the methodological novelty is sufficiently established or that the experiments demonstrate generalization beyond the shared synthetic framework.
Reply to Reviewer VqRD (1/3)
W1: Novelty.
Thanks for asking us to clarify CDFM's contribution. We start from identifiability across unknown mechanisms and use it to design a learning procedure that couples mechanism inference, reconstruction, and graph prediction.
W2: Ablations.
Thanks for asking for the ablation study. To evaluate the contribution of CDFM's three main components, including test-time adaptation, auxiliary reconstruction loss, and the mixture-of-experts (MoE) module, we conduct controlled ablations. Specifically, we retrained all variants from scratch for 10,000 steps on the same 15-mechanism prior, using identical initialization. All variants were evaluated on the same 3,600-task panel (15 mechanisms × 4 dimensions × 2 types of sample sizes × 30 datasets). Details are provided in Appendix G.2.
(1) Test-time adaptation. This is the main source of improvement. It increases AUROC by 0.0472 without auxiliary reconstruction and by 0.0506 with it. The similar gains show that adaptation is effective.
(2) Auxiliary reconstruction loss. Its contribution is smaller but consistent. The results show that the performance improves by +0.0051 without adaptation and +0.0085 with adaptation.
(3) Mixture of experts. We test it by removing the expert bank. The exclusion of the expert bank decreases the performance from 0.7437 to 0.7424, indicating a modest contribution from the MoE module. To further examine its role, we added a routing visualization to the appendix showing the evidence that the expert router indeed captures mechanism-specific structure.
W3 & W4: Variational formulation; Theoretical guarantees.
Thanks for asking what the encoder represents and how it enters the identification argument. In ELBO, Q(M∣X) motivates the learned mechanism representation, P(X∣G,M) supplies the reconstruction term, and P(G∣M) supplies the graph term. In CDFM, the implenmentation is indeed optimize the variational formulation above. To see this, first, the reconstruction term is implemented by the structure-aware data reconstruction component. Second, the graph inference are learned by direct supervised by the BCE loss. The remaining KL term in Eq. (6) usually serves as the regularization term, and we would like to omit it if we consider the uniform prior over the causal mechanism. We have revised this part of the paper to make it clearer.
For identification, Corollary 1 compares the full ELBO, including the reconstruction term. Appendix A.5 makes explicit that a sufficiently small variational gap at the true graph preserves its likelihood advantage. The structural decoder inherits this conclusion when its scores preserve the same separation. The argument compares the full bounds directly, which addresses the concern about the posterior identity.