- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces CausalArena, a benchmark for evaluating causal discovery methods in the foundation-model era, motivated by inconsistent evaluation protocols and potential overlap between pretraining and test distributions. It combines 1,200 executable SCMs—1,000 synthetic, 100 semantic operational, and 100 formula-grounded scientific models—under shared observational and interventional protocols, with an interface designed to accommodate future additions. The evaluation compares 18 classical, neural, and pretrained methods, supplemented by real-data experiments and analyses of graph structure, mechanisms, sample size, intervention design, and runtime. Results show that relative performance varies across evaluation settings and that advantages on generated data do not consistently carry over to real-data reference graphs.
Critical strengths
CausalArena combines SCMs across synthetic, operational, and formula-grounded settings and evaluates 18 methods using shared observational and interventional interfaces. This provides substantial practical value: researchers can assess methods across diverse environments without assembling separate evaluation pipelines, while retaining access to the underlying graphs and generation mechanisms.
Critical weaknesses
The models are trained on different datasets, and some training distributions may be closer to the evaluation distribution than others, raising concerns about the fairness of the comparison. Have the authors considered training different model architectures on the same dataset to compare their performance? Alternatively, could they fix the architecture and compare the training datasets used by different models?
Questions for authors
What specific statistical structures and causal mechanisms does semantically guided data construction introduce that are not already covered by conventional synthetic data generators? The neural networks evaluated in this benchmark do not appear to explicitly process the semantic information.
2 comments
Dear Reviewer shQ6,
As the discussion period is approaching its end, we would like to follow up on our rebuttal. Our response addresses comparison fairness and both retraining suggestions, highlights our existing OOD analysis, and clarifies the role of semantics in data construction and case-by-case analysis.
If our response addresses your concerns, we hope you will consider raising your score. Please let us know if anything needs further clarification. We appreciate your time and constructive feedback.
Best regards,
The Authors
Thank you for recognizing the practical value of evaluating 18 methods across diverse environments under shared protocols. We also appreciate your acknowledgment that researchers can use a unified evaluation pipeline while retaining access to the underlying graphs and generation mechanisms.
Q1. How fair is the comparison when pretraining distributions differ? Could models be retrained on the same data, or training datasets compared using a fixed architecture?
CausalArena aims to compare the complete causal discovery methods that users can actually apply. Fully ruling out the effects of overlap between pretraining and evaluation distributions is difficult without complete information about each model’s pretraining. As with evaluating released language models, each new model can be tested directly under the same evaluation protocol. Its architecture, pretraining data, and training recipe are treated as parts of the method.
Retraining all models with different architectures on the same dataset would still require choosing learning rates, curricula, and compute budgets separately for each model. A shared recipe may disadvantage some architectures, while replacing data generators removes contributions such as TabCausal’s broad causal priors and CauScale’s staged curriculum. Fixing a single architecture and training it separately on the datasets used by different models would make the comparison depend on that architecture’s capacity and compatibility with each training distribution. These controls are useful for targeted studies, but do not establish a universally neutral retraining standard. Repeating such training and tuning for every new model would also be costly.
Within this evaluation setup, our existing OOD analysis assesses sensitivity to documented pretraining coverage (Appendix E.2, Figs. 16–18). For example, FoundCause and TabCausal, two leading models on Synthetic observation-only F1, achieve about 60% and 22% higher F1 than the mean of four graph-search baselines outside documented coverage, averaged over four factor axes. The full-ID and full-OOD estimates in Fig. 18 show only small performance changes, with both models retaining their leading positions. These results support generalization beyond documented pretraining coverage.
To help interpret model performance when pretraining overlap is uncertain, Semantic and Formula provide a complementary, case-by-case view of model capabilities. Their named variables and explicit mechanisms let readers examine whether a method recovers the intended relationships in a concrete scenario. Our downloadable leaderboard makes this analysis interactive, providing evidence about specific successes and failures alongside aggregate scores.
To reduce dependence on a fixed evaluation pool, CausalArena is also designed as a living benchmark (Sec. 2.3; Appendix F). Its shared SCM interface allows Synthetic, Semantic, and Formula to expand with new graphs, mechanisms, and scenarios under the same evaluation protocol. Half of the SCMs are currently included in the anonymous review package and will be publicly released with the paper; the remainder will be released progressively after acceptance. We will continue adding new SCMs in subsequent versions to evaluate models beyond previously released cases.
Q2. What statistical structures and causal mechanisms does semantic construction introduce when the evaluated models do not read the text?
Semantics guide SCM construction before numerical data are generated. They constrain graph structure, mechanisms, measurement noise, and intervention targets according to the scenario. In the wildfire example (Fig. 6), a filter intervention changes indoor pollution, which is then measured through a noisy sensor. Conventional generators can implement these components; semantic guidance determines their combination and causal roles, shaping the joint distribution and intervention responses. The evaluated models encounter these choices through the generated numerical data.
Beyond guiding data construction, the semantic grounding of Semantic and Formula helps readers analyze each method’s strengths and failure modes case by case. In the leaderboard’s Semantic/Formula Analysis, readers can select a scenario and protocol, compare methods, and inspect recovered, missed, or extra edges using the variables’ meanings. Browse SCM Library exposes the graphs, equations, and intervention targets. Readers can identify scenarios resembling their application, understand which relationships each method handles well or poorly, and shortlist methods for validation on their own data.
The leaderboard was included in the anonymous package linked in the Reproducibility Statement. It is now packaged as a Leaderboard ZIP (7.1MB) for easier download. Extract it and open
leaderboard/index.html.