acceptodds
Under review as a conference paper at ICLR 2027

CausalArena: Benchmarking Causal Discovery in the Foundation Model Era

Abstract

Causal discovery aims to uncover causal structures from data and is fundamental to scientific reasoning and intervention-based decision making. Its evaluation relies heavily on structural causal models (SCMs), which specify a causal graph together with the mechanisms that generate data, yet existing studies differ substantially in graph families, mechanisms, and evaluation protocols. The emergence of causal discovery foundation models (CDFMs) further complicates evaluation: performance may reflect not only causal discovery ability, but also overlap between pretraining environments and test SCMs, making results on fixed synthetic benchmarks difficult to interpret. We introduce CausalArena, a unified and evolvable benchmark for causal discovery under a common protocol. Synthetic SCMs supply breadth over structures and mechanisms; semantic operational SCMs provide human-auditable, semantically grounded environments beyond standard synthetic generators; and formula-grounded SCMs test discovery under explicit scientific mechanisms. Public real-world datasets provide an additional external-validity check. Experiments across classical, neural, and pretrained methods reveal substantial ranking shifts across SCM families and protocols, showing that strong performance in one benchmark regime does not reliably transfer to others. These results highlight benchmark diversity and pretraining–evaluation overlap as central challenges for evaluating causal discovery in the foundation model era.

Primary area:
causal reasoning
Keywords:
Causal Discovery, Causal Benchmark, Structural Causal Models, Interventional Data

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_shQ6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CausalArena, a benchmark for evaluating causal discovery methods in the foundation-model era, motivated by inconsistent evaluation protocols and potential overlap between pretraining and test distributions. It combines 1,200 executable SCMs—1,000 synthetic, 100 semantic operational, and 100 formula-grounded scientific models—under shared observational and interventional protocols, with an interface designed to accommodate future additions. The evaluation compares 18 classical, neural, and pretrained methods, supplemented by real-data experiments and analyses of graph structure, mechanisms, sample size, intervention design, and runtime. Results show that relative performance varies across evaluation settings and that advantages on generated data do not consistently carry over to real-data reference graphs.

Critical strengths

CausalArena combines SCMs across synthetic, operational, and formula-grounded settings and evaluates 18 methods using shared observational and interventional interfaces. This provides substantial practical value: researchers can assess methods across diverse environments without assembling separate evaluation pipelines, while retaining access to the underlying graphs and generation mechanisms.

Critical weaknesses

The models are trained on different datasets, and some training distributions may be closer to the evaluation distribution than others, raising concerns about the fairness of the comparison. Have the authors considered training different model architectures on the same dataset to compare their performance? Alternatively, could they fix the architecture and compare the training datasets used by different models?

Questions for authors

What specific statistical structures and causal mechanisms does semantically guided data construction introduce that are not already covered by conventional synthetic data generators? The neural networks evaluated in this benchmark do not appear to explicitly process the semantic information.

2 comments

Thank you for recognizing the practical value of evaluating 18 methods across diverse environments under shared protocols. We also appreciate your acknowledgment that researchers can use a unified evaluation pipeline while retaining access to the underlying graphs and generation mechanisms.

Q1. How fair is the comparison when pretraining distributions differ? Could models be retrained on the same data, or training datasets compared using a fixed architecture?

CausalArena aims to compare the complete causal discovery methods that users can actually apply. Fully ruling out the effects of overlap between pretraining and evaluation distributions is difficult without complete information about each model’s pretraining. As with evaluating released language models, each new model can be tested directly under the same evaluation protocol. Its architecture, pretraining data, and training recipe are treated as parts of the method.

Retraining all models with different architectures on the same dataset would still require choosing learning rates, curricula, and compute budgets separately for each model. A shared recipe may disadvantage some architectures, while replacing data generators removes contributions such as TabCausal’s broad causal priors and CauScale’s staged curriculum. Fixing a single architecture and training it separately on the datasets used by different models would make the comparison depend on that architecture’s capacity and compatibility with each training distribution. These controls are useful for targeted studies, but do not establish a universally neutral retraining standard. Repeating such training and tuning for every new model would also be costly.

Within this evaluation setup, our existing OOD analysis assesses sensitivity to documented pretraining coverage (Appendix E.2, Figs. 16–18). For example, FoundCause and TabCausal, two leading models on Synthetic observation-only F1, achieve about 60% and 22% higher F1 than the mean of four graph-search baselines outside documented coverage, averaged over four factor axes. The full-ID and full-OOD estimates in Fig. 18 show only small performance changes, with both models retaining their leading positions. These results support generalization beyond documented pretraining coverage.

To help interpret model performance when pretraining overlap is uncertain, Semantic and Formula provide a complementary, case-by-case view of model capabilities. Their named variables and explicit mechanisms let readers examine whether a method recovers the intended relationships in a concrete scenario. Our downloadable leaderboard makes this analysis interactive, providing evidence about specific successes and failures alongside aggregate scores.

To reduce dependence on a fixed evaluation pool, CausalArena is also designed as a living benchmark (Sec. 2.3; Appendix F). Its shared SCM interface allows Synthetic, Semantic, and Formula to expand with new graphs, mechanisms, and scenarios under the same evaluation protocol. Half of the SCMs are currently included in the anonymous review package and will be publicly released with the paper; the remainder will be released progressively after acceptance. We will continue adding new SCMs in subsequent versions to evaluate models beyond previously released cases.

Q2. What statistical structures and causal mechanisms does semantic construction introduce when the evaluated models do not read the text?

Semantics guide SCM construction before numerical data are generated. They constrain graph structure, mechanisms, measurement noise, and intervention targets according to the scenario. In the wildfire example (Fig. 6), a filter intervention changes indoor pollution, which is then measured through a noisy sensor. Conventional generators can implement these components; semantic guidance determines their combination and causal roles, shaping the joint distribution and intervention responses. The evaluated models encounter these choices through the generated numerical data.

Beyond guiding data construction, the semantic grounding of Semantic and Formula helps readers analyze each method’s strengths and failure modes case by case. In the leaderboard’s Semantic/Formula Analysis, readers can select a scenario and protocol, compare methods, and inspect recovered, missed, or extra edges using the variables’ meanings. Browse SCM Library exposes the graphs, equations, and intervention targets. Readers can identify scenarios resembling their application, understand which relationships each method handles well or poorly, and shortlist methods for validation on their own data.

The leaderboard was included in the anonymous package linked in the Reproducibility Statement. It is now packaged as a Leaderboard ZIP (7.1MB) for easier download. Extract it and open leaderboard/index.html.

Dear Reviewer shQ6,

As the discussion period is approaching its end, we would like to follow up on our rebuttal. Our response addresses comparison fairness and both retraining suggestions, highlights our existing OOD analysis, and clarifies the role of semantics in data construction and case-by-case analysis.

If our response addresses your concerns, we hope you will consider raising your score. Please let us know if anything needs further clarification. We appreciate your time and constructive feedback.

Best regards,
The Authors

Reviewer_44uHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes to us CasualArena, combining synthetic, semantically grounded or formula-grounded scientific SCMs into one benchmark aiming to evaluate classical, neural, and pretrained causal discovery methods under a unified observational/interventional protocol. Experiments on 18 methods over 1200 SCMs find that no method dominates on performance in all environments, and strong performance on synthetic benchmarks does not necessarily transfer to semantic, scientific, or real-world data, providing a foundation for continued evaluation as causal discovery methods and foundation models evolve.

Critical strengths
  1. The sheer scale and variety of SCMs included in CasualArena. This benchmark contains 1200 executable SCMs: 1000 synthetic, 100 semantic operational, and 100 formula-grounded scientific SCMs, providing broad coverage of graph structures, mechanisms and such.
  2. The paper evaluates 18 methods across classical, neural, and a variety of families under a shared protocol. It includes multiple ways to improve replicability and is a substantial research, namely the introduction of observation-only and observation-plus-intervention splits, SCM replications, multiple graph-recovery metrics, and extensive slicing for furthur detailed analyses.
  3. The benchmark reserves a held-out subset to keep faithfulness of data, and claims to add newly authored SCMs in future versions, which suggests a fine principle on benchmark building: researchers should prevent benchmark data leakage since contamination might occur in a week.
  4. This paper includes real world datasets with published graphs or interventions, such as Sachs, PetShop, and Causal Chambers, not only under their synthetic scenarios. This strengthens the practical relevance of the benchmark.
Critical weaknesses
  1. The proportions of the 1200 SCMs is clearly presented in the paper, but it does not discuss how half of each family is public, which results in 50 semantic and 50 scientific SCMs, might affect replication. It also lacks explicit statement that cross-family aggregations are unweighted and not dominated by the synthetic pool (Figure 11 in the appendix: Average F1 follows the table Avg. column, which averages only the family–split cells available for each method.).
  2. Table 2 as the main statistical result in this paper is a bit unclear, since it requires double-check the captions to see which metric each numerical value represent. It would be recommended if metrics like F1 / SHD is included in the column headers.
  3. The paper suggests that rankings shift across families, but the avg column mixes family difficulty, and there is no inference on correlations across different settings.
  4. The paper mentions varsortability and related artifacts but does not report them for the synthetic pool. Reporting such diagnostics would clarify whether synthetic results are driven by generator-specific features.
Questions for authors
  1. For synthetic SCMs as mentioned in Weaknesses, how do you ensure that the synthetic pool does not suffer from varsortability or sortability? And since real-world graphs are described as reference structures rather than uniquely verified causal truth, how should readers interpret their results? It would be best to furthur emphasize the methods' soundness.
  2. Could you clarify whether the strong synthetic performance of FoundCause and TabCausal is partly attributable to similarity between their pretraining distributions and the synthetic generator? The OOD analysis is helpful, but it relies on documented prior coverage.
2 comments

Thank you for recognizing the breadth of our 1,200 SCMs and the systematic evaluation of 18 methods under a shared protocol. We also appreciate your recognition of our reproducibility measures, real-data relevance, and design to limit benchmark contamination.

Q1. How does releasing half affect reproducibility?

We guarantee full reproducibility and will progressively release all remaining SCMs in versioned updates after acceptance. Providing half limits the risk of evaluation data entering pretraining. This staged release balances fair evaluation with reproducibility in a foundation-model arena. It is not intended to impede reproduction.

The 600 SCMs (500 Synthetic, 50 Semantic, 50 Formula) already support independent evaluation through the anonymous package linked in the Reproducibility Statement. The existing leaderboard, now packaged as a Leaderboard ZIP (7.1MB) for easier download, reports scores on the released half, reserved half, and all 1,200 SCMs.

As an additional check, with 99% confidence, the difference between each method’s F1 on the released half and the full dataset lies within [−0.0301, +0.0301] for all 18 methods simultaneously.

Q2. How are the averages in Table 2 and Fig. 11 computed?

Table 2 equally weights each method’s available family × protocol cells. Within each protocol, Synthetic (1,000 SCMs), Semantic (100), and Formula (100) contribute equally. Fig. 11 uses the same Avg. F1. We will clarify this in the Table 2 caption.

Q3. Table 2 presents the main statistical results, but its column headers do not identify the metrics.

We will label each result column F1 ↑ / SHD ↓, including Avg.

Q4. The Avg. column combines families of different difficulty, and rank correlations across settings are missing.

In addition to the overall average, we separately report and analyze performance by SCM family and protocol (Table 2 and Figs. 1 and 3). Avg. intentionally summarizes overall performance across a method’s supported settings. Equal weighting gives the families equal importance, without assuming equal difficulty.

Observation-only Spearman correlations across the same 18 methods are shown below, with pointwise 95% SCM bootstrap CIs.

Family pairF1SHD
Synthetic–Semantic.79 [.75, .83].33 [.29, .45]
Synthetic–Formula.62 [.57, .69].28 [.26, .42]
Semantic–Formula.79 [.72, .89].88 [.85, .92]

Rankings share common tendencies but vary across families and metrics, supporting the benchmarks’ complementarity in revealing methods’ strengths and weaknesses.

Q5. The paper mentions varsortability and related artifacts but does not report them for the synthetic pool.

Before standardization, observation-only varsortability was 0.3979, 0.5020, and 0.4914 for Synthetic, Semantic, and Formula (0.3981, 0.5068, and 0.5023 on the respective released halves), and 0.4189 across the 13 real datasets. All three SCM benchmarks use column-wise standardized observation-only inputs. Varsortability is approximately 0.5000 on each full pool and released half, so marginal variance provides essentially no causal ordering signal in these evaluations.

Q6. How should readers interpret results on real-world datasets with reference graphs?

Real-data reference graphs may differ from the true causal structures, and this uncertainty is hard to eliminate. We follow established practice for comparability with prior work. Real measurements remain indispensable for testing methods on actual systems. We designed Semantic and Formula precisely to address this limitation: their SCMs preserve operational or scientific meaning and provide ground-truth DAGs known by construction for the generated data.

Q7. Could pretraining similarity explain the strong Synthetic performance of FoundCause and TabCausal?

Pretraining similarity may contribute to Synthetic scores, and fully ruling out overlap is difficult in foundation-model evaluation. We address this concern through complementary diagnostics and benchmark design.

Our existing OOD analysis (Appendix E.2) evaluates categories outside documented pretraining coverage. In Fig. 17, FoundCause and TabCausal achieve about 60% and 22% higher F1 than the mean of four graph-search baselines, averaged over four factor axes. FoundCause’s relative lead remains stable, while TabCausal’s narrows by only about 4 percentage points. Their ranks are preserved in Fig. 18’s full-ID and full-OOD estimates.

Semantic and Formula broaden evaluation beyond a fixed synthetic generator. Their semantic grounding enables case-by-case analysis of recovered relationships and failure modes. The construction pipeline also lets us add newly authored scenarios as training distributions evolve, reducing reliance on a fixed evaluation pool.

Dear Reviewer 44uH,

As the discussion period is approaching its end, we would like to follow up on our rebuttal. Our response addresses reproducibility, cross-family rank correlations, varsortability, real-data reference graphs, and pretraining overlap, along with the averaging and table-labeling issues.

Please let us know if any concerns remain or need further clarification. We appreciate your time and constructive feedback.

Best regards,
The Authors

Reviewer_rcGQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a benchmark for causal discovery algorithms. The benchmark comprises SCMs sampled from (“classical”) synthetic simulators, as well as LLM-based semantic cases, partially with mechanisms based on scientific formulas.

The main contribution is the proposed semantic, language based causal graphs/SCMs.

A set of proposed semantic/language based graphs is evaluated through a multi-step LLM-pipeline.

Finally, existing causal discovery algorithms are evaluated on the “classical” synthetic simulator, the semantic simulator, and some real-world datasets.

Critical strengths

The identified issue of better evaluation of causal discovery methods is pressing, and I also agree that the advent of amortized causal discovery methods makes relying on synthetic data an even bigger issue than before. The paper could therefore be very important for the field.

The idea to use LLM-based simulators as an alternative seems novel, interesting and promising. Also the proposed agentic pipeline seems reasonable. This aspect is consequential for my assessment as it is the central contribution of the paper beyond existing or standard evaluation setups.

Critical weaknesses

I see the main weakness of the paper as not focussing enough on the core novelty in form of the LLM-based semantic simulators and the corresponding pipeline. The “classical” synthetic evaluation is, to me, nothing really new. I am also not convinced that this part of the paper has real potential for addressing the issue of just training on the simulator (sure, it is nice to have an extendable codebase, but this should be standard, not a contribution). The paper is therefore filled with existing evaluation approaches while, the, to me, interesting part is not further explained/justified/evaluated/ablated and hidden in the Appendix.

Questions for authors

Why don't the authors put the full focus or at least the main focus of the paper on the, to me, novel pipeline of using "semantic SCMs" for evaluation?

To my understanding, the authors do not utilize the internal standardization in Ormaniec, Weronika, et al. "Standardizing structural causal models." International Conference on Learning Representations. Vol. 2025. 2025.?

4 comments

Thank you for recognizing the pressing need for better evaluation of causal discovery, particularly for amortized methods, and the potential importance of this work for the field. We also appreciate your assessment of LLM-based semantic simulators as novel and promising, and of the proposed agentic pipeline as reasonable.

Q1. Why does the main text give more space to conventional synthetic evaluation, and how is the Semantic and Formula SCM pipeline evaluated?

We agree that the agentic construction of Semantic and Formula SCMs is a core contribution of this work. Our broader goal is to build a comprehensive evaluation framework for causal discovery, covering both foundation models and conventional methods. Synthetic is an essential component of this framework, even though semantic construction is the more novel contribution.

We also devote space in the main text to Synthetic evaluation for two reasons. First, synthetic simulators underpin much of the existing evaluation of causal discovery methods. This connects CausalArena to prior evaluation practice and provides a familiar basis for comparison. Second, large scale comparisons need many datasets and controlled variation to summarize performance by graph structure, dimension, mechanism, noise, sample size, and intervention. Semantic and Formula each contain 100 scenarios, with structures, mechanisms, and noise chosen to fit the scenarios. These factors cannot be varied freely and independently while preserving that grounding. Subdividing these pools would yield small, uneven groups, making factor effects hard to separate from scenario differences. Synthetic is therefore better suited to these systematic comparisons, while Semantic and Formula support case-by-case analysis.

Making evaluation more comprehensive is itself a contribution of our Synthetic benchmark. It covers 1,000 SCMs across 10 graph families and five dimensions, with 16 mechanism, 14 root-distribution, and 15 noise families. For example, Fig. 13 shows that CauScale’s observation-only F1 rises from 0.35 to 0.48 as dimension increases from 10 to 100, while almost all other methods decline. Such comparisons reveal differences in scaling that an overall rank would hide.

Semantic and Formula complement this broad statistical coverage with case-by-case analysis. Named variables, measurements, and equations help readers interpret edge errors, understand a method’s capabilities and failure modes, and identify suitable methods to validate for a particular application. In our photovoltaic scenario, irradiance raises power, while higher cell temperature reduces power at fixed irradiance. In all five observation-only repeats, FoundCause recovers irradiance → power but misses cell temperature → power in both subarrays; GIES recovers both (leaderboard).

The submission already describes and analyzes Semantic and Formula. Secs. 3.2–3.4 and Appendix B.4 cover construction and validation; Figs. 1, 3, and 4 report results across all four benchmarks. The construction ablation (Fig. 9; Table 19) raises mean LLM-judge scores from 3.80 (seed) to 3.87 (grounding), 4.37 (planning), and 4.48 (review). Figs. 5–6 show examples and distributions; Appendix D.2 (Fig. 8) reports skeleton probes, and Appendix E.3 (Fig. 19; Table 5) analyzes domains and method behavior.

To help readers use Semantic and Formula for interactive, scenario-level analysis, we provided an interactive leaderboard as a supplement to the paper. It was included in the anonymous package linked in the Reproducibility Statement and is now also packaged as a ZIP for easier download. In Semantic/Formula Analysis, readers can choose a released scenario, compare methods, and inspect recovered, missed, or extra edges with semantic meaning. Browse SCM Library exposes graphs, variable definitions, equations, and intervention targets. To access the full interactive leaderboard, download and extract the Leaderboard ZIP (7.1MB), then open leaderboard/index.html in a browser.

We will revise the presentation to clarify the overall design and highlight the semantic contribution. We will move the detailed Synthetic inventory (Sec. 3.1) to Appendix B.2 and consolidate the scaling, intervention, and runtime summaries in Appendix D. This will make room in Sec. 3.4 for the wildfire smoke example (Fig. 6) and the construction ablation (Fig. 9 and Table 19). We will also explain the leaderboard analyses more clearly.

Q2. Does the Synthetic generator use the internal standardization proposed by Ormaniec et al. (2025)?

Your understanding is correct. We standardize each column after generating the full dataset. In Ormaniec et al.’s approach, each variable is standardized during generation, before it is passed to downstream mechanisms. We will clarify this distinction in the Synthetic benchmark description.

Dear authors,

thank you for your reply!

I am now more on your side in terms of seeing the need for comprehensive synthetic evaluations, which was a major concern.

I think the paper is a meaningful, albeit maybe not perfect, contribution to the field of causal discovery.

However one concern that remains partly, also after reading your reply to Reviewer 44uH is that of var-sortability in your benchmark. I think your answer that

Before standardization, observation-only varsortability was 0.3979, 0.5020, and 0.4914 for Synthetic, Semantic, and Formula (0.3981, 0.5068, and 0.5023 on the respective released halves), and 0.4189 across the 13 real datasets.

does address this issue, but if you could you perhaps reassure me why var-sortability is not an issue in your benchmark?

I would be happy to increase my score if you can help me with this concern.

Thank you for following up on this concern. The values quoted in your comment were measured before standardization. As noted in our response to Reviewer 44uH, Synthetic, Semantic, and Formula use column-wise standardized inputs, with post-standardization varsortability 0.5000 on each pool. This standardization removes the marginal-variance ordering signal.

Reisach et al. (2021), who introduced varsortability in Beware of the Simulated DAG! Causal Discovery Benchmarks May Be Easy to Game, used two baselines to assess the benefit of variance-based ordering:

  • SortnRegress sorts variables by increasing marginal variance and then applies regression.
  • RandomRegress uses a random variable order and then applies the same regression procedure.

Their performance gap therefore measures the benefit of variance-based ordering over random ordering. When the two methods perform similarly, variance-based ordering contributes almost no additional benefit in this comparison. Our additional experiments below follow this approach to assess whether variance-based ordering improves performance on our benchmarks.

Here, we first introduce a toy example to explain why standardization removes the variance-ordering signal. Consider the chain A→B→CA\rightarrow B\rightarrow C:

A=ϵA,B=2A+ϵB,C=2B+ϵC,A=\epsilon_A,\qquad B=2A+\epsilon_B,\qquad C=2B+\epsilon_C,

where the noises are independent, zero-mean, and have unit variance. Then

\operatorname{Var}(B)=4+1=5,\qquad \operatorname{Var}(C)=4\times5+1=21.$$ **In this case, sorting by increasing variance recovers the causal order.** After standardizing each variable, $$\widetilde A=A,\qquad \widetilde B=\frac{B}{\sqrt5},\qquad \widetilde C=\frac{C}{\sqrt{21}},$$ the same system becomes $$\widetilde A=\epsilon_A,\qquad \widetilde B=\frac{2}{\sqrt5}\widetilde A+\frac{\epsilon_B}{\sqrt5},\qquad \widetilde C=\frac{2\sqrt5}{\sqrt{21}}\widetilde B+\frac{\epsilon_C}{\sqrt{21}}.$$ Thus, $$\operatorname{Var}(\widetilde A)=1,\qquad \operatorname{Var}(\widetilde B)=\frac45+\frac15=1,\qquad \operatorname{Var}(\widetilde C)=\frac{20}{21}+\frac1{21}=1.$$ Varsortability assigns half credit to equal variances. All three directed paths receive half credit, so $$v=\frac{0.5+0.5+0.5}{3}=0.5.$$ **Thus, methods can no longer rely on the variables’ marginal variances to determine their causal order.** We additionally ran SortnRegress and RandomRegress on our actual evaluation inputs, keeping regression and scoring procedures the same: | Benchmark | RandomRegress F1 | SortnRegress F1 | 99% upper confidence bound on absolute mean F1 difference | |---|---:|---:|---:| | Synthetic | 0.2815 | 0.2794 | 0.0062 | | Semantic | 0.2743 | 0.2685 | 0.0178 | | Formula | 0.2232 | 0.2256 | 0.0125 | | Real | 0.1994 | 0.1584 | 0.2127 | Synthetic, Semantic, and Formula contain 1,000, 100, and 100 SCMs, respectively. We calculated benchmark-specific confidence bounds using a paired, stratified bootstrap. The small average F1 gaps and narrow bounds support that variance-based ordering offers little benefit on these three benchmarks. Real contains only 13 datasets. Its much wider confidence interval gives a less precise estimate, we do not consider it. Together, standardization and these results support that **the variance-ordering shortcut associated with varsortability provides almost no advantage on the three benchmarks**. We will also include this additional experiment in the revised manuscript to make this point clearer to readers.

Thank you for your detailed explanation. This addresses my second concern and I raised my score accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.