acceptodds
Under review as a conference paper at ICLR 2027

CausalMix: Defining and Benchmarking Hybrid Causal Discovery

Abstract

Classical causal discovery uses numerical observations and interventions, while large language models (LLMs) also bring knowledge associated with variable names. A graph-recovery score alone does not reveal which information source supported the prediction. We introduce **CausalMix**, a simulation and evaluation suite for hybrid causal discovery that compares names-only, data-present, anonymized, and classical data-only conditions under controlled graph and data budgets. The benchmark separates output validity, name effects, local responses to data, and accurate recovery. Our evaluation spans six reference networks and four synthetic graph configurations with 5–50 variables. On the evaluated reference graphs, recovery is strongly name-mediated, while gains from numerical evidence are small or unstable. These name effects may reflect domain knowledge or benchmark recall; our controls do not distinguish the two. We also post-train Qwen3-4B with supervised fine-tuning (SFT) followed by group relative policy optimization (GRPO). Post-training improves output validity, but adding data improves Sachs recovery only when names are anonymized, and recovery on 20–50-variable graphs remains far below a classical interventional baseline. CausalMix provides a reproducible protocol for testing whether hybrid methods use supplied evidence reliably, rather than inferring evidence use from aggregate scores.

Primary area:
causal reasoning
Keywords:
Causal discovery, Large language model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ixM5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies hybrid causal discovery methods (LLM + statistical methods) by introducing CausalMix, an evaluation framework that measures the causal graph recovery contributions of LLMs and numerical data separately. The framework evaluates causal discovery methods under different semantic conditions, data budgets, and data representations, with classical causal discovery methods as baselines. Additionally, the work introduces a causal discovery benchmark simulator and proposes a post-training procedure using SFT and GRPO for LLM-based causal discovery. The experiments cover several LLMs and causal graphs of different sizes. The results show that LLMs can recover considerable graph structures based on variable names alone, while adding numerical data does not consistently improve their performances. The post-training procedure also improves output validity.

Critical strengths
  1. The paper provides a systematic evaluation framework for hybrid causal discovery methods, allowing examining the performances contributed by semantic and numerical evidence separately.
  2. The framework includes different data representations and interventional data, which could be helpful for understanding the limitations of existing methods and developing more effective hybrid causal discovery approaches.
Critical weaknesses
  1. The paper's novelty and overall contribution are limited. The definition of hybrid causal discovery is straightforward, and the idea of evaluating the contributions of semantic and numerical evidence separately has been explored in existing works. I acknowledge that the proposed evaluation framework provides a more systematic combination of these components, however, the underlying idea is not fundamentally new. Furthermore, fine-tuning LLMs for causal discovery has also been studied in previous works. Although the paper introduces GRPO, its application to graph recovery does not appear to provide significant methodological novelty.
  2. The experiments may not sufficiently reflect practical causal discovery tasks. Although the paper evaluates graphs with up to 50 variables, many experiments still rely on old benchmarks, especially Asia, Earthquake, and Cancer. These datasets are relatively small, and the LLMs may have memorised their structures. More importantly, some of those old benchmarks contain knowledge misaligned with the domains, needing further discussion of their fitness for evaluating LLM-based methods. The larger-scale experiments provide additional results, but the synthetic graph configurations remain limited. I believe more experiments on new causal graphs would be helpful to demonstrate the generality and usefulness of the evaluation framework.
  3. The effectiveness of the proposed post-training procedure is not sufficiently demonstrated. The experiments show improvements in output validity, but the improvements in graph recovery and numerical evidence usage are limited. For instance, on Sachs, adding numerical data to the SFT+GRPO model provides almost no F1 improvement with original variable names. Moreover, the training graphs overlap with several evaluation graphs, while the held-out-family experiments only evaluate SFT rather than GRPO. The comparisons between SFT and GRPO also do not fully isolate the contribution of GRPO. Therefore, it remains unclear whether the proposed training procedure can significantly improve LLMs' causal discovery capabilities.
  4. Another concern is whether the experiments sufficiently demonstrate the usefulness of the evaluation framework for hybrid causal discovery. In practice, numerical evidence may be unreliable due to data quality, where semantic knowledge could help correct the results. However, the experiments mainly consider data generated from known Bayesian networks under strong assumptions. I believe more experiments under such conditions would be helpful to demonstrate the usefulness of the framework for evaluating the complementary strengths of semantic and statistical evidence.
Questions for authors

See the weakness part.

Reviewer_9Q74OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CausalMix, a benchmark for causal discovery that evaluates whether LLMs recover causal graphs from semantic information (variable names), from numerical evidence (observational and interventional data), or from both. The benchmark compares four settings on several reference networks and synthetic graphs: names only, original names with data, anonymized names with the same data, data only. The experiments demonstrate that LLM performance is strongly driven by variable names, while additional numerical evidence yields small and inconsistent improvements. The authors also post-train Qwen3-4B with SFT followed by GRPO. This improves output validity but provides little evidence of improved data-driven causal discovery.

Critical strengths

The question of how LLMs can contribute to causal discovery, by combining prior knowledge with observational and interventional data, is relevant. The goal of separating the contributions of these two sources, as well as the effect of how the data is presented, is worthwhile.

The systematic comparison of semantic-only, anonymized, data-present, and data-only conditions is well motivated and appears sound.

Critical weaknesses

The main problem is that the core experiments use very well-known Bayesian networks from bnlearn, with which LLMs are likely already familiar. The prompts even mention the network identifier explicitly (e.g., “a Bayesian network named cancer”). Strong names-only performance is therefore unsurprising and may largely reflect benchmark memorization rather than domain reasoning. This makes it difficult to judge whether the conclusions generalize beyond these specific graphs.

SFT and GRPO repeatedly train on samples from a small number of fixed causal graphs, several of which (Cancer, Earthquake, Asia, Sachs) are also used for evaluation. The model may therefore learn these particular graph structures rather than a transferable causal-discovery procedure.

Questions for authors

Did the authors consider evaluating on semantically meaningful graphs that are less well known than the included bnlearn networks?

What exactly does the reasoning trace in the first SFT stage contain, and how is it generated? The paper states that Stage 2 rationales are generated by Qwen2.5-72B, but the content of the <think> block in Stage 1 is not specified.

Reviewer_xez8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper formalizes hybrid causal discovery (HYBRID-CD): recovering a DAG from semantic side information S (variable names/descriptions) and numerical evidence (D_obs, D_int), with classical algorithms and name-only prompting as limit cases. Its argument is that a headline graph-recovery score cannot be attributed to an information source, so every result must be reported alongside three matched references: a names_only control at N=M=0, an anonymized-name control at the same budget, and a data-only classical baseline on the same samples.

CausalMix V1 operationalizes this as a cell expander: graphs × semantic-cue conditions × budgets (N,M) × data representations (summary histograms vs tabular rows) × models × baselines × seeds, where matched cells differ in exactly one factor. The release has 14 graph instances in three tiers — six bnlearn reference graphs (Cancer-5, Earthquake-5, Asia-8, Sachs-11, Child-20, Alarm-37), six Synthetic-25 ENCO topology controls, two large challenge graphs — with Chain/Jungle at 25 and 50 nodes. Three studies run on it: an attribution study over ten LLMs (GPT-5, Qwen2.5, Qwen3, Llama-3.1 families) with PC/GES/ENCO references; five benchmark-adapted LLM-CD strategies on Sachs; and a post-training study that SFT+GRPO-trains Qwen3-4B-Thinking on data from the same simulator against a ground-truth verifier. Headline claims: recovery is strongly name-mediated, data gains are small or unstable, post-training improves output validity (331/360 vs 298/360) but improves Sachs recovery only under anonymization (+0.134 vs −0.002), and 20–50-node recovery (F1 0.005–0.115) stays far below ENCO (0.609–0.790).

Critical strengths

The limitations reporting and the released control infrastructure are genuinely good. Appendix B is specific where most papers are vague: it states that each synthetic size/type cell uses one generating DAG; that anonymization "measure[s] name effects without separating domain knowledge from benchmark recall" and "changes the naturalness of prompts"; that both SFT stages contain gold graph targets with "no schema-only supervision control," so "output-format learning, target imitation, and pretraining familiarity remain incompletely separated"; that the 50-node runs use format instructions while 20–37-node runs enforce format during generation, so "their difference is not a controlled estimate of graph-size effects"; and that an output-token cutoff "does not show that a model would fail with a larger generation budget." H.2 flags that at 50 nodes "all 40 no-data outputs are invalid, so the positive data-use gains there compare against zero and do not show successful use of data" — a disclosure that voids one of its own positive results. The distinction between unavailable (context rejection, API failure) and invalid generated outputs is the right one and is carried consistently through Table 6 and E.3. Combined with the released cell expander, prompt templates, and per-row source paths, this is infrastructure the community can use.

Critical weaknesses
  1. The attribution study runs at the recovery ceiling on three of its four graphs, which mechanically produces the paper's headline finding — and on the one graph with headroom the finding reverses. For any cell where names_only F1 is already 1.00, the data gain (F1 with data minus F1 without) can only be ≤ 0 by arithmetic. Figure 2 reports that of the 15 displayed names_only cells on Cancer, Earthquake and Asia, "seven are exact recoveries and ten have F1 ≥ 0.75," and Table 7 shows GPT-5-Mini at exactly 1.00 on all three graphs under names-only, original-summary and original-tabular alike. So "gains from numerical evidence are small or unstable" is partly guaranteed by construction on three of the four attribution graphs. The remaining graph is Sachs, and there the paper's only uncertainty-quantified analysis (Tables 9–10) points the other way: the data gain is positive with a bootstrap interval excluding zero for three of the five well-sampled models — GPT-5.2-Pro +0.107 [+0.057, +0.158], Qwen3-4B +0.176 [+0.090, +0.249], Qwen2.5-72B +0.112 [+0.021, +0.203] — with one near-zero (+0.036 [−0.142, +0.214]) and only GPT-5-Mini negative (−0.050). Over the same cells the anonymization drop is positive-with-interval-excluding-zero for exactly one of six estimable models (GPT-5.2-Pro +0.115), is exactly 0.000 for Qwen3-4B and Qwen2.5-14B, and is negative for Qwen2.5-72B (−0.031) and Qwen3-30B-A3B (−0.222). Table 11 sharpens this: GPT-5-Mini's anonymized Sachs recovery improves from 0.52 to 0.62 as N goes 1000 → 5000 while original-name recovery declines 0.51 → 0.47, i.e. names impede data use for that model. Against this, the abstract's "recovery is strongly name-mediated" rests on a sign count — "reduces valid-output F1 in 26 of 33 supported comparisons" — carrying no magnitudes or intervals, computed valid-only under a ≥3-valid filter that drops 7 of 40 candidate cells non-randomly, and dominated by the saturated graphs. The construct is also unstable under the suite's own second axis: in Table 7 the Cancer name effect is 1.00 − 0.51 = 0.49 under summary and 1.00 − 1.00 = 0.00 under tabular, same model, same graph, same data. A quantity that moves by 0.49 under a serialization switch is not yet a measurement of name dependence. Finally, the axis that could settle any of this is structurally absent rather than merely unrun: the only non-memorized graphs have neutral source labels, so "Chain/Jungle appear only under anonymized names," and the name axis is never crossed with the novel-graph axis anywhere in the suite.

  2. The post-training contribution does not establish what it is offered for, and the cross-method comparison that frames it is not matched by the paper's own definition. Table 13 localizes the SFT+GRPO effect precisely: the large gains are two 5-node graphs with original names (Cancer 0.544 → 0.926, Earthquake 0.550 → 0.990), both in the training families; Asia rises 0.264 → 0.505; and from Sachs upward the effect vanishes (Sachs 0.259 → 0.287, Child 0.081 → 0.086, Alarm 0.035 → 0.033), with five of sixteen conditions declining. After training a model on this simulator against a ground-truth verifier, anonymized recovery on a five-variable graph is 0.332 (Cancer) and 0.121 (Earthquake), and every graph ≥ 20 nodes sits at 0.005–0.115. The paper also reports no controlled comparison isolating GRPO: Table 14 places two-stage SFT beside SFT+GRPO and SFT is better on both 50-node graphs in F1 and in validity (Chain-50 0.024 at 19/20 vs 0.011 at 9/20; Jungle-50 0.011 at 14/20 vs 0.005 at 9/20), while the caption concedes the pipelines "differ in training data, objectives, duration, and checkpoint selection, so these comparisons do not isolate the training algorithm." With no schema-only supervision control either, the validity improvement is not separable from gold-target imitation — as B states. What remains is the ENCO gap, and that comparison fails the suite's own matched-cell standard: Table 13's note says "ENCO values retain the existing 20-run scaling reference at the same nominal budget; sample arrays are not verified identical," echoed in B ("numerical references, not strictly identical-dataset comparisons"), while §3 warns that "matching data budgets alone does not establish that two methods received identical sample arrays." No classical reference is reported at all for the four small graphs (Table 13's dashes), which are exactly the graphs where the LLMs do well. Compounding the roster claim: Table 10 shows the primary Sachs contrasts are not estimable for four of the ten evaluated models (0 paired runs for Qwen2.5-7B and all three Llama-3.1 checkpoints) and rest on 2 paired runs for Qwen3-30B-A3B, with Table 5 showing many 0/5 validity cells, and the reported validity itself is taken "after top-valid-rate prompt-config selection" — a selection on the outcome inside a suite whose purpose is clean attribution.

Questions for authors
  1. Ceiling effect in the attribution study. On Cancer, Earthquake and Asia the names_only F1 is 1.00 for GPT-5-Mini and GPT-5.2-Pro (Table 7; 7 of 15 cells in Figure 2 are exact recoveries), so the data gain on those cells cannot be positive. How much of the "data gains are small or unstable" conclusion survives if you restrict to cells with headroom - say names_only F1 below 0.8 - and report the data gain only there? This seems to me the single most important re-analysis, and it can be done on existing runs.

  2. Direction of the headline claim on the one non-saturated graph. On Sachs at the primary budget, Tables 9 and 10 give data gains of +0.107 [+0.057, +0.158], +0.176 [+0.090, +0.249] and +0.112 [+0.021, +0.203] for GPT-5.2-Pro, Qwen3-4B and Qwen2.5-72B, while the anonymization drop excludes zero for GPT-5.2-Pro only and is 0.000 for Qwen3-4B and Qwen2.5-14B and negative for Qwen2.5-72B (-0.031) and Qwen3-30B-A3B (-0.222). Table 11 shows GPT-5-Mini's anonymized Sachs F1 rising 0.52 to 0.62 with N while its original-name F1 falls 0.51 to 0.47. On what evidence does the abstract's "recovery is strongly name-mediated, while gains from numerical evidence are small or unstable" rest, once these are taken into account? Can you report the 26-of-33 anonymization count with per-comparison magnitudes and intervals, split by whether names_only was saturated?

  3. Is the name effect well defined? In Table 7 the Cancer name effect is 0.49 under summary serialization and 0.00 under tabular, with the same graph, data and model. If a construct moves by 0.49 under a format switch, what is the recommended way for a future paper to report "the name effect" as a single number, and should the benchmark instead report it jointly over representations?

  4. Crossing the two axes. The name axis exists only on six long-published bnlearn graphs; the non-memorized Chain/Jungle graphs have neutral labels and therefore appear only under anonymized names. Is there any plan to attach plausible, non-public semantic names to the generated graphs - or to use post-cutoff scientific graphs - so that name effects and structural novelty can be varied independently? As released, I do not think the suite can measure what it is designed to measure.

  5. Statistical power of the primary contrasts. The flagship contrasts use 5 paired runs (Tables 9, 10), with 20,000 bootstrap resamples drawn from those 5 points, and one model's reported -0.431 data gain comes from 2 paired runs. Four of the ten models have 0 estimable paired runs. Given this, is the ten-model roster a defensible basis for the claims in §5.2, and could you raise the replicate count on the Sachs cells, which are cheap relative to the 50-node runs?

  6. Isolating GRPO. Table 14 shows two-stage SFT beating SFT+GRPO on both 50-node graphs in F1 and validity (0.024 at 19/20 vs 0.011 at 9/20; 0.011 at 14/20 vs 0.005 at 9/20). Is there any setting in which GRPO is shown to help over its own SFT initializer under matched data, duration and checkpoint selection? And given that both SFT stages use gold adjacency targets with no schema-only control, what supports attributing the 331/360 validity figure to anything beyond target imitation on the training families?

  7. The ENCO comparison. Table 13 states that ENCO's sample arrays "are not verified identical" to the LLM runs, and §3 says budget matching alone does not establish identical sample arrays. Since the gap to ENCO is the paper's main quantitative claim about scale, can you rerun ENCO on the exact released arrays? And can you add PC/GES/ENCO references for Cancer, Earthquake, Asia and Sachs, where Table 13 currently has dashes and where the LLMs score highest?

  8. Figure 5 backbone. Which LLM backs each of the five adapted LLM-CD strategies, and is it the same model that produces the names_only reference they are compared against? If the adapted strategies run on a weaker backbone than GPT-5.2-Pro, then "four of the five strategies fail to exceed names_only" is confounded by backbone rather than by strategy, and Causal-LLM at 0.49 against a floor near 0.50 is within that confound.

  9. Prompt-config selection. Table 5 reports validity "after top-valid-rate prompt-config selection" per graph, model and representation. Are the F1 contrasts computed in those selected configurations? If so, the matched cells are matched on graph and data but not on prompt configuration across models, and the selection criterion is an outcome - how should a reader interpret the cross-model comparisons under that procedure?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Withdrawal of Submission

Dear Reviewers and Area Chair,

We have decided to withdraw our submission. Thank you for the time and effort you spent reviewing our work. We appreciate your constructive feedback and will use it to improve the paper.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.