- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces CausaLab, a benchmark for interactive causal discovery. It utilizes randomized structural causal models (SCMs) and step-by-step DSL (Domain-Specific Language) logs to separately evaluate target predictions and the recovery of the complete causal graph. Experiments across multiple models demonstrate that achieving correct task outcomes does not necessarily imply the recovery of the underlying causal mechanisms.
Critical strengths
The use of randomized SCMs provides a clear ground truth, enabling the independent verification of both predictions and mechanism recovery. Furthermore, the step-by-step DSL logs facilitate the analysis of hypothesis revisions and failure modes, offering utility for both evaluation and diagnosis.
Critical weaknesses
The "Golden" baseline in Section 5.3 suffers from issues regarding non-interventional trajectories and discontinuities in the initial transitions (see Appendix F.2). Notably, for the 36 six-node samples involving interventions, the F1 score actually increases from 0.685 to 0.725; consequently, the conclusions drawn in the main text should be stated with greater caution.
In Section 5.5, the authors highlight that consistency checks improve accuracy from 48% to 60%. However, Appendix H reports a p-value of approximately 0.31 for the paired test, and each condition was evaluated with only a single run. The current evidence is insufficient to support a claim of consistent improvement.
Questions for authors
Appendix D.4 notes that the "paired-shift" method leverages prior knowledge—such as the assumption that agents possess unknown positive coefficients and that the target variable acts as a sink node. How does the study, in Section 6, rule out the influence of these additional priors and numerical solvers to ensure that the observed performance gaps are primarily attributable to the experimental design itself?
Appendix B.2 mentions the presence of unsolvable samples that exceed operational limits, yet the evaluation protocol allows for three attempts. If the results were reported based on the first-attempt accuracy for the subset of solvable problems, would the conclusions regarding graph scale and model rankings still hold?
7 comments
Response to Reviewer Q215 (2/3)
W2: the consistency-check gain
We agree that the original evidence was too weak: one run per arm on 50 graphs gave exact McNemar p ≈ 0.31. To raise confidence, we ran 50 more paired episodes on new four-node graphs, with both arms in one batch and identical environments within each pair.
Pooled over all 100 pairs, task success is 50/100 with the standard prompt and 73/100 when the prompt tells the agent to re-check its current equation against every data point it has recorded. The check changes neither the number of interventions (5.4 vs 5.5) nor graph-recovery F1 (−0.019 [−0.059, +0.020]), so the gain comes from the check itself; this larger experiment raises our confidence that the consistency check improves task success.
| Standard | With check | Difference, points [95% CI] | Exact McNemar p | |
|---|---|---|---|---|
| Appendix H (50 pairs) | 24/50 | 30/50 | +12 [−6, +30] | 0.31 |
| 50 new graphs | 26/50 | 43/50 | +34 [+20, +50] | < 0.001 |
| Pooled (100 pairs) | 50/100 | 73/100 | +23 [+11, +35] | < 0.001 |
Q1: the priors of the paired-shift reference and the exact solver
Neither prior affects our comparison between the deterministic reference program and the LLM agents. We take the two priors in turn.
Positive coefficients. The reference program uses positivity only to order the variables. Inverting the total-effect matrix, (with including its unit diagonal), recovers every direct coefficient without any sign assumption. We reran the reference program on the 250 main-suite graphs and on 1,250 copies with randomly flipped coefficient signs (same setting as App. D.4: one shift per variable):
| Solver | Original graphs (250) | Sign-flipped copies (1,250) |
|---|---|---|
| Matrix inversion (no sign assumption) | 100% exact | 100% exact |
So with or without the positivity prior, paired shift with matrix inversion recovers every graph exactly, edges and coefficients alike.
Frequency as a sink. The agents are not told that frequency has no downstream variables, and the hypothesis format allows edges out of it. Yet none of the final hypotheses in the 2,850 GPT episodes contains such an edge (0 of 11,460 edges). The agents behave as if they already had this prior, so it gives the reference program no advantage.
The gap between the agents and the deterministic reference is not only about experiment design: using the collected data, and computing and reasoning with it exactly, also matter. To trace the gap to its source, we measure separately how well the models design experiments and how well they use the data they collect. For each agent, we apply an exact solver to its own records under the main-suite assumptions; a run's data identify the system when this solver recovers the graph and all coefficients from them.
At six nodes, GPT-5.2-high and DeepSeek-V4-Pro still return an incorrect graph in 79% and 50%, respectively, of the logged runs whose data identify the system (see the tables in our response to Reviewer L4ja). We will revise the abstract and §6 to distinguish evidence collection from inference.
Effect of graph size (both GPT models, 3–7 nodes):
| Model | Nodes | Data identify the structure and every edge coefficient | Given that, final graph still wrong | Given that, final frequency still wrong |
|---|---|---|---|---|
| GPT-5-mini | 3 | 83% | 10% | 10% |
| GPT-5-mini | 4 | 81% | 66% | 53% |
| GPT-5-mini | 5 | 59% | 93% | 74% |
| GPT-5-mini | 6 | 47% | 96% | 70% |
| GPT-5-mini | 7 | 30% | 100% | 86% |
| GPT-5.2-high | 3 | 98% | 0% | 0% |
| GPT-5.2-high | 4 | 100% | 24% | 7% |
| GPT-5.2-high | 5 | 89% | 52% | 5% |
| GPT-5.2-high | 6 | 62% | 79% | 17% |
| GPT-5.2-high | 7 | 37% | 88% | 24% |
Six models (main suite, 6 nodes):
| Model | Data identify the structure and every edge coefficient | Given that, final graph still wrong | Given that, final frequency still wrong |
|---|---|---|---|
| GPT-5-mini | 47% | 96% | 70% |
| Qwen3.5-Non-thinking | 46% | 100% | 78% |
| Qwen3.5-Thinking | 54% | 96% | 56% |
| DeepSeek-V4-Flash | 18% | 67% | 22% |
| DeepSeek-V4-Pro | 48% | 50% | 12% |
| GPT-5.2-high | 62% | 79% | 17% |
Response to Reviewer Q215 (3/3)
Q2: unsolvable episodes and three attempts
These conclusions do not depend on unsolvable tasks or retries. Unsolvable tasks (true frequency above the dial range) are rare: none at 3–4 nodes and at most 7 of 50 per cell at 5–7 nodes. We recomputed success from the raw logs under all four definitions:
| Success, % (all / solvable / first / solvable first) | 3 nodes | 4 nodes | 5 nodes | 6 nodes | 7 nodes |
|---|---|---|---|---|---|
| GPT-5.2-high | 100 / 100 / 100 / 100 | 94 / 94 / 94 / 94 | 92 / 92 / 92 / 92 | 80 / 85 / 78 / 83 | 64 / 70 / 64 / 70 |
| Qwen3.5-Thinking | 94 / 94 / 94 / 94 | 52 / 52 / 52 / 52 | 36 / 36 / 36 / 36 | 34 / 38 / 28 / 31 | 36 / 39 / 32 / 35 |
| GPT-5-mini | 90 / 90 / 90 / 90 | 48 / 48 / 46 / 46 | 18 / 19 / 18 / 19 | 24 / 26 / 24 / 26 | 16 / 16 / 16 / 16 |
| DeepSeek-V4-Pro | – | – | – | 84 / 91 / 76 / 83 | – |
| DeepSeek-V4-Flash | – | – | – | 64 / 74 / 60 / 70 | – |
All is the paper's metric; solvable counts only solvable tasks; first counts only the first dial setting. The solvable denominator excludes tasks whose true frequency lies above the dial range, so this rate can exceed the paper's metric.
The conclusions are therefore essentially the same under all four definitions:
- Graph size: under every definition, GPT-5.2-high's success falls monotonically with size (100% at 3 nodes, 64–70% at 7); the other models drop sharply between 3 and 5 nodes and stay low.
- Model ranking: under every definition, GPT-5.2-high leads GPT-5-mini at every size (intervals exclude 0), Qwen3.5-Thinking leads GPT-5-mini at every size, and DeepSeek-V4-Pro leads -Flash. At 6 nodes, GPT-5.2-high and DeepSeek-V4-Pro stay within 6.2 points of each other under every definition (intervals include 0), i.e. in the same tier.
- Small differences between definitions: first-attempt success is at most 4 episodes below the paper's metric in any cell, and the solvable-only metric only removes the few unsolvable tasks from the denominator; both are far smaller than the differences between models and between sizes.
Response to Reviewer Q215 (1/3)
Thank you for recognizing the value of CausaLab's ground truth and hypothesis logs.
TL;DR. Your questions led us to revise the controls and add the following evidence:
None of this changes the central result: task success and recovery of the causal structure come apart. Throughout, task success means reactor activation; graph-recovery F1 compares the final graph with ground truth.
W1: the Golden control and the six-node traces without interventions
Verified Golden traces improve both outcomes, but successful prediction still does not imply full discovery. At six nodes, task success is 60% and exact graph recovery is 8%. We therefore revise the Golden conclusion and report both settings.
The original search distinguished graphs only within the 50-graph suite; two passive records met that criterion for 14 graphs, leaving empty intervention chains. We agree that singling a graph out within the suite is weaker than identifying it from the data, which is what this comparison tests, so we add a second setting with a stronger criterion: the trace must identify the graph among all DAGs. In the new setting the 14 graphs get one shift per variable, and the other traces keep their interventions.
Each new trace intervenes on every controllable variable. Sign-free matrix inversion recovers the graph from all 100 traces. GPT-5-mini receives the trace with no further interventions. The comparison uses the existing runs on the same graphs and prompt.
Exact graph recovery: the agent's final hypothesis has exactly the true set of directed edges, including the edges into the frequency, with no missing, extra or reversed edge.
We will correct the Golden conclusion in §1, §5.3 and §6 and report both settings in Fig. 13b, Table 16 and App. F.2.