acceptodds
Under review as a conference paper at ICLR 2027

CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientists

Abstract

We introduce CausaLab, a scalable environment for evaluating interactive causal discovery by LLM agents. Unlike prior evaluations, CausaLab evaluates both whether an agent can solve a problem using causal evidence and whether its answer is grounded in a faithful recovered causal mechanism. Each episode hides a randomly sampled structural causal model (SCM) that no prior knowledge supplies: the agent reads prior records, intervenes on a manipulator crystal, and predicts the held-out frequency of a reactor crystal built from the same SCM. Because the reactor exposes every property except frequency, the prediction needs only that variable’s parents and coefficients, so we score this Target Mechanism Recovery separately from Full System Discovery. Experiments show that correct prediction does not imply Full System Discovery. Across matched controls on functional form, hidden perturbations, and graph structure, the two levels move independently or in opposite directions. From ten observations alone on 6-node graphs, GPT-5.2-high recovers the parents of frequency exactly and answers 86% of reactors correctly, yet reaches 0.471 all-edge F1. The same ten rounds spent on interventions lift all- edge F1 from 0.47 to 0.82 for ten accuracy points. Failures come from premature commitment, not exhausted budget, and a consistency check recovers part of the loss. A shift control on a quarter of the agents’ budget recovers every main- suite graph exactly, so the gap measures experiment design. CausaLab therefore separates the two levels and exposes current LLM agents’ limits as experimental causal reasoners. We release the full SCM dataset and benchmark code to advance future research.

Primary area:
datasets and benchmarks
Keywords:
agent evaluation, causal discovery, interactive environments, experiment design with LLMs, benchmark

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_Q215OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces CausaLab, a benchmark for interactive causal discovery. It utilizes randomized structural causal models (SCMs) and step-by-step DSL (Domain-Specific Language) logs to separately evaluate target predictions and the recovery of the complete causal graph. Experiments across multiple models demonstrate that achieving correct task outcomes does not necessarily imply the recovery of the underlying causal mechanisms.

Critical strengths

The use of randomized SCMs provides a clear ground truth, enabling the independent verification of both predictions and mechanism recovery. Furthermore, the step-by-step DSL logs facilitate the analysis of hypothesis revisions and failure modes, offering utility for both evaluation and diagnosis.

Critical weaknesses

The "Golden" baseline in Section 5.3 suffers from issues regarding non-interventional trajectories and discontinuities in the initial transitions (see Appendix F.2). Notably, for the 36 six-node samples involving interventions, the F1 score actually increases from 0.685 to 0.725; consequently, the conclusions drawn in the main text should be stated with greater caution.

In Section 5.5, the authors highlight that consistency checks improve accuracy from 48% to 60%. However, Appendix H reports a p-value of approximately 0.31 for the paired test, and each condition was evaluated with only a single run. The current evidence is insufficient to support a claim of consistent improvement.

Questions for authors

Appendix D.4 notes that the "paired-shift" method leverages prior knowledge—such as the assumption that agents possess unknown positive coefficients and that the target variable acts as a sink node. How does the study, in Section 6, rule out the influence of these additional priors and numerical solvers to ensure that the observed performance gaps are primarily attributable to the experimental design itself?

Appendix B.2 mentions the presence of unsolvable samples that exceed operational limits, yet the evaluation protocol allows for three attempts. If the results were reported based on the first-attempt accuracy for the subset of solvable problems, would the conclusions regarding graph scale and model rankings still hold?

7 comments
AuthorsOpenReview ↗

Response to Reviewer Q215 (1/3)

Thank you for recognizing the value of CausaLab's ground truth and hypothesis logs.

TL;DR. Your questions led us to revise the controls and add the following evidence:

  • Golden control (W1): In a second setting whose traces must identify the graph among all DAGs, both task success and graph recovery rise; we revise the conclusion accordingly.
  • Consistency check (W2): 50 new paired graphs (100 pairs in total); the gain in task success from consistency check is significant (+23 points, p < 0.001).
  • Priors and attribution (Q1): We removed the positivity requirement and recovered all 250 original and 1,250 sign-flipped graphs exactly. Agents never draw an edge out of frequency, so the sink assumption gives the reference no advantage.
  • Evaluation protocol (Q2): Under first-attempt success on solvable tasks, the conclusions on graph size and model ranking hold.

None of this changes the central result: task success and recovery of the causal structure come apart. Throughout, task success means reactor activation; graph-recovery F1 compares the final graph with ground truth.

W1: the Golden control and the six-node traces without interventions

Verified Golden traces improve both outcomes, but successful prediction still does not imply full discovery. At six nodes, task success is 60% and exact graph recovery is 8%. We therefore revise the Golden conclusion and report both settings.

The original search distinguished graphs only within the 50-graph suite; two passive records met that criterion for 14 graphs, leaving empty intervention chains. We agree that singling a graph out within the suite is weaker than identifying it from the data, which is what this comparison tests, so we add a second setting with a stronger criterion: the trace must identify the graph among all DAGs. In the new setting the 14 graphs get one shift per variable, and the other traces keep their interventions.

Each new trace intervenes on every controllable variable. Sign-free matrix inversion recovers the graph from all 100 traces. GPT-5-mini receives the trace with no further interventions. The comparison uses the existing runs on the same graphs and prompt.

NodesSettingTask successGraph-recovery F1Exact graph recovery
4Golden trace (new setting)90% [79, 96]0.93050% [37, 63]
4Own experiments64% [50, 76]0.72110% [4, 21]
6Golden trace (new setting)60% [46, 72]0.7538% [3, 19]
6Own experiments26% [16, 40]0.4340% [0, 7]

Exact graph recovery: the agent's final hypothesis has exactly the true set of directed edges, including the edges into the frequency, with no missing, extra or reversed edge.

We will correct the Golden conclusion in §1, §5.3 and §6 and report both settings in Fig. 13b, Table 16 and App. F.2.

AuthorsOpenReview ↗

Response to Reviewer Q215 (2/3)

W2: the consistency-check gain

We agree that the original evidence was too weak: one run per arm on 50 graphs gave exact McNemar p ≈ 0.31. To raise confidence, we ran 50 more paired episodes on new four-node graphs, with both arms in one batch and identical environments within each pair.

Pooled over all 100 pairs, task success is 50/100 with the standard prompt and 73/100 when the prompt tells the agent to re-check its current equation against every data point it has recorded. The check changes neither the number of interventions (5.4 vs 5.5) nor graph-recovery F1 (−0.019 [−0.059, +0.020]), so the gain comes from the check itself; this larger experiment raises our confidence that the consistency check improves task success.

StandardWith checkDifference, points [95% CI]Exact McNemar p
Appendix H (50 pairs)24/5030/50+12 [−6, +30]0.31
50 new graphs26/5043/50+34 [+20, +50]< 0.001
Pooled (100 pairs)50/10073/100+23 [+11, +35]< 0.001

Q1: the priors of the paired-shift reference and the exact solver

Neither prior affects our comparison between the deterministic reference program and the LLM agents. We take the two priors in turn.

Positive coefficients. The reference program uses positivity only to order the variables. Inverting the total-effect matrix, B=I−T−1B = I - T^{-1} (with TT including its unit diagonal), recovers every direct coefficient without any sign assumption. We reran the reference program on the 250 main-suite graphs and on 1,250 copies with randomly flipped coefficient signs (same setting as App. D.4: one shift per variable):

SolverOriginal graphs (250)Sign-flipped copies (1,250)
Matrix inversion (no sign assumption)100% exact100% exact

So with or without the positivity prior, paired shift with matrix inversion recovers every graph exactly, edges and coefficients alike.

Frequency as a sink. The agents are not told that frequency has no downstream variables, and the hypothesis format allows edges out of it. Yet none of the final hypotheses in the 2,850 GPT episodes contains such an edge (0 of 11,460 edges). The agents behave as if they already had this prior, so it gives the reference program no advantage.

The gap between the agents and the deterministic reference is not only about experiment design: using the collected data, and computing and reasoning with it exactly, also matter. To trace the gap to its source, we measure separately how well the models design experiments and how well they use the data they collect. For each agent, we apply an exact solver to its own records under the main-suite assumptions; a run's data identify the system when this solver recovers the graph and all coefficients from them.

At six nodes, GPT-5.2-high and DeepSeek-V4-Pro still return an incorrect graph in 79% and 50%, respectively, of the logged runs whose data identify the system (see the tables in our response to Reviewer L4ja). We will revise the abstract and §6 to distinguish evidence collection from inference.

Effect of graph size (both GPT models, 3–7 nodes):

ModelNodesData identify the structure and every edge coefficientGiven that, final graph still wrongGiven that, final frequency still wrong
GPT-5-mini383%10%10%
GPT-5-mini481%66%53%
GPT-5-mini559%93%74%
GPT-5-mini647%96%70%
GPT-5-mini730%100%86%
GPT-5.2-high398%0%0%
GPT-5.2-high4100%24%7%
GPT-5.2-high589%52%5%
GPT-5.2-high662%79%17%
GPT-5.2-high737%88%24%

Six models (main suite, 6 nodes):

ModelData identify the structure and every edge coefficientGiven that, final graph still wrongGiven that, final frequency still wrong
GPT-5-mini47%96%70%
Qwen3.5-Non-thinking46%100%78%
Qwen3.5-Thinking54%96%56%
DeepSeek-V4-Flash18%67%22%
DeepSeek-V4-Pro48%50%12%
GPT-5.2-high62%79%17%
AuthorsOpenReview ↗

Response to Reviewer Q215 (3/3)

Q2: unsolvable episodes and three attempts

These conclusions do not depend on unsolvable tasks or retries. Unsolvable tasks (true frequency above the dial range) are rare: none at 3–4 nodes and at most 7 of 50 per cell at 5–7 nodes. We recomputed success from the raw logs under all four definitions:

Success, % (all / solvable / first / solvable first)3 nodes4 nodes5 nodes6 nodes7 nodes
GPT-5.2-high100 / 100 / 100 / 10094 / 94 / 94 / 9492 / 92 / 92 / 9280 / 85 / 78 / 8364 / 70 / 64 / 70
Qwen3.5-Thinking94 / 94 / 94 / 9452 / 52 / 52 / 5236 / 36 / 36 / 3634 / 38 / 28 / 3136 / 39 / 32 / 35
GPT-5-mini90 / 90 / 90 / 9048 / 48 / 46 / 4618 / 19 / 18 / 1924 / 26 / 24 / 2616 / 16 / 16 / 16
DeepSeek-V4-Pro–––84 / 91 / 76 / 83–
DeepSeek-V4-Flash–––64 / 74 / 60 / 70–

All is the paper's metric; solvable counts only solvable tasks; first counts only the first dial setting. The solvable denominator excludes tasks whose true frequency lies above the dial range, so this rate can exceed the paper's metric.

The conclusions are therefore essentially the same under all four definitions:

  • Graph size: under every definition, GPT-5.2-high's success falls monotonically with size (100% at 3 nodes, 64–70% at 7); the other models drop sharply between 3 and 5 nodes and stay low.
  • Model ranking: under every definition, GPT-5.2-high leads GPT-5-mini at every size (intervals exclude 0), Qwen3.5-Thinking leads GPT-5-mini at every size, and DeepSeek-V4-Pro leads -Flash. At 6 nodes, GPT-5.2-high and DeepSeek-V4-Pro stay within 6.2 points of each other under every definition (intervals include 0), i.e. in the same tier.
  • Small differences between definitions: first-attempt success is at most 4 episodes below the paper's metric in any cell, and the solvable-only metric only removes the few unsolvable tasks from the denominator; both are far smaller than the differences between models and between sizes.
Reviewer_C3QcOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a new way to evaluate LLMs for interactive causal discovery tasks. Those tasks are framed/verbalized as investigating a crystal in a laboratory.

More specifically, LLMs are asked to predict a specific variable (frequency) of the crystal by observing or intervening upon the SCM. The LLMs are evaluated in terms of recovering the mechanism that determines frequency and in terms of recovering the full causal graph.

This new interactive evaluation setup, together with the actual evaluation results can be seen as the main contribution of the paper.

Critical strengths

The core idea of evaluating LLMs in the proposed interactive setup is very meaningful and promising to me, as the question whether LLMs can, and to what degree, reason causally remains somewhat open. In that context, the crystal- and laboratory-based framings make a lot of sense while I appreciate the relatively fine-grained analysis of the final results.

Critical weaknesses

I see the simplicity of the causal mechanisms that are used to generate the data as a major weakness: linear mechanisms and quadratic mechanisms might be substantially too simple compared to real-world scenarios and leave it unclear if the findings of the benchmark generalize to more complex cases which one might expect to occur in the real world. The relatively narrow causal scope is further exacerbated by only considering shift interventions.

Questions for authors

Why do the authors restrict their SCM-mechanisms to very simply linear and quadratic functions?

How do the evaluated LLMs perform when the causal challenges become more difficult (larger graphs, different intervention types, more complex mechanisms)?

4 comments
AuthorsOpenReview ↗

Response to Reviewer C3Qc (1/2)

Thank you for recognizing the value of the interactive setup and for suggesting the extensions below.

TL;DR. We expanded the experiments along the three directions you suggested:

  • Mechanisms (W1, Q1): We added saturating and multiplicative-interaction mechanisms with matched linear controls. Under saturation, success falls from 52% to 18%, while graph-recovery F1 stays near 0.70. We also clarify why exact verification and coefficient scoring motivated the original families.
  • Intervention types (W2): We implemented hard (do) interventions and evaluated them at four and six nodes. The new trajectory analysis links 26–36% of missed edges to nodes pinned before their parents were varied.
  • Larger graphs (Q2): We evaluated GPT-5.2-high and Qwen3.5-Thinking on 50 ten-node graphs. GPT-5.2-high achieves 64% success with graph-recovery F1 0.604; the reference recovers all 50 graphs exactly.

Here, task success means reactor activation; graph-recovery F1 compares the final graph with ground truth.

W1 and Q1: Why linear and quadratic mechanisms, and do the findings generalize?

CausaLab now separates structural from functional recovery beyond linear and quadratic mechanisms. In this revision, we will add experiments on two additional mechanism families to further investigate this question. Under saturation, success is 18% versus 52% for the matched linear control, and frequency-coefficient F1 is 0.180 versus 0.573; graph-recovery F1 stays near 0.70.

The benchmark already goes beyond linear mechanisms: we introduced quadratic mechanisms to test relationships beyond linear ones (Table 9). Linear and quadratic families form the core because they make every task provably solvable and let us score every coefficient exactly (App. D.3–D.4); the family is a parameter of the generator (§6). The added families are bounded saturating mechanisms s·tanh(p/s) and non-additive multiplicative interactions v·pᵢ·pⱼ. GPT-5-mini ran each family on the 50 four-node graphs, against a linear run on the same graphs in the same batch:

MechanismTask successGraph-recovery F1F1 of frequency-equation coefficients
Linear52%0.7010.573
Saturating18%0.7210.180
Multiplicative interaction10%0.7110.161

Across both families, graph-recovery F1 moves by at most 0.022 relative to the linear run on the same graphs (paired 95% intervals include zero), while task success and frequency-coefficient F1 fall sharply (paired 95% intervals exclude zero). GPT-5-mini still finds the edges but gets the functional form on them wrong: in the interaction suite, only 2 of the 50 final hypotheses include an interaction coefficient. Table 9 shows the same split when moving from linear to quadratic mechanisms, and it now holds under bounded and non-additive mechanisms as well. Recovering the causal graph does not imply recovering the mechanism, so graph recovery and task success can diverge.

W2: Only shift interventions.

We added hard (do) interventions so CausaLab can test how intervention order affects the available evidence. Edges into nodes pinned before their parents were varied account for 26–36% of missed edges. A pin cuts incoming effects and holds the variable until the agent sets it again. Shift is the environment's default, not a limit of the framework: intervention semantics live in one function. An offline reference that pins one variable at a time in causal order recovers all 50 graphs at 4 and 6 nodes.

GPT-5-mini ran on 50 graphs at each of four and six nodes. The shift comparison uses the existing shift runs on the same graphs:

NodesTask success (hard / shift)Graph-recovery F1 (hard / shift)Change in F1 [95% CI]Edge recall (hard / shift)
458% / 48%0.807 / 0.865−0.058 [−0.108, −0.008]0.741 / 0.846
630% / 24%0.585 / 0.677−0.093 [−0.151, −0.035]0.497 / 0.620

Task success is slightly higher in hard mode, with an uncertain difference, while graph recovery is lower, entirely through recall. The edges hidden by early pinning were almost never recovered (missed 22/22 at 4 nodes and 58/61 at 6, vs. 18% and 45% of testable edges). We will describe hard mode in §3.2 and report the missed-edge analysis and the offline sweep in the appendix.

AuthorsOpenReview ↗

Response to Reviewer C3Qc (2/2)

Q2: How do the models perform when the tasks become harder?

On 10-node graphs, GPT-5.2-high solves 64% of tasks but recovers 57% of true edges (F1 0.604). CausaLab thus measures incomplete system recovery even when prediction succeeds at the larger scale. The reference recovers all 50 graphs exactly.

We sampled the suite from the same generator and widened the dial range to keep the tasks operationally solvable. The paper also varies size from 3 to 7 nodes for four models (Table 17) and tests quadratic mechanisms, hidden disturbances, and target-outgoing-edge settings (Tables 9–11).

Model7 nodes (paper): success / F110 nodes (new): success / F1
GPT-5.2-high64% / 0.74564% / 0.604
Qwen3.5-Thinking36% / 0.45432% / 0.258

On the same 10-node graphs, GPT-5.2-high exceeds Qwen3.5-Thinking on both metrics. We will add these results to Table 17.

Thank you for these suggestions. We will include the new settings and analyses in the revision and state the models and conditions tested.

Thanks for addressing my concerns and running additional experiments.

I believe that the evaluation is more sound and thorough now through the inclusion of further mechanisms and a different intervention type, as well as more models.

I assess the paper as an interesting contribution and raised my score to 4 accordingly.

However, I kindly ask the AC to take my confidence of 2 into account since I have only broad familiarity with LLM evaluations.

Thank you for reading our response and for raising your score.

We are glad that the added mechanism families, the hard-intervention mode and the additional models address your concerns.

Reviewer_L4jaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper evaluates interactive causal discovery in LLM agents where each discovery episode procedurally samples a hidden 3–7 node structural causal model (SCM). Agents observe passive crystal records, execute budgeted interventions on a manipulator crystal, log stepwise hypotheses, and predict a held-out reactor crystal’s resonance frequency. The results suggest online interventions drive graph recovery far better than passive observations, model scale lifts performance without saturating 6–7 node graphs, and agents fail largely via premature commitment to self-contradictory hypotheses rather than budget exhaustion.

Critical strengths
  • procedurally generating scms helps to eliminate benchmark memorization
  • scoring both the agent's final prediction and its recovered causal graph exposes a blind spot in benchmarks that only grade final answers: an agent can predict the target variable accurately while still getting most of the underlying causal system wrong
Critical weaknesses
  • benchmarked graphs are fairly small (3-7 nodes), linear, and noiseless
  • most analysis is done using gpt-5-mini, making transfer of conclusions to other models unclear
  • error bars are missing, with only one evaluation done per graph
Questions for authors

The paper concludes that experiment design is the main bottleneck for performance, but Section 5.5 notes that most 4- and 5-node runs do intervene on every controllable variable while still committing to hypotheses that contradict their own collected data. This suggests that much of the gap may come from failing to infer the right graph and equations from collected evidence rather than solely from choosing poor experiments. Do larger/more capable models also suffer from this failure mode?

5 comments
AuthorsOpenReview ↗

Response to Reviewer L4ja (1/3)

Thank you for recognizing the value of evaluating prediction and causal recovery separately.

TL;DR. In response to your questions, we added:

  • Evidence use across models (Q, W2): A new analysis of all six configurations distinguishes collecting evidence from using it. Among six-node runs whose logged data identify the graph and coefficients, GPT-5.2-high and DeepSeek-V4-Pro still return incorrect graphs in 79% and 50% of cases.
  • Uncertainty estimates (W3): At the same budget, paired 95% intervals show that spending it on interventions rather than observations improves graph recovery in 7 of 8 comparisons. These quantify uncertainty across sampled episodes.
  • Broader evaluation (W1): New 10-node experiments and two further nonlinear families (saturating and multiplicative interaction) extend the original settings. We also clarify the existing hidden-disturbance tests.

Throughout, task success means the agent activates the reactor, and graph-recovery F1 is the edge-level F1 between the agent's final causal graph and the true graph.

W1: "benchmarked graphs are fairly small (3-7 nodes), linear, and noiseless"

We have conducted additional experiments to address each of them. Before presenting the new results, we would first like to clarify what was already included in the previous version with respect to these three aspects.

On linear: we have already included quadratic mechanisms, which we introduced to probe relationships beyond linear ones (Table 9).

On noiseless: we have introduced six noisy settings in which an unobserved variable shifts one to three observed variables by ±0.5 to ±50 after every intervention (Table 10).

On small: We have constructed graphs with 3 to 7 nodes. Performance is far from saturated at 7 nodes (Table 17).

As for our revision, CausaLab now tests the prediction/recovery distinction on 10-node graphs and two additional nonlinear families. On 10 nodes, GPT-5.2-high solves 64% of tasks with graph-recovery F1 0.604; under saturation, GPT-5-mini's task success falls from 52% to 18% while F1 stays near 0.70. These results answer the size and mechanism questions directly. The new results are:

DirectionSettingModelTask successGraph-recovery F1
Size7 nodes (paper)GPT-5.2-high64%0.745
10 nodes (new)GPT-5.2-high64%0.604
7 nodes (paper)Qwen3.5-Thinking36%0.454
10 nodes (new)Qwen3.5-Thinking32%0.258
Nonlinearity (new, 4 nodes)LinearGPT-5-mini52%0.701
Saturating, s·tanh(p/s)GPT-5-mini18%0.721
Multiplicative interaction*GPT-5-mini10%0.711

We will add these results to the appendix and update the scope statement in §6.

W2: "most analysis is done using gpt-5-mini, making transfer of conclusions to other models unclear"

The evidence-use diagnosis extends beyond GPT-5-mini: we now analyze all six configurations from three model families. The logged trajectories show graph-recovery errors even when the collected data identify the graph. The cross-model results are in our answer to Q below.

The main benchmark evaluates GPT-5-mini, GPT-5.2-high and Qwen3.5 with and without thinking at 3–7 nodes, plus DeepSeek-V4-Flash and -Pro at six nodes (Fig. 4, Tables 17–18). The matched-budget study (Fig. 3) and classical-baseline comparison (Table 1) also use GPT-5.2-high. Repeating every experiment on several models is beyond our current budget, and we leave it to future work: the main goal of this work is a scalable framework for constructing causal environments on which the community can run its own causal-discovery studies, and our analyses are examples of what it supports.

AuthorsOpenReview ↗

Response to Reviewer L4ja (2/3)

W3: "error bars are missing, with only one evaluation done per graph"

We will add error bars (95% confidence intervals) to every result in the final version. Because of the rebuttal’s space limit, we show them here for the data behind the RQ2 analysis (observation vs. intervention at a matched budget, Fig. 3) and show that its main conclusion still holds with them.

The paired analysis supports CausaLab's distinction between prediction and structural recovery: interventions improve graph-recovery F1 in 7 of 8 comparisons.

At the same budget, the table compares interventions only, or observations plus interventions, with observations only, on the same 50 graphs (RQ2, Fig. 3). Intervals resample graph episodes.

Model, nodesBudgetChange in graph-recovery F1Change in task success (points)
GPT-5.2-high, 6interventions only+0.35 [0.29, 0.40]−10 [−22, +2]
GPT-5.2-high, 6observations + interventions+0.33 [0.28, 0.37]−12 [−22, −2]
GPT-5.2-high, 4interventions only+0.32 [0.28, 0.35]0
GPT-5.2-high, 4observations + interventions+0.31 [0.27, 0.35]−4 [−10, 0]
GPT-5-mini, 6interventions only+0.13 [0.07, 0.19]−8 [−18, 0]
GPT-5-mini, 6observations + interventions+0.18 [0.12, 0.24]+6 [−6, +18]
GPT-5-mini, 4interventions only+0.06 [−0.04, 0.14]−26 [−44, −8]
GPT-5-mini, 4observations + interventions+0.10 [0.04, 0.16]−10 [−28, +8]

Each result uses 50 independently sampled graphs, and the paper already reports intervals for the DeepSeek comparison (§5.4). Because each graph is run once, intervals that resample graphs already include the agent’s run-to-run variation.

Running each graph once is also a deliberate use of the budget. An agentic episode is a long multi-turn interaction, so every episode is costly, and for a fixed number of episodes, one run on each of more graphs estimates average performance more precisely than several runs on fewer graphs. Interactive agent benchmarks of similar size built on synthetic tasks likewise spread their budget over distinct task instances, e.g. DiscoveryWorld (120 instances: 24 scenarios × 5 parametric seeds, averaged over the seeds) and ALFWorld (134 unseen synthetic text games). Our main suite has 50 graphs at each size from 3 to 7 nodes, 250 tasks per model, so each of our settings averages over 50 instances where a DiscoveryWorld scenario averages over 5; the variant suites with quadratic mechanisms, unobserved disturbances and an extra frequency edge add more (Tables 9–11).

In the revision, every result will carry a 95% interval: all columns of every results table and every plotted point, in the main text and the appendix (task success, all F1 variants, precision/recall, SHD, and the classical baselines), with a paired interval for every comparison that a figure or the text draws.

References

[1] P. Jansen, M.-A. Côté, T. Khot, E. Bransom, B. Dalvi Mishra, B. P. Majumder, O. Tafjord, and P. Clark. DiscoveryWorld: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents. In NeurIPS Datasets and Benchmarks Track, 2024.

[2] M. Shridhar, X. Yuan, M.-A. Côté, Y. Bisk, A. Trischler, and M. Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In ICLR, 2021.

AuthorsOpenReview ↗

Response to Reviewer L4ja (3/3)

Q: "This suggests that much of the gap may come from failing to infer the right graph and equations from collected evidence rather than solely from choosing poor experiments. Do larger/more capable models also suffer from this failure mode?"

Yes. CausaLab's logs identify graph-recovery failures even when stronger models have collected sufficient evidence. At six nodes, GPT-5.2-high and DeepSeek-V4-Pro return incorrect graphs in 79% and 50% of logged runs whose data identify the graph and every coefficient. Their task-failure rates in the same subset are 17% and 12%.

Your question led us to measure evidence use directly. We have measured this and below are our method and results.

Method. On the main-suite logs, an exact solver tests whether the agent's records uniquely determine the graph and every coefficient, including the frequency equation, under the suite's assumptions. Within that evidence-sufficient subset, we measure wrong final graphs and task failures.

Effect of graph size (both GPT models, 3–7 nodes):

ModelNodesRuns whose data identify the structure and every coefficientGiven that, final graph still wrongGiven that, frequency still wrong
GPT-5-mini383%10%10%
GPT-5-mini481%66%53%
GPT-5-mini559%93%74%
GPT-5-mini647%96%70%
GPT-5-mini730%100%86%
GPT-5.2-high398%0%0%
GPT-5.2-high4100%24%7%
GPT-5.2-high589%52%5%
GPT-5.2-high662%79%17%
GPT-5.2-high737%88%24%

All six models (main suite, 6 nodes):

ModelRuns whose data identify the structure and every coefficientGiven that, final graph still wrongGiven that, frequency still wrongFailed runs whose data identify the frequency equation
GPT-5-mini47%96%70%84%
Qwen3.5-Non-thinking46%100%78%75%
Qwen3.5-Thinking54%96%56%82%
DeepSeek-V4-Flash18%67%22%83%
DeepSeek-V4-Pro48%50%12%100%
GPT-5.2-high62%79%17%100%
  • Most failures are failures to use the evidence, for every model: at 6 nodes, 75–100% of failed runs already held data that pin down the frequency equation.

  • Stronger models suffer less but are not immune: at 6 nodes, among runs whose data suffice, GPT-5.2-high and DeepSeek-V4-Pro rarely get the frequency wrong (17% and 12%, against 56–78% for GPT-5-mini and both Qwen models), yet still commit to a wrong graph in 79% and 50% of them.

  • Collecting and using evidence both get harder as graphs grow: from 3 to 7 nodes, fewer runs collect data that identify the structure and every coefficient (GPT-5.2-high 98% → 37%, GPT-5-mini 83% → 30%), and among runs whose data suffice, more still commit to a wrong graph (GPT-5.2-high 0% → 88%, GPT-5-mini 10% → 100%) or get the frequency wrong (0% → 24%, 10% → 86%).

At 6 nodes, in all six models, at least 82% of successful runs end with a hypothesis consistent with every observed state, versus at most half of failed runs.

We will add these diagnostics and clarify in the abstract and §6 how CausaLab distinguishes evidence acquisition from its use.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.