- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces CAUSAL-DISCOVERY-BENCH, a suite of sandbox environments designed to evaluate LLM agents on interactive, sequential experimental design and causal discovery under resource constraints. Operating on linear Structural Equation Models (SEMs) with randomized variable names to eliminate domain-semantic priors, agents must allocate finite budgets between observational sampling and experimental -interventions to identify direct causal parents of an outcome variable and estimate their direct treatment effects. Empirically, the benchmark tracks agent behavior using 40 metrics across 7 families, establishing a four-level difficulty hierarchy: single-experiment design, sequential hypothesis coverage, resource-aware sample sizing, and resolving conflicting evidence.
The paper demonstrates a fundamental gap between causal inference and sequential experimentation: 93.9% of agent errors stem from experimental design failures, specifically failing to run properly controlled experiments. Furthermore, increasing action budgets merely increases candidate coverage without inducing controlled experimentation. In contrast, hard interface constraints (such as capping the per-call sample purchase size) successfully force controlled experimentation and yield far larger performance gains than prompt engineering alone.
Critical strengths
S1 Actionable Error-Attribution Methodology: By charging every wrong conclusion back to the agent's evidentiary state via four ordered levels and cross-validating with targeted ablations, the paper provides a much denser, more actionable diagnostic signal than existing benchmarks. This cleanly proves that failures stem from experimental design rather than downstream inference.
S2 Rigorous Measurement Validity: The authors thoroughly stress-test their benchmark using IRT-based ability fits and bootstrap-bounded sampler robustness checks. This rigor ensures that reported cross-model performance differences reflect genuine agent capabilities rather than benchmark artifacts.
Critical weaknesses
W1 Key diagnostic claims rely on a single-model ablation. Although the broad benchmark evaluates 13 models, the specific ablation experiments producing the headline finding (~94% of errors caused by experimental design) were conducted exclusively on Claude Sonnet 4.6. Because default controlled-experiment behavior varies widely across model families, generalizing this single-model error-distribution ratio to LLM agents as a class lacks broader empirical validation.
W2 Scope Restricted to Unconfounded Linear Models. While assuming fully observed variables and linear structural equations enables clean closed-form ground truth and exact budget derivations, it limits the benchmark's realism. Real-world scientific discovery frequently involves non-linear dynamics and unobserved confounders, which this idealized setup does not test.
Questions for authors
- Clarification Can the central ~94% design-vs-inference error-attribution claim be replicated on at least 2–3 additional models spanning the performance range already shown in the cross-model comparison?
- Clarification How does this benchmark's design relate mechanistically to [1,2] and other closely-dated 2026 agentic causal-discovery benchmarks — specifically, what differentiates it beyond the error-attribution ladder and remediation ablations?
- Suggestion for future work Is there a plan to extend the sandbox to full DAG/skeleton recovery rather than single-outcome parent identification, and would the design-vs-inference conclusion be expected to hold at that scale?
[1] Sagar Deb et al. "Active Causal Discovery Benchmark: Evaluating LLM Agents Under Budgeted Interventions, 2026."
[2] Zi-Rong Li et al. "CausalArena: Benchmarking Causal Discovery in the Foundation Model Era, 2026."
2 comments
Dear Reviewer 6kMe,
Thank you very much for your encouraging score and detailed review! We would be very happy to clarify any concerns that you may have.
Best, The Authors
Replication across models, literature comparison and future work
Thank you for the comments.
Replication across models. It replicates. The ablations that motivate the ladder were run on
claude-sonnet-4-6, which carries most of the attributed errors. Appendix G now reads the same ladder on every model carrying at least twenty attributed errors.claude-sonnet-5claude-sonnet-4-6qwen3.7-plusgpt-5.6-terraEach row is that model's own attributed errors split by the ladder, so these are the per-model version of the 94.4% in the body and not a ranking of the models. Across three vendors the share runs from 85.8% to 96.7%, with an unweighted median of 93.2% over the four.
Relation to the concurrent benchmarks. This is answered in our general comment on distinction from relevant concurrent works, which sets out what the ablations test that the four benchmarks do not.
Full DAG and skeleton recovery. That is the extension we would build next. It has two problems that the single-outcome task does not. The agent would have to split one budget across many targets, and it would have to work out which way each edge points, which is free now because every edge runs into the one outcome. Whether the design-versus-inference split survives there is exactly what we would want to measure. We would like to measure and experiment first before extrapolating from the single-outcome result.