acceptodds
Under review as a conference paper at ICLR 2027

CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimation

Abstract

Many benchmarks for automated causal inference evaluate a system's performance based on a single numerical output, such as an Average Treatment Effect (ATE). This approach conflates two distinct steps in causal analysis: identification—formulating a valid research design under stated assumptions—and estimation—implementing that design numerically on finite data. We introduce CausalReasoningBenchmark, a benchmark of 174 queries across 139 real-world datasets, curated from 73 peer-reviewed research papers and three widely used causal-inference textbooks. For each query, a system must produce (i) a structured identification specification that names the strategy, the treatment, outcome, and control variables, and all design-specific elements, and (ii) a point estimate with a standard error. By scoring these two components separately, our benchmark enables granular diagnosis: it distinguishes failures in causal reasoning from errors in numerical execution. Baseline results with a state-of-the-art LLM show that, while the model correctly identifies the high-level strategy in 81% of cases, full identification-specification correctness drops to only 38%, revealing that the bottleneck lies in the nuanced details of research design rather than in computation. CausalReasoningBenchmark is publicly available on Hugging Face and is designed to foster the development of more robust automated causal-inference systems.

Primary area:
datasets and benchmarks
Keywords:
Causal inference, Causal reasoning, Identification, Estimation, Experimental designs, Observational Studies

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mcKBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes CausalReasoningBenchmark, a causal benchmark to assess the causal reasoning of large language models. Unlike existing benchmarks, the proposed benchmark aims to distinguish causal reasoning errors from numerical execution errors by focusing on scoring structure identification, treatment outcome, and control variable identification. The queries are from different experimental designs, extracted from research papers and causal inference textbooks. Experimental results on a state-of-the-art language model show that while the model could correctly identify high-level strategy in 81% of cases, its performance drops to 38% when trying to fully identify specification correctness.

Critical strengths
  • The benchmark moves from the standard evaluation of average treatment effect (ATE) estimate to evaluate the full experimental design;
  • The queries come from different datasets published in peer-reviewed papers and causal inference textbooks, covering full causal experimental designs, such as instrumental variables, difference-in-differences, regression discontinuity design, and randomized controlled trials.
  • It is a complementary strategy to assess the capacity of language models to specify causal experimental designs by identifying the necessary instrumental design.
Critical weaknesses
  • The benchmark questions are limited to one domain and a single country's point of view. Although this does not diminish the benchmark's value, it severely limits its application.
  • The experiment is limited to one language model, and there is no comparison with any of the other benchmarks listed in Table 1, in the related work section, or even closely related work not mentioned in the paper, such as CausalVerify [1].
  • There is no conditional analysis to identify when the errors are due to implementation issues, for instance.
  • It is unclear whether the model produced its estimates via code execution or built-in statistical functions, as the benchmark includes executing scripts.
  • The paper comprises many repeated statements.

[1] CausalVerify: End-to-End Verification of Causal Analyses by Language Models

Questions for authors
  • How would the proposed benchmark help handle the issue of data contamination and memorization when trying to assess the causal capability of language models? How was such an issue considered in the experiment?
  • What would be the results of the benchmark in a multi-agent configuration? How different would the results be if one used CLadder instead of the proposed benchmark?
2 comments

We thank the reviewer for the comments.

Scope. The benchmark covers central causal designs using reproducible empirical studies. Political science provides the initial corpus, with examples spanning Japan, Brazil, China, and other settings. Our contribution is a framework for evaluating complete identification specifications and numerical estimation.

Execution and baselines. As described in Section 7, the model only gets access to the data, a metadata file describing the data and a causal question, it does not have access to the estimation code- the estimation code is provided as reference and for transparency around how we constructed the numerical solution. We use this baseline to demonstrate the benchmark's diagnostic value: an agent can produce a reasonable effect estimate while failing to specify the underlying identification design correctly.

Related work. CLadder focuses on formal causal queries over graphs, while CausalVerify includes executable synthetic scenarios and paper-linked synthetic twins. Our research-paper tasks build on empirical study datasets, pairing them with detailed identification specifications and executable reference code. We will clarify how this focus complements the settings covered by these benchmarks.

Contamination. Our evaluation requires more than recalling a published estimate, since the agent must also specify the design, variables, controls, and estimand. However, this does not rule out exposure to the source studies, a concern shared by benchmarks built from public research. We will make this limitation clearer.

I appreciate the authors' clarifications. However, they don't fully address my concerns.

First, although the corpus may include cases from different countries, it still reflects a single editorial perspective. This may bias which cases are selected and how their causal claims are framed, which limits how well the proposed framework can be assessed and how far the results generalize.

Second, CLadder is presented as related work (Table 1), but no empirical comparison with it is provided once the design is defined under the CausalReasoningBenchmark framework. My point is not that CLadder itself must be executed, but that the authors should provide evidence that (a) the proposed framework actually handles the issues they identified in existing benchmarks; (b) it yields the same or similar causal reasoning outcomes on cases that both approaches can handle, as suggested in Table 1; and (c) whether, and how, the two approaches are complementary. The same applies to the other existing approaches listed in Table 1.

Finally, although data contamination comprises a serious issue, it should be accounted for and, at minimum, evaluated through a controlled experiment.

Reviewer_ZKDpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CausalReasoningBenchmark, a benchmark for evaluating automated causal analysis by separately assessing identification specifications and numerical estimation. It includes 174 queries across 139 datasets collected from 73 research papers and three textbooks, covering five causal inference designs. Each task requires a system to identify the appropriate research design, specify the relevant variables, and estimate the causal effect with its standard error. Experiments with GPT-5.3 show 81.0% accuracy in identifying the design family but only 37.9% correctness on complete identification specifications, with a median relative estimation error of 13.5%. These findings suggest that constructing accurate identification specifications remains a major challenge for automated causal analysis.

Critical strengths

a. The benchmark incorporates research-derived questions, real datasets, and multiple causal identification strategies. By requiring detailed identification specifications alongside effect estimates, it provides a more comprehensive evaluation of automated causal analysis.

b. The benchmark evaluates individual components of identification specifications, helping reveal errors that may be overlooked by final estimation accuracy alone. The analysis across different identification strategies and error categories provides useful insights into the limitations of current models.

Critical weaknesses

a. The evaluation relies on reference specifications that may not fully account for alternative valid adjustment sets. Additionally, some RDD errors (e.g., CATE vs. LATE) may reflect terminology differences rather than substantive identification mistakes, potentially affecting the interpretation of the reported accuracy.

b. The distinction between identification and estimation errors could be better supported. Since numerical estimates depend on the model's chosen identification strategy, estimation errors may reflect both incorrect identification and execution failures.

c. The evaluation is also limited to a single model, and some error patterns are concentrated in specific datasets, making it difficult to establish identification as the primary bottleneck.

Questions for authors

a. For the 28 CATE-to-LATE errors, could the authors clarify how many represent substantive identification mistakes rather than terminology differences? An analysis of the generated explanations and code would help distinguish these cases.

b. Could the authors evaluate estimation performance when the model is provided with the correct identification specification, and report estimation accuracy conditional on correct identification? It would also be helpful to verify consistency between the submitted specifications and executed code, and examine whether the conclusions hold across dataset families.

2 comments

We thank the reviewer for the comments.

Identification and estimation. A reasonable numerical estimate does not establish that an agent correctly understood the underlying data-generating process: numerical agreement can occur despite an incorrect design. This is a central limitation of evaluating causal reasoning through estimation alone. Our benchmark crucially highlights this limitation of prior work and addresses it by scoring the identification specification separately.

RDD estimands. A dependable causal agent must clearly specify both the target effect and the population to which it applies. In sharp RDD, conditional average treatment effect (CATE) refers to the average treatment effect at the running-variable cutoff. In fuzzy RDD, the complier conditional local average treatment efect concerns those at that cutoff whose treatment receipt changes with threshold eligibility. Local average treatment effect (LATE) in the context of IV analysis instead averages treatment effects over individuals whose treatment receipt changes with the instrument. Of the 28 CATE/LATE mismatches, seven also contain errors in other fields.

Scope. The baseline agent demonstrates the benchmark's diagnostic value by exposing errors that numerical agreement alone can obscure. We agree that a given design can admit multiple valid control sets, and the benchmark permits such variations. A control set is accepted if it includes the required controls and excludes bad controls, such as post-treatment variables.

Thank you for the response and clarifications. I appreciate the authors' explanations, but I still have two concerns.

For the 28 CATE/LATE mismatches, the authors mentioned that only 7 cases have errors in other fields. For the remaining 21 cases, I wonder how many are actual identification errors and how many are just terminology differences.

Another is that whether identification is really the main bottleneck. Could the authors report estimation performance when the identification is correct, or maybe provide the correct identification to the model and evaluate estimation again.

Reviewer_oWT8OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a benchmark for evaluating how well LLMs carry out causal analysis on data from research papers and textbooks. It evaluates both the research design specified by the model and the resulting effect estimate. The main finding is that choosing the right general strategy does not necessarily mean that the model can correctly specify the variables and other details needed for the analysis. The benchmark makes these mistakes easier to examine by evaluating the individual components of the model’s answer.

Critical strengths
  1. The paper clearly demonstrates that a model can identify the appropriate causal strategy while still making important errors in specifying the analysis. Figure 1 and the accompanying examples show how evaluating individual components reveals errors that strategy accuracy or numerical agreement alone would miss. This diagnostic detail is a substantial strength of the benchmark.

  2. The benchmark is carefully organized, with contextual metadata, reference identification specifications, and estimation code provided for each question. These resources make the evaluation transparent and allow individual cases to be examined in detail. The benchmark provides a useful foundation for evaluating causal analysis on data drawn from empirical studies and textbooks.

Critical weaknesses
  1. The breadth of the empirical evaluation is limited. The research-paper examples are drawn entirely from political science, identification strategies are unevenly represented, and the experiments evaluate only a single model. This makes it difficult to determine whether the observed error patterns reflect broader limitations of LLM-based causal analysis or are specific to the selected model and dataset composition. Although the benchmark provides useful depth, broader validation is needed to support the generality of its conclusions.

  2. The validity of the reference specifications needs clearer justification. The authors acknowledge that a question may admit more than one defensible causal analysis, but the evaluation relies on a single reference specification. It is not sufficiently clear how alternative designs or adjustment sets were considered, or how the required and forbidden control variables were validated. This leaves uncertainty about whether some rejected answers reflect genuine reasoning errors or valid departures from the reference, which matters when interpreting the full-specification accuracy.

Questions for authors
  1. Can you provide evidence or references showing how well the selected political-science studies cover the causal designs the benchmark aims to evaluate? This would help justify the domain focus beyond the availability of data.

  2. How were the reference identification specifications and the required and forbidden control sets validated? Were alternative defensible designs or adjustment sets considered? It would be helpful to know whether any rejected responses were independently reviewed to assess how often valid alternatives are marked incorrect.

  3. How does the proposed evaluation extend the diagnostics already provided by CauSciBench (ICML 2026), which separately evaluates method selection, variable selection, and effect estimation? A concrete example of an important failure captured by your scoring but missed by those metrics would help clarify the additional contribution.

1 comment

We thank the reviewer for the comments.

Coverage and baselines. The benchmark covers central causal designs using studies from established replication collections (Lal et al., Stommes et al., and Chiu et al.), and the political science literature provides a well-documented starting point. The baseline evaluation demonstrates the benchmark's diagnostic value: recognizing the broad strategy does not imply recovering a complete design. Broader domains and model comparisons are natural extensions.

Reference specifications. Each reference corresponds to a fixed causal question and the source study's identifying assumptions. We retain cases whose design can be reconstructed from the paper and replication materials, and provide code for the reference estimates. We agree that a given design can admit multiple valid control sets, and the benchmark permits such variations. A control set is accepted if it includes the required controls and excludes bad controls, such as post-treatment variables.

In comparison to CauSciBench, our additional contribution is explicit scoring of the estimand, inclusion of required controls, exclusion of bad controls, and compulsory design fields. For example, an RDD answer can select the correct method and variables but use the wrong cutoff. Even when all required controls are included, adding bad controls, such as post-treatment mediators or colliders, can invalidate the control set. This distinguishes recognizing a method from specifying the design needed to apply it. CauSciBench's released evaluator scores effect accuracy by point-estimate proximity. We additionally evaluate confidence-interval overlap, standard-error differences, and agreement on statistical conclusions, since valid estimation choices can yield different point estimates for the same causal target.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.