- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes CausalReasoningBenchmark, a causal benchmark to assess the causal reasoning of large language models. Unlike existing benchmarks, the proposed benchmark aims to distinguish causal reasoning errors from numerical execution errors by focusing on scoring structure identification, treatment outcome, and control variable identification. The queries are from different experimental designs, extracted from research papers and causal inference textbooks. Experimental results on a state-of-the-art language model show that while the model could correctly identify high-level strategy in 81% of cases, its performance drops to 38% when trying to fully identify specification correctness.
Critical strengths
- The benchmark moves from the standard evaluation of average treatment effect (ATE) estimate to evaluate the full experimental design;
- The queries come from different datasets published in peer-reviewed papers and causal inference textbooks, covering full causal experimental designs, such as instrumental variables, difference-in-differences, regression discontinuity design, and randomized controlled trials.
- It is a complementary strategy to assess the capacity of language models to specify causal experimental designs by identifying the necessary instrumental design.
Critical weaknesses
- The benchmark questions are limited to one domain and a single country's point of view. Although this does not diminish the benchmark's value, it severely limits its application.
- The experiment is limited to one language model, and there is no comparison with any of the other benchmarks listed in Table 1, in the related work section, or even closely related work not mentioned in the paper, such as CausalVerify [1].
- There is no conditional analysis to identify when the errors are due to implementation issues, for instance.
- It is unclear whether the model produced its estimates via code execution or built-in statistical functions, as the benchmark includes executing scripts.
- The paper comprises many repeated statements.
[1] CausalVerify: End-to-End Verification of Causal Analyses by Language Models
Questions for authors
- How would the proposed benchmark help handle the issue of data contamination and memorization when trying to assess the causal capability of language models? How was such an issue considered in the experiment?
- What would be the results of the benchmark in a multi-agent configuration? How different would the results be if one used CLadder instead of the proposed benchmark?
2 comments
I appreciate the authors' clarifications. However, they don't fully address my concerns.
First, although the corpus may include cases from different countries, it still reflects a single editorial perspective. This may bias which cases are selected and how their causal claims are framed, which limits how well the proposed framework can be assessed and how far the results generalize.
Second, CLadder is presented as related work (Table 1), but no empirical comparison with it is provided once the design is defined under the CausalReasoningBenchmark framework. My point is not that CLadder itself must be executed, but that the authors should provide evidence that (a) the proposed framework actually handles the issues they identified in existing benchmarks; (b) it yields the same or similar causal reasoning outcomes on cases that both approaches can handle, as suggested in Table 1; and (c) whether, and how, the two approaches are complementary. The same applies to the other existing approaches listed in Table 1.
Finally, although data contamination comprises a serious issue, it should be accounted for and, at minimum, evaluated through a controlled experiment.
We thank the reviewer for the comments.
Scope. The benchmark covers central causal designs using reproducible empirical studies. Political science provides the initial corpus, with examples spanning Japan, Brazil, China, and other settings. Our contribution is a framework for evaluating complete identification specifications and numerical estimation.
Execution and baselines. As described in Section 7, the model only gets access to the data, a metadata file describing the data and a causal question, it does not have access to the estimation code- the estimation code is provided as reference and for transparency around how we constructed the numerical solution. We use this baseline to demonstrate the benchmark's diagnostic value: an agent can produce a reasonable effect estimate while failing to specify the underlying identification design correctly.
Related work. CLadder focuses on formal causal queries over graphs, while CausalVerify includes executable synthetic scenarios and paper-linked synthetic twins. Our research-paper tasks build on empirical study datasets, pairing them with detailed identification specifications and executable reference code. We will clarify how this focus complements the settings covered by these benchmarks.
Contamination. Our evaluation requires more than recalling a published estimate, since the agent must also specify the design, variables, controls, and estimand. However, this does not rule out exposure to the source studies, a concern shared by benchmarks built from public research. We will make this limitation clearer.