- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces CausalVerify, a benchmark for evaluating LLM-generated causal analyses across four dimensions: method recognition, identification specification, executable implementation, and numerical estimation. It includes 259 real-paper contexts, 100 synthetic scenarios, and a paired evaluation involving 23 synthetic twins and nine models. The paired protocol compares model-generated analyses with an oracle setting that provides reference designs and execution conventions.
Critical strengths
a. The paired protocol evaluates model-generated and reference-guided analyses on matched tasks, allowing a more informative comparison of identification and execution performance. The use of consistent evaluation conditions and careful reporting of denominators strengthens the experimental analysis.
b. The paper examines scoring sensitivity, model and task differences, and potential evaluation errors, while also including prospective experiments. The authors explicitly acknowledge limitations and report unfavorable findings, which strengthens the credibility of the evaluation.
Critical weaknesses
a. The benchmark evaluates agreement with reference estimates and standard errors, while the oracle receives execution conventions that are not fully specified to the main agent. Consequently, some failures may reflect reasonable alternative implementations rather than incorrect causal reasoning.
b. The paired reference designs, synthetic tasks, and adjudication procedures rely substantially on agents from the same model family, with limited independent expert validation. The sensitivity of reported success rates to adjudication decisions raises questions about the robustness and generalizability of the evaluation.
Questions for authors
a. Could the authors clarify how many failures reflect incorrect causal identification or implementation, rather than differences in unspecified estimation conventions? A targeted analysis, particularly for RDD, and a comparison where both arms receive the same execution conventions would help clarify the contribution of reference-design information.
b. Could the authors provide an independent expert assessment of a subset of paired reference designs, synthetic tasks, and adjudication decisions? Additionally, how is consistency between the committed causal-design representation and the subsequently executed code verified?
c. Could the authors provide a breakdown of the remaining 48.5% oracle failures, distinguishing runtime issues, interface restrictions, truncation, and numerical mismatches? This would help clarify how much of the remaining difficulty reflects substantive causal-analysis errors rather than execution constraints.
Author response to Reviewer XLmq
Thank you for asking how scoring differences arise. We traced the original RDD submissions and revised Figure 3 to show where oracle runs fail.
What the RDD failures measure. The oracle receives both the reference design and pinned execution conventions. On the six original RDD pairs, 370/1,060 eligible main-arm seeds failed the primary check after execution, versus 0/1,060 for the oracle. Removing 68 failures on the diagnostic-target pair and 20 non-finite estimates leaves 282 finite point misses on five conventional RDDs. We examined all 16 historical main-arm scripts that produced those misses against their frozen commitments, available variables and registered estimators. Across 320 seed runs we independently reconstructed the point outputs, including placeholders. This post hoc, AI-assisted review selects scripts with a finite miss. Each script is reused across 20 seeds (Appendix D).
Two scripts returned invalid primary outputs. One reported zero cost per life saved and zero SE when the twin contained no spending data. Another regressed the age-65 treatment indicator on itself after retaining only treated observations. Two debt-panel scripts each omitted one fixed effect explicitly named in the data dictionary. Adding just the omitted term to each regression reproduced all 20 registered points per script. On the election twin all 20 recomputed IK bandwidths exceeded o3's committed cap of 0.05. Its triangular local-linear fit reproduced every submitted point. Sixteen estimates missed the full-sample quartic reference. Election-year effects in its commitment could not be checked because the twin supplied no year.
Of the other eleven scripts, five combined several discrepancies including identifiable implementation departures, and four changed outcome scale or returned only an observable component. On the Medicare twin two programs estimated the same jump in raw admissions on all 20 seeds. On seed 0 it was 52.12 admissions per age cell, while the reference was 0.03786 log points. We leave two Peru scripts unattributed because the brief came from a replication and the reference targeted the original study. These cases do not identify a population share of failures due to incorrect causal identification.
Common inference conventions. In a separate election-twin pilot, two fresh Codex implementations shared the same clustering, joint CR1 formula and conventional point-estimate rule. This overrode the original local design's bias-correction requirement. Both scripts reproduced their assigned estimator on all 20 datasets. The local fit passed the registered joint check on 15/20 and the reference quartic on 20/20. On the full original panel the point-only gain was +32.6 pp [23.0, 42.8]. Design and conventions still change together.
Independent assessment and commitment fidelity. Completed human audits cover paper labels (method κ = 0.61, sign κ = 0.29) and R coefficient extraction (40/44 numeric agreements). The prepared external packet covers 11 designs with eight questions for each of two independent economists. It includes a data dictionary unavailable at Stage 1. Ratings are pending. Twin generators and individual adjudications still await independent assessment. Of 187 commitments, 23 passed the raw design verdicts, 55 after removing the inference field, and 140 after the full review. Review voided 318 field mismatches. V4 still scores SEs. Internal re-adjudication agreed on 120/123 sampled critical-field matches. All three disagreements were corrected. The hash identifies the design passed to the coding model. Table 15 shows the limit of that guarantee. GPT-5 committed to average and wage-specific employment effects, reported the average, documented missing baseline wages for the second effect, and passed on all 20 datasets. A field-by-field fidelity audit across the panel remains open.
The 48.5% residual. Figure 3(c) and Table 16 account for 1,814 failures among 3,740 eligible oracle seeds. Of these, 54 missed the primary target, 80 met it but missed the SE-ratio band, 940 encountered a forbidden import, 500 another runtime exception, 140 a capped generation, 60 a timeout, 40 an extraction-rule failure and zero missing JSON. The runner accepted no output on 1,680/1,814 failures (92.6%). Of 134 accepted failures, 53 had a non-finite estimate or SE and 81 had two finite values. In total, 1,733/1,814 failures (95.5%) lacked an accepted finite estimate and SE. With fresh generations on the same 207 cells under the expanded harness, oracle execution rose from 57.9% to 75.8% and recovery from 52.7% to 67.7%. Several harness settings changed together.