- Rating
- 3(+1)
- Confidence
- 1(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CertID, an evaluation pipeline for causal identification in LLMs. Given a causal graph and an interventional query, the model is expected to determine whether the query is identifiable from observational data and, if it is, provide a valid estimand. The paper focuses on two consequential failure modes: falsely refusing an identifiable query and, more importantly, confidently answering a non-identifiable query. It uses ID algorithm to generate certified labels for both identifiable and non-identifiable queries, and numerically verifies generated estimands against structural causal models with interventional effects. Across 1,200 certified graph-query instances, the evaluated frontier models decide identifiability with 97–100% accuracy, while their false-claim rates on non-identifiable queries differ.
Critical strengths
-
It is well-motivated to distinguish whether the query is identifiable or not in according to observation data, and grading estimands numerically is a sound design choice.
-
The paper is careful and candid about its limits. This transparency makes the paper a useful foundation.
Critical weaknesses
-
On the main results (Table 2), all models sit at 0.978–0.998 accuracy with false-claim rates ≤0.03. The false-claim rates may not directly comparable to an accuracy difference.
-
The setting assumes that the user already holds an explicit ADMG. The more realistic and harder case, where the graph is implicit in a natural-language study description, is not evaluated.
Questions for authors
-
The paper contains useful worked examples in Appendix H. However, the main paper would be substantially more accessible if it included one full prompt-format example.
-
Could you discuss how CertID would extend to open-ended causal QA? For example, evaluate identification from natural-language causal descriptions, raw observational datasets, or realistically incomplete/ambiguous causal graphs.
-
Including a mid-size open-weights model on the verdict task would show whether the phenomenon is general.
3 comments
Response by Authors
Dear Reviewer qyFa,
As the interactive discussion period is drawing to a close, we would like to kindly ask whether our response has adequately addressed your concerns. Please let us know if any further clarification or additional information would be helpful.
Best regards,
Authors
Dear authors, thank you for your response. The evaluation using natural language descriptions has addressed some of my concerns. My remaining concern is the paper’s unique contribution, given that this paper uses the existing ID algorithm to assess correctness. Nevertheless, the paper highlights the issue of false alarms, which I consider a useful finding. I have therefore decided to raise my score accordingly.
Response to Reviewer qyFa
W1. In Table 2, false-claim rates are at most 0.03 and may not be comparable to accuracy differences.
We agree that the two numbers are on different scales; they measure different things. Accuracy counts misses and false claims over all queries. The false-claim rate counts only false claims, over non-identifiable queries, and a false claim is the error that no observational data can expose. A small accuracy gap can therefore hide a large difference in false claims.
Because accuracy on the main pool is near ceiling, we compare the models on the fresh scale grid (General Response 1). Their accuracies lie within 6 points, while both Gemini models make significantly more false claims than GPT-5.5 (Fisher exact test, p < 0.001). At least two thirds of every model’s errors are false claims, so accuracy is driven by this error without isolating it. Sections 4.4 and 5.3 now explain this difference.
W2. Evaluation with implicit graphs described in natural language.
Thank you for the suggestion. We have added a natural-language evaluation setting (Section 4.2) using 240 main-pool instances, where models receive only a study description instead of an explicit graph. For example, "V0 and V3 are confounded by a factor that was not measured. V0 is directly influenced by V4." The descriptions are generated from the graphs, preserving certified ground-truth labels.
GPT-5.5, Gemini Pro, and Gemini Flash achieve 240/240, 238/240, and 238/240 correct identifiability verdicts, respectively, with 0, 1, and 2 false claims among 120 non-identifiable queries. Their false-claim rates are no higher than on the full main pool with explicit edge lists (0%, 2.8%, and 2.0%, respectively). Formula correctness also remains high: 115/115, 112/114, and 114/115, respectively. These results suggest that model performance is not dependent on the explicit edge-list format. The full results are in Appendix J.5, and Appendix I.4 discusses them.
Q1. Include a full prompt example.
Figure 3 in Section 4.2 shows the full prompt. Appendix J.5 gives the natural-language and step-by-step versions.
Q2. Extension to open-ended causal QA with natural-language descriptions, observational data, or incomplete/ambiguous causal graphs.
We added a discussion of this extension in Appendix I.4. Any method starting from a natural-language description or an observational dataset to an estimated effect must decide identification correctly, as more data cannot correct a wrong identification verdict. Our benchmark isolates this step so that it can be graded exactly. We have evaluated natural-language descriptions in Response to W2.
For incomplete or ambiguous causal graphs, the answer must hold for every graph that agrees with what is known. For example, if the direction of one edge is unknown, the query counts as identifiable only if it is identifiable for both directions. Since identification can be decided exactly for such sets of graphs, this setting also admits certified labels.
Raw observational data additionally requires causal discovery, a separate task preceding identification. Extending our benchmark to this setting would require ground-truth graphs and separate evaluations of causal discovery and identification.
Q3. Evaluations on a mid-size open-weights model on the verdict task.
We added evaluations on gpt-oss-20b (21B parameters), gpt-oss-120b and DeepSeek-R1, each served by a fixed provider (Section 5.4, Appendix J.1).
DeepSeek-R1 returned no verdict in 49 of 592 responses; its rates are computed only over responses containing a verdict. On the fresh grid, the gpt-oss models exhibit over 50x the false-claim rate of GPT-5.5, and 37%–47% of their checked correct claims come with a wrong formula. On the main pool, 128 of the 145 checkable false claims of the three open-weights models (88%) become correct when certain confounding edges are removed. These results show that the same failure patterns persist in open-weight models, with substantially higher error rates.