acceptodds
Under review as a conference paper at ICLR 2027

Reasoning Models Are Accurate but Unsound on Identification

Abstract

A reasoning model asked whether a causal effect is recoverable from observational data can fail in two ways: it refuses an identifiable query or answers a non-identifiable one. The latter is more consequential, as no observational data can validate the claimed formula. Measuring this failure requires queries that are provably non-identifiable, which prior evaluations lack, and grading that accepts correct formulas in any equivalent form, which string matching cannot provide. We build , a formal identification pipeline that addresses both limitations. uses the sound and complete causal identification algorithm id to certify whether an effect is identifiable from a given graph and query, and verifies returned formulas against structural causal models whose interventional distributions are known exactly. further develops theoretical results to mitigate structural leakage, repair non-identifiable queries, and establish grading guarantees. We evaluate three frontier reasoning models (Gemini Flash, Gemini Pro, and GPT-5.5) on certified instances spanning 4 to 50 vertices. Accuracy proves a poor proxy for soundness: on identical instances, the false-claim rate on non-identifiable queries varies by across models. We also find that models decide identifiability with accuracy on graphs generated after the strongest model's training snapshot. Instances, the certification procedure, the verifier, and per-instance records are available at https://anonymous.4open.science/r/certid-D718.

Primary area:
causal reasoning
Keywords:
causal inference, causal identification, benchmark, large language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qyFaOpenReview ↗
Rating
3(+1)
Confidence
1(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CertID, an evaluation pipeline for causal identification in LLMs. Given a causal graph and an interventional query, the model is expected to determine whether the query is identifiable from observational data and, if it is, provide a valid estimand. The paper focuses on two consequential failure modes: falsely refusing an identifiable query and, more importantly, confidently answering a non-identifiable query. It uses ID algorithm to generate certified labels for both identifiable and non-identifiable queries, and numerically verifies generated estimands against structural causal models with interventional effects. Across 1,200 certified graph-query instances, the evaluated frontier models decide identifiability with 97–100% accuracy, while their false-claim rates on non-identifiable queries differ.

Critical strengths
  1. It is well-motivated to distinguish whether the query is identifiable or not in according to observation data, and grading estimands numerically is a sound design choice.

  2. The paper is careful and candid about its limits. This transparency makes the paper a useful foundation.

Critical weaknesses
  1. On the main results (Table 2), all models sit at 0.978–0.998 accuracy with false-claim rates ≤0.03. The false-claim rates may not directly comparable to an accuracy difference.

  2. The setting assumes that the user already holds an explicit ADMG. The more realistic and harder case, where the graph is implicit in a natural-language study description, is not evaluated.

Questions for authors
  1. The paper contains useful worked examples in Appendix H. However, the main paper would be substantially more accessible if it included one full prompt-format example.

  2. Could you discuss how CertID would extend to open-ended causal QA? For example, evaluate identification from natural-language causal descriptions, raw observational datasets, or realistically incomplete/ambiguous causal graphs.

  3. Including a mid-size open-weights model on the verdict task would show whether the phenomenon is general.

3 comments
AuthorsOpenReview ↗

Response to Reviewer qyFa

W1. In Table 2, false-claim rates are at most 0.03 and may not be comparable to accuracy differences.

We agree that the two numbers are on different scales; they measure different things. Accuracy counts misses and false claims over all queries. The false-claim rate counts only false claims, over non-identifiable queries, and a false claim is the error that no observational data can expose. A small accuracy gap can therefore hide a large difference in false claims.

Because accuracy on the main pool is near ceiling, we compare the models on the fresh scale grid (General Response 1). Their accuracies lie within 6 points, while both Gemini models make significantly more false claims than GPT-5.5 (Fisher exact test, p < 0.001). At least two thirds of every model’s errors are false claims, so accuracy is driven by this error without isolating it. Sections 4.4 and 5.3 now explain this difference.

W2. Evaluation with implicit graphs described in natural language.

Thank you for the suggestion. We have added a natural-language evaluation setting (Section 4.2) using 240 main-pool instances, where models receive only a study description instead of an explicit graph. For example, "V0 and V3 are confounded by a factor that was not measured. V0 is directly influenced by V4." The descriptions are generated from the graphs, preserving certified ground-truth labels.

GPT-5.5, Gemini Pro, and Gemini Flash achieve 240/240, 238/240, and 238/240 correct identifiability verdicts, respectively, with 0, 1, and 2 false claims among 120 non-identifiable queries. Their false-claim rates are no higher than on the full main pool with explicit edge lists (0%, 2.8%, and 2.0%, respectively). Formula correctness also remains high: 115/115, 112/114, and 114/115, respectively. These results suggest that model performance is not dependent on the explicit edge-list format. The full results are in Appendix J.5, and Appendix I.4 discusses them.

Q1. Include a full prompt example.

Figure 3 in Section 4.2 shows the full prompt. Appendix J.5 gives the natural-language and step-by-step versions.

Q2. Extension to open-ended causal QA with natural-language descriptions, observational data, or incomplete/ambiguous causal graphs.

We added a discussion of this extension in Appendix I.4. Any method starting from a natural-language description or an observational dataset to an estimated effect must decide identification correctly, as more data cannot correct a wrong identification verdict. Our benchmark isolates this step so that it can be graded exactly. We have evaluated natural-language descriptions in Response to W2.

For incomplete or ambiguous causal graphs, the answer must hold for every graph that agrees with what is known. For example, if the direction of one edge is unknown, the query counts as identifiable only if it is identifiable for both directions. Since identification can be decided exactly for such sets of graphs, this setting also admits certified labels.

Raw observational data additionally requires causal discovery, a separate task preceding identification. Extending our benchmark to this setting would require ground-truth graphs and separate evaluations of causal discovery and identification.

Q3. Evaluations on a mid-size open-weights model on the verdict task.

We added evaluations on gpt-oss-20b (21B parameters), gpt-oss-120b and DeepSeek-R1, each served by a fixed provider (Section 5.4, Appendix J.1).

ModelMain pool: accuracyMain pool: false claimsFresh grid: accuracyFresh grid: false claims
gpt-oss-20b83.5%53/300 (17.7%)71.3%117/300 (39.0%)
gpt-oss-120b84.8%44/300 (14.7%)74.1%107/299 (35.8%)
DeepSeek-R183.1%63/265 (23.8%)not runnot run
GPT-5.599.8%0/30099.7%2/300 (0.7%)

DeepSeek-R1 returned no verdict in 49 of 592 responses; its rates are computed only over responses containing a verdict. On the fresh grid, the gpt-oss models exhibit over 50x the false-claim rate of GPT-5.5, and 37%–47% of their checked correct claims come with a wrong formula. On the main pool, 128 of the 145 checkable false claims of the three open-weights models (88%) become correct when certain confounding edges are removed. These results show that the same failure patterns persist in open-weight models, with substantially higher error rates.

AuthorsOpenReview ↗

Response by Authors

Dear Reviewer qyFa,

As the interactive discussion period is drawing to a close, we would like to kindly ask whether our response has adequately addressed your concerns. Please let us know if any further clarification or additional information would be helpful.

Best regards,

Authors

Dear authors, thank you for your response. The evaluation using natural language descriptions has addressed some of my concerns. My remaining concern is the paper’s unique contribution, given that this paper uses the existing ID algorithm to assess correctness. Nevertheless, the paper highlights the issue of false alarms, which I consider a useful finding. I have therefore decided to raise my score accordingly.

Reviewer_cPcNOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors assert that existing procedures for evaluating causal reasoning in LLMs do not assess performance on non-identifiable queries. To fill this gap, they propose CERTID, a benchmark whose ground-truth labels are certified by the ID algorithm. Each CERTID instance presents an LLM with a graph and causal query and asks whether the query is identifiable. If the query is identifiable, the LLM is further asked to provide an estimand. The paper’s results shine favorably on the causal reasoning abilities of frontier LLMs circa April 2026. All three models achieved accuracies in the high 90s, albeit with some variance in their false-claim rates.

Critical strengths

The authors provide a rigorous assessment of causal reasoning in LLMs. They use the ID algorithm, a workhorse of the causal inference community, to certify ground truth labels in a way that they claim is novel. If true, their approach is long overdue. In contrast to other papers in this space, the paper does not denigrate LLMs in the “stochastic parrot” tradition but rather evaluates their abilities in a fair and clean manner (no string matching, no training data contamination). The work raises questions about how the LLMs come to their answers. Do they rely on heuristic approaches, or do they formally walk step-by-step through ID-like algorithms?

Critical weaknesses

Some of the results seem inconsequential to the paper’s central focus. For example, the authors briefly highlight work on the repair task and experiments on small models. But they do not tie these threads back to the main conclusion, and they feel like a bit of a distraction as a consequence.

Questions for authors

In Section 5.3, you state that “even the best model is not sound,” since no model achieved a false-claim rate of 0.0. But the best-performing model, GPT 5.5, asserted an answer to just three provably unanswerable queries out of 300.

  1. The neural networks underlying reasoning models are not deductive systems. They can certainly make false positives. But is “soundness” the right term here?

  2. In Section 5.2, you state that all three models gave structural reasons for their answers upon modifying the prompt. To what extent can the prompt explain trends in accuracy and false-claim rate? Might a different prompt have allowed GPT 5.5 to answer those last three questions correctly? Even if it did answer them correctly, would we have the license to call it “sound” for causal identification queries?

  3. The ID algorithm is sound and complete. Why do you need to run 2 implementation of this algorithm?

  4. The ID algorithm was presented by Jin Tian in his thesis . The version by Shpitser came later. You should cite the original publication by Tian in 2002 in his thesis.

" Tian, J. 2002. Studies in Causal reasoning and Learning. Ph.D. thesis, University of California, Los Angeles."

1 comment
AuthorsOpenReview ↗

Response to Reviewer cPcN

W1. Connecting repair tasks and experimental results to the main findings.

We agree that the original submission did not sufficiently connect the repair task to our main findings. We now establish this connection through Theorem 1 and additional experiments.

Theorem 1 characterizes the minimal sets of bi-directed edges whose removal makes a query identifiable (minimal repairs). If a model’s formula is correct after some edges are deleted, it stays correct after more are deleted. For each false claim, we therefore delete all bi-directed edges and restore them one at a time while the formula stays correct. Of 611 false claims across all experiments, 448 are computationally tractable for exact verification:

  • 305 (68%) become correct when certain confounding edges are removed.
  • For 171, removing a single bidirected edge suffices; for 217, the removed edges form a minimal repair characterized by Theorem 1.
  • The remaining 143 remain incorrect even after all bidirected edges are removed.

This pattern holds across experiments: 53/71 (75%) in the original submission and 128/145 (88%) for open-weight models on the main pool.

These findings reveal that many false claims are not arbitrary errors but formulas that become valid under unstated assumptions about the absence of specific confounders. Theorem 1 helps identify these implicit assumptions, directly connecting the repair task to our main empirical findings.

Q1. Is "soundness" the right term?

We use soundness in its standard logical sense: a decision procedure is sound if it never falsely declares a non-identifiable query identifiable. Unlike ID, neural models lack theoretical soundness guarantees. We therefore assess their soundness empirically using the false-claim rate, which measures the frequency of soundness violations. We have clarified this terminology in Definition 5.

Q2. Impact of prompting on accuracy, false claims, and soundness.

a) To what extent can prompting explain accuracy and false-claim rates? GPT-5.5's false-claim rate is 1.0% (95% CI: 0.3%–2.9%) on the original scale grid and 0.7% on the fresh grid. Although small, such errors remain consequential because observational data cannot resolve non-identifiability. The strongest models approach zero false claims but do not eliminate them: Claude Opus 5.5 makes 1 false claim in 599 non-identifiable queries, and GPT-6.1-sol makes 3 in 800. In contrast, Gemini models exhibit rates of 5.7%–8.0%, and open-weight models 36%–39% on the fresh grid.

b) Could different prompts correct GPT-5.5's errors? We compare the default prompt with two alternatives: the justification prompt in Section 5.2 and a prompt explicitly instructing models to execute ID step by step. Neither significantly changes false-claim rates beyond rerun variability (paired McNemar tests, p≥0.27p \geq 0.27). GPT-5.5 consistently exhibits the lowest false-claim rates across prompts, suggesting that the observed model differences are not explained by these prompting strategies. Under step-by-step prompting, GPT-5.5 makes 3 false claims compared with 2 under the default prompt (Section 5.2).

c) Would eliminating these errors establish soundness? No. Soundness requires that a model never falsely declares a non-identifiable query identifiable. Zero errors on a finite test set cannot establish this guarantee. For example, even zero false claims among 300 queries permits a true error rate of up to 1.3% (95% upper confidence bound). Indeed, Claude Opus 5.5 makes no false claims on the 300 fresh-grid queries but makes one on the 299 main-pool queries. Thus, empirical evaluation can estimate or bound false-claim rates, but cannot establish theoretical soundness. Section 5.3 now states this point.

Q3. Why two ID implementations?

Although the ID algorithm is theoretically sound and complete, its implementations may contain errors. For example, we found that the y0 library produces numerically incorrect formulas on 12 of 269 fresh-grid instances, despite returning correct identifiability verdicts. The ananke implementation produces correct formulas on all 12 instances. We therefore use two independent implementations to cross-check the results and guard against implementation errors. Both implementations agree on all identifiability verdicts. Section 3.1 states this reason.

Q4. Cite Tian (2002).

Thanks for the suggestion. We now credit ID to Tian (2002, thesis) and Tian and Pearl (2002) in Sections 1 and 2 and Appendix B.

Reviewer_riTiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the accuracy of LLMs in determining identifiability and finding formulas for causal effects. Key theoretical results include “hedge destruction,” which characterizes bidirected-edge removals that make target causal effects identifiable. Another result justifies checking identifying formulas by comparing their outputs with ground truth under random parameterizations. The paper reports empirical results on the accuracy and soundness of LLM identifiability checks.

Critical strengths
  1. The paper contains both theoretical contributions and extensive empirical studies. The theoretical results appeared sound to me.
  2. Theorem 2 provides an interesting method for detecting incorrect identifying formulas for an SCM, though it requires an absolutely continuous sampling distribution over the parameters to attain the guarantee (i.e., excluding positive mass on equal parameters).
Critical weaknesses
  1. The key theoretical contributions concern characterizing the sets of bidirected edges whose removal restores identifiability (the repair task and Theorem 1) and detecting incorrect identifying formulas (Theorem 2). However, the empirical studies focus mainly on the accuracy and soundness of LLM identifiability verdicts. Formula verification is discussed only in Lines 410–413, while repair results are covered briefly in Lines 475–477. I suggest expanding these discussions so that the empirical section better matches the theoretical contributions.
  2. The repair task is not sufficiently motivated. Deleting a bidirected edge amounts to assuming that certain unobserved confounding is absent. This can lead to incorrect identifiability results. The paper should clarify how such assumptions can be justified and why the original graphs can overstate the confounding.
  3. The empirical setup is somewhat limited. It considers only one treatment variable and one outcome variable per query, which limits the generality of the findings. The main random-graph evaluation contains at most 13 variables, and even though it's extended to 50 later, the relevant graph still contains at most 15.1 variables in average. Evaluations involving multiple treatments or outcomes would strengthen the results.
  4. Lines 19-20 was confusing to me: why do we need to know the interventional distribution?
  5. In Lines 153–159, the notion of “string matching” was never introduced. Literal exact matching can reject equivalent correct formulas, but it is unclear how it could accept an incorrect estimand. Also, in Definition 4, shouldn't τ\tau be zero in the theoretical sense? My understanding is that it's only needed due to floating-point precision. Finally, Definition 4 should specify how M1, … M_k are picked.
  6. Section 3 is titled “Theoretical Foundation,” but Sections 3.1–3.2 primarily describe benchmark construction and diagnostics that only become relevant in the empirical study. Reorganizing these sections would make the paper’s structure clearer.
Questions for authors

Given that the paper recommends using ID in practice, what practical role remains for LLM-generated estimands? Do they offer any advantage over ID’s outputs, such as returning shorter identifying formulas?

3 comments
AuthorsOpenReview ↗

Response to Reviewer riTi W1-W6

W1. Expanding discussions on experiments to match theory. Thank you for the suggestion. We have added experiments corresponding to each theorem in Sections 5.1 and 5.4 in the revised manuscript, with detailed results in Appendices J.3–J.4.

  • Experiment to match Theorem 1 (Repair). The theorem characterizes minimal sets of bidirected edges whose removal makes a query identifiable. Among 448 checkable false claims, 305 become correct after removing certain edges, and 217 of these correspond exactly to a minimal edge-removal set characterized by Theorem 1.
  • Experiment for Theorem 2 (Formula verification). The theorem establishes how incorrect formulas can be detected through evaluation on randomly sampled causal models. We now verify formulas from all evaluated models, including GPT-5.5 and those in the scale grid experiments (not assessed in the original submission). As a positive control, we grade ID’s own formulas in the same way. The expression printed by y0 is wrong on 12 of 269 checkable fresh-grid instances, and the expression of the second implementation (ananke) is correct on all 12 (see our reply to Reviewer cPcN, Q3).

W2. Why repair, when deleting a bi-directed edge assumes no confounding.

We thank the reviewer for this important comment and agree with both concerns. Removing a bidirected edge X↔ZX \leftrightarrow Z assumes the absence of unmeasured confounding between XX and ZZ, which may not hold in practice. We have clarified the motivation for the repair task in Section 3.3. Importantly, Theorem 1 does not suggest that edges should be removed; rather, it identifies which confounders, if absent, would make the effect identifiable. Whether these assumptions hold must be justified by domain knowledge. The repair task is useful in three aspects:

  1. Guiding measurement. Minimal repairs identify the smallest sets of confounding relationships that must be addressed to achieve identifiability, through additional measurements, randomization, or domain knowledge. For example, Robins et al. (2000) assume no unmeasured confounding of the treatment, whereas a weaker assumption suffices for their graph (Appendix H.4).
  2. Explaining conservative graphs. Analysts may include bidirected edges when confounding cannot be ruled out, since omitting genuine confounding can invalidate causal conclusions, whereas including unnecessary edges may only prevent identifiability. Minimal repairs highlight which uncertain edges are worth examining.
  3. Explaining model errors. 305 of 448 checkable false claims (68%) become correct under graphs with specific confounding edges removed. Theorem 1 identifies the implicit assumptions underlying these incorrect claims.

W3. Evaluations with multiple treatments or outcomes.

We introduce a new challenging evaluation grid of 400 queries with one or two treatments and one or two outcomes, using graphs with 30 or 50 vertices and 20–36 ancestors of Y. False claims and misses (of 200 each): GPT-5.5 1 and 2, gpt-6.1-sol 1 and 0, Gemini Pro 10 and 36, Gemini Flash 17 and 30 (Gemini Flash answered 387 of 400 queries; 13 calls timed out). Two treatments or two outcomes do not raise GPT-5.5's rate. Details are in Section 5.3 and Appendices J.1–J.2.

W4. Why the interventional distribution is needed.

To verify a formula, we need the true value P(y | do(x)). In an SCM whose latent variables are explicit, this value can be computed exactly and compared with the formula's value. We have clarified this in the abstract.

W5. String matching, choice of τ\tau, and sampling of M1,…,MkM_1,\ldots,M_k in Definition 4.

We have clarified all three points in the revised manuscript.

String matching. Lines 159-161 under Definition 3 defines string matching as a syntactic comparison between the LLM-generated formula and the output of a reference implementation, without checking mathematical equivalence. We agree that string matching can accept an incorrect formula only when the reference itself is incorrect, and have corrected our earlier statement. This occurs for y0 in 12 of 269 fresh-grid instances. However, its primary limitation is false rejection: 73% of correct model-generated formulas differ syntactically from the reference and are therefore rejected.

Choice of τ\tau. The theorem uses τ=0\tau=0, while experiments use τ=10−12\tau=10^{-12} solely to account for numerical rounding. Correct formulas deviate by at most 1.1×10−131.1\times10^{-13}, while incorrect ones are at least 4.5×10−84.5\times10^{-8}.

Sampling of M1,…,MkM_1,\ldots,M_k. Definition 4 now specifies that the causal models are sampled independently, with each conditional probability table drawn from an absolutely continuous distribution. Section 4.5 specifies our sampling procedure (0.12+0.76$U$), where UU is uniform, followed by normalization) and k=4.

W6. Reorganize Section 3.

Thanks for the suggestion. Please see the revised version.

AuthorsOpenReview ↗

Response to Reviewer riTi Q

Q. Roles and advantages of LLM-generated estimands when ID exists.

LLM-generated estimands offer two key benefits. First, users increasingly rely on LLMs to answer causal questions end-to-end, making it important to evaluate the soundness of their generated estimands. Second, LLM-generated formulas are substantially more concise, with a median length of 12–21 characters compared to 71 for ID-generated formulas (Appendix J.3). LLMs often produce intuitive adjustment formulas, whereas ID returns more complex expressions involving nested summations.

Moreover, when Gemini Flash can call ID as a tool, every repair it returns is valid and minimal at all graph sizes we tested (Section 5.4, Appendix G.9). These findings suggest a complementary workflow in which LLMs propose and explain causal estimands, while ID verifies their correctness.

AuthorsOpenReview ↗

Response by Authors

Dear Reviewer riTi,

As the interactive discussion period is drawing to a close, we would like to kindly ask whether our response has adequately addressed your concerns. Please let us know if any further clarification or additional information would be helpful.

Best regards,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: Revision & New Experiments

Dear AC and PCs,

We have uploaded a revised manuscript with changes highlighted in blue. We have also evaluated two additional latest frontier models and three open-weights models. The new experiments comprise approximately 12,000 model calls across eight models on 1,000 new instances. Detailed results are provided in Appendix J.

Below we summarize the major changes in the revised PDF:

  • Section 1: why two ID implementation; new findings on false claims. ID algorithm credited to Tian (2002, thesis) and Tian and Pearl (2002).
  • Section 2: string matching defined; explain τ\tau = 0 in theory and how the SCMs are drawn; new Definition 5 (soundness); ID algorithm credited to Tian (2002, thesis) and Tian and Pearl (2002).
  • Section 3: retitled “Construction and Theory”; reason for two ID implementations (3.1); motivation for repair (3.3).
  • Section 4: fresh scale grid and hard grid (4.1); additional models, natural-language condition, step-by-step prompt, and the full prompt in Figure 3 (4.2); verifier settings and ancestral reduction (4.3); false-claim vs accuracy differences (4.4).
  • Section 5: estimand verification for all models with a positive control (5.1); prompt effects (5.2); replication, hard grid, open-weights models and soundness (5.3); repairs, what false claims assume, results on two more latest models and three open-weights models (5.4).
  • Appendix I: discussions on open-ended QA (I.4) and limitaions on latest models (I.5).
  • Appendix J (new): full results on new models and instances, and prompt variants.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.