acceptodds
Under review as a conference paper at ICLR 2027

Evaluating Causal Reasoning in LLMs: A Case Study in the Challenges of Procedural Generation

Abstract

Large language models (LLMs) appear to represent and reason about a large range of phenomena. In particular, recent studies have examined whether LLMs can accurately reason about causation. One of the most commonly used benchmarks for evaluating formal causal reasoning in LLMs is CLADDER, a framework for procedural generation of benchmark data sets that was released in 2023. Experiments with CLADDER appear to indicate that LLMs can perform formal causal reasoning in a large percentage of generated examples. In this paper, we examine what can be reliably inferred when LLMs perform well on test sets produced by CLADDER. We identify several examples of superficial and highly predictive patterns that the CLADDER generator embeds in data sets it generates. We find that even simple models such as random forests perform well on CLADDER-generated data sets after training on a sample of generated data. We also find that the performance of many commercial LLMs on CLADDER test sets is extremely brittle and that removing large sections of the text that are critical for formal causal reasoning has almost no effect on performance. Given these results, the most likely hypothesis for the strong performance of state-of-the-art LLMs on CLADDER is that they are exploiting superficial patterns learned from exposure to training data from the generator. Any inference that LLMs are exhibiting more complex causal reasoning must rule out this hypothesis.

Primary area:
datasets and benchmarks
Keywords:
Causal Reasoning, Evaluation Methodology, Procedural Generation, Shortcut Learning, Benchmark Evaluation, LLM Evaluation, Data Contamination

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7Za8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether high performance on CLADDER reflects genuine causal reasoning or shortcut learning from procedurally generated data. The authors identify predictive patterns in the generator, demonstrate that simple models can achieve strong performance, and evaluate LLMs through fine-tuning, information removal, token ablations, and transfer to CounterBench.

Critical strengths

The paper addresses an important problem in evaluating LLM reasoning capabilities. The analysis of CLADDER's generation process is particularly valuable, as it identifies concrete sources of answer predictability beyond simple linguistic patterns. The strong random forest baseline and information-removal experiments provide useful evidence that high benchmark accuracy alone is insufficient to establish causal reasoning ability.

Critical weaknesses
  1. There's a very strong claim "The most likely hypothesis for the strong performance of state-of-the-art LLMs on CLADDER is that they are exploiting superficial patterns learned from exposure to training data from the generator." However, after reading this paper, I find this claim is more like a deduction rather than a well-grounded conclusion. The paper convincingly shows that CLADDER contains shortcuts that simple models can exploit. However, this does not establish that high-performing LLMs actually rely on these shortcuts, much less that shortcut learning is their primary reasoning mechanism. For example, the random forest achieves 86.8% overall accuracy but only 48.6% on deterministic counterfactual questions, where reasoning models exceed 90%. This substantial difference remains unexplained.
  2. I don't quite understand the "Evaluations Confounded by Exposure" in Figure 1? May you elaborate?
  3. The analysis in 3.4 and 3.5 are still very shallow. Despite the visualization, they stay at the observation level without providing a mechanistic explanation. The authors remove information needed to solve causal questions but still evaluate models against the original answers. Above-chance accuracy may simply reflect remaining statistical patterns in the dataset. This does not necessarily mean the models are unable to perform causal reasoning when given complete information.
  4. The cross-benchmark results are difficult to interpret. While frontier LLMs perform near chance on CounterBench, a random forest trained on superficial CLADDER features achieves 79.1% accuracy. This suggests that some predictive patterns transfer across the two benchmarks, despite the authors' emphasis on generator-specific shortcuts. Moreover, CLADDER and CounterBench differ in their causal tasks and question formats, and the models are evaluated under different training conditions. The observed performance gap therefore does not clearly establish whether LLMs rely on memorized shortcuts, fail to generalize their causal reasoning, or struggle with the different task formulation. This distinction deserves more careful analysis. I think LLM's causal reasoning capability can't be discussed without studying the context you give to them, like harness.
Questions for authors

please check weakness

2 comments

Thank you for your thoughtful comments and detailed feedback.

You note that the evidence we provide “does not establish that high-performing LLMs actually rely on these shortcuts, much less that shortcut learning is their primary reasoning mechanism.” We agree, but that is not the point of the paper. Our primary point (lines 89-97) is that insufficient evidence exists for the explicit and implicit claims of many published papers (namely, that high performance on CLadder implies formal causal reasoning in LLMs). Instead, we point out that an alternative inference is equally plausible: Exposure to data drawn from a generator that embeds patterns that are superficial, learnable, and highly predictive. We show that this latter inference is plausible, though not definitively proven. This calls into question the claims of dozens of papers.

You note that you don't quite understand the "Evaluations Confounded by Exposure" in Figure 1 and ask us to elaborate. We mean that possible exposure provides an alternative explanation for the performance that is often attributed to learning formal causal reasoning. Such alternative explanations are often labeled “confounds” in experimental design. We will update the text to make that clear. The CLadder generator and example datasets were released publicly before many of the models we evaluated were trained, which means that learning generator-specific patterns through exposure during training is an alternative explanation for later performance. We will clarify this in the figure and caption.

You note that “The analysis in 3.4 and 3.5 are still very shallow.” We agree that the analysis in 3.5 is merely suggestive. However, the analysis in 3.4 is far different. As we note in that section (329-330): “we remove key components of the queries that leave them unsolvable by formal causal reasoning, and we find that such ablations produce no substantial drop in LLM performance.” The experiments in Section 3.4 are based on the formal theory of identifiability that CLadder is based on and that clearly identifies which information is critical to correctly and uniquely answering causal queries. We show in this section that removing information that is absolutely essential for identifiability (assumptions, graph structure, and probability values) either has little effect on LLM accuracy on CLadder (in the case of assumptions and graph structure) or results in accuracy that is still substantially above random (probability values).

You note that “The cross-benchmark results are difficult to interpret.” We agree this section could benefit from some additional clarification, and we will make those changes. The most important point is that simple and highly-predictive features exist and are easy to find in both datasets, yet the models only perform well on one. We have updated Section 3.6 to show that simple patterns exist in both datasets, and that the models' performance does not transfer between them.

Consider the examples below from each dataset. The similarities in structure and wording make it clear why it is easy to find simple features that transfer between them, and also why it is so surprising when an LLM that can solve CLadder queries performs poorly on CounterBench.

Example CLadder query: Imagine a self-contained, hypothetical world with only the following conditions, and without any unmentioned factors or causal relationships: Rixq has a direct effect on zuph and xevu. Zuph has a direct effect on xevu. We know that rixq causes zuph. rixq and zuph causes xevu. Would an individual is not xevu if rixq instead of not rixq?

Example CounterBench query: Imagine a self-contained, hypothetical world with only the following conditions, and without any unmentioned factors or causal relationships: Ziklo has a direct effect on Blaf, Blaf has a direct effect on Trune, Trune has a direct effect on Vork, and Vork has a direct effect on Lumbo. We know that Ziklo causes Blaf, Blaf causes Trune, Trune causes Vork, and Vork causes Lumbo. Would Lumbo occur if not Ziklo instead of Ziklo?

Thanks for the response. I think most of my concerns haven't been clearly addressed, and I will keep my score.

Reviewer_ZcrXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper examines whether the strong LLM performance on CLADDER, a causal reasoning benchmark, reflects LLM's causal reasoning capability. They identify a few superficial patterns and find simple classifiers such as random forest perform well after training. They also test LLMs' performance change after removing parts of the query text. They also report LLM performance on another benchmark CounterBench that uses abstract variable names and find large preformance gap compared to CLADDER.

Critical strengths

The paper provides an extensive evaluation of different LLMs on CLADDER and conducts several experiments that modify the benchmark queries to examine how these changes affect model performance. The identified patterns are useful when interpreting performances on CLADDER.

Critical weaknesses
  1. This paper has several central claims that are not supported by the experiment design. For example, Section 3.2 shows that classifers using superficial features can perform well, but does not establish that fine-tuned LLMs actually rely on those features. Section 3.3 does not show direct evidence of data exposure. Section 3.4, removing the introductory sentence or the causal graph sentence does not eliminate the semantic information in the remaining query.

  2. Shortcut learning, spurious correlation, benchmark contamination, and sensitivity to input features are well-known issues in LLM evaluation, including reasoning becnhmarks and other NLP tasks. The paper provides a detailed case study of CLADDER, but its contributin is unclear with the presence of other works studying limitations with CLADDER and proposing other benchmarks (that this paper has cited).

Questions for authors

Please see the weaknesses above.

2 comments

Thank you for the valuable comments and feedback.

You note that “Section 3.2 shows that classifiers using superficial features can perform well, but does not establish that fine-tuned LLMs actually rely on those features.” You are entirely correct. We do not claim that LLMs use the same features as our simple models. We identify multiple feature sets (manually selected, BERT embeddings, and fine-tuned models) that all perform well across a variety of query types. The difference in performance across query types is an indication the models are learning different patterns from each feature set, which supports our conclusion that there are many learnable patterns a model could exploit if it is exposed to examples of the data.

In section 3.1 and 3.2, our claims are that CLadder embeds simple patterns in data sets that it produces, and that these patterns are learnable by machine learning models. Our surprising and novel finding is that small numbers of extremely simple features are strongly predictive of correct answers. These patterns are embedded by the procedural generation process such that any model can exploit them, and they generalize to any data produced by the CLadder generator. The features we highlight demonstrate how simple and effective they can be. As we say in the concluding paragraph: “[T]hese results show that combinations of superficial features are sufficient to perform at state-of-the-art levels on CLadder test sets.”

You note that “Section 3.3 does not show direct evidence of data exposure.” Again, you are entirely correct. The primary point of the paper does not require definitive evidence of exposure. Our primary point (lines 89-97) is that insufficient evidence exists for the explicit and implicit claims of many published papers (namely, that high performance on CLadder implies formal causal reasoning in LLMs). Instead, we point out that an alternative inference is equally plausible: Exposure to data drawn from a generator that embeds patterns that are superficial, learnable, and highly predictive. We show that this latter inference is plausible, though not definitively proven. This calls into question the claims of dozens of papers.

You note that, in “Section 3.4, removing the introductory sentence or the causal graph sentence does not eliminate the semantic information in the remaining query.” This is incorrect. As we note in lines 329-330: “we remove key components of the queries that leave them unsolvable by formal causal reasoning, and we find that such ablations produce no substantial drop in LLM performance.” The experiments in this section are based on the formal theory of identifiability that CLadder is based on and that clearly identifies which information is critical to correctly and uniquely answering causal queries. We show in this section that removing information that is absolutely essential for identifiability (assumptions, graph structure, and probability values) either has little effect on LLM accuracy on CLadder (in the case of assumptions and graph structure) or results in accuracy that is still substantially above random (probability values).

You note that “Shortcut learning, spurious correlation, benchmark contamination, and sensitivity to input features are well-known issues in LLM evaluation, including reasoning becnhmarks and other NLP tasks.” You are entirely correct. However, this misinterprets a key point of our paper. As we note in the introduction, CLadder was explicitly designed to overcome many of the problems associated with static benchmarks. However, we demonstrate a novel failing of CLadder that calls into question the conclusions of dozens of papers: CLadder’s procedural generator embeds consistent patterns in each (supposedly unique) data set that it generates. As a result, exposure to a single CLadder data set enables high performance on any future data set generated by CLadder.

Thank you for the clarifications. However, my concern about the significance of the contribution remains. While the CLADDER-specific analysis is useful to some extent, I'm still not convinced by the contribution relative to prior work (such as CounterBench).

Some of my concerns about the supporting evidence also remain. For example, I acknowledge removing certain parts would leave the queries unsolvable by formal causal reasoning, but it does not eliminate potentially informative features.

I will maintain my scores.

Reviewer_xB2YOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper uses extensive experiments to show that LLMs’ good performance on CLADDER is not sufficient to claim the causal reasoning abilities. This paper finds out LLMs examined are likely exposed to CLADDER in training or is able to utilize the superficial non-pattern (non-causal shortcuts) in CLADDER to achieve a high score. Interestingly, this paper finds that outcome of ATE and ETT questions in CLADDER can be linearly separated given only two key probability values, and even a simple models like random forest can learn such patterns. Also, removing essential causal information such as background statement or causal graph verbal description does not significantly lower LLM’s performance. On the contrary, removing some non-causal token such as “hard” significantly affect LLM’s performance. Furthermore, LLM’s performance on CLADDER cannot generalize to CounterBench (another benchmark). Therefore, this paper claims that LLM’s performance on CLADDER cannot be used as evidence to prove LLM’s causal ability.

Critical strengths
  1. The experiment setting of this paper is very clear. For instance, in section 3.1, this paper clearly visualizes that outcome variables given two probability values can be easily separated. Using random forests to benchmark LLM is also a clear way to show that the outcome of questions are predictable given superficial patterns.

  2. The experiments are comprehensive, such that we gathers different pieces of evidence to arrive at the conclusion “good performance on CLADDER is not sufficient prove causal ability”.

Critical weaknesses
  1. This paper claims possible model training data exposure to CLADDER. However, there is no experiment driven evidence for this claim in this paper. Merely presents Figure 4 is not sufficient, and this paper admits that “without access to the training corpus, there is no way to verify whether the models trained after the dataset’s release were exposed during training”. I was wondering whether there can be some further ablation study to strengthen this hypothesis, such that letting the model predict masked token given masked questions from CLADDER (if LLMs can successfully predicts tokens, especially those incidental tokens, this may indicate training data exposure). Without such experiment, “models have been exposed to one or more CLADDER data sets” is merely a suspicion.

  2. In Section 3.5 “removing incidental token”, the framework of such experiment does not seem to be scalable/easily-generalizable, despite the intuitive result. Some token is crucial for causal information or grammar structure, while some token is “inconsequential to a human” (thus non-crucial). This experiment’s purpose is to find if removing non-crucial token affect the LLM’s performance, but it seems that the author(s) of this paper differentiate crucial vs non-crucial token by manually check in Figure 7. The result is valid, but there is no way to generalize this methodology on all different causal questions and other benchmarks. It would be better if author(s) of this paper propose some scalable methods, such as (1) how to systematically identify such non-crucial token given the sentence structure (2) whether we can use some automatic rephrasing methods to test if LLM performs worse in semantically identical but verbally different questions.

Questions for authors

See Weakness 1 and 2.

2 comments

We greatly appreciate the thoughtful comments and constructive suggestions in your review.

As you note, we argue that exposure to CLadder data is likely for many modern LLMs (Line 108), and you are correct that we do not present explicit experimental evidence of exposure. We are intrigued by the experiment that you suggest, and we are working to complete it before the end of the rebuttal period.

That said, we want to emphasize that the primary point of this paper does not require definitive evidence of exposure. Our primary point (lines 89-97) is that insufficient evidence exists for the explicit and implicit claims of many published papers (namely, that high performance on CLadder implies formal causal reasoning in LLMs). Instead, we point out that an alternative inference is equally plausible: Exposure to data drawn from a generator that embeds patterns that are superficial, learnable, and highly predictive. We show that this latter inference is plausible, though not definitively proven. This calls into question the claims of dozens of papers.

We agree that section 3.5 lacks an unambiguous standard for which tokens are “crucial”. Therefore, section 3.5 is now titled "Individual tokens greatly affect performance," and we have rewritten it to highlight the central issues of brittleness and sensitivity to changes in individual tokens. We no longer mention the relationship between the masked tokens and a human's ability to solve the problem. Instead, we advocate checking if the model is sensitive to arbitrary minor perturbations of the data (however that may apply in a particular context), as this could be an indication of shortcut learning.

In addition, we note that the experiments in the previous section (3.4) are based on the formal theory of identifiability that clearly identifies which information is critical to correctly and uniquely answering causal queries. We show in this section that removing information that is absolutely essential for identifiability (assumptions, graph structure, and probability values) either has little effect on LLM accuracy on CLadder (in the case of assumptions and graph structure) or results in accuracy that is still substantially above random (probability values). This is strong evidence that LLMs are not performing formal causal reasoning.

We attempted the experiment you suggested, but we did not obtain conclusive results. We have posted an updated version of the paper with changes that address the other comments in the review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.