Which Reasoning Steps Drive the Answer? A Causal Analysis of LLM Reasoning Steps

Large language models often generate multi-step reasoning traces to solve complex problems, yet many evaluations assess only the final answer. The accuracy of the final answer alone, however, does not reveal which intermediate steps causally influence the answer or how their effects propagate through reasoning. This paper introduces a causal framework for identifying reasoning steps with statistically significant effects on the final answer and for characterizing the pathways through which these effects arise. We use $e$-values to assess the statistical significance of each reasoning step's total effect, apply multiple testing procedures to select significant steps, and decompose their total effects into the interventional direct and indirect effects for visualization. We demonstrate the utility of our approach for evaluating LLMs’ reasoning capabilities across multiple reasoning models and datasets.