- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper proposes a framework for measuring how individual reasoning steps affect answer correctness. It combines step interventions and repeated continuations with e-value-based multiple testing, then decomposes selected steps’ effects into direct effects and effects through subsequent reasoning. Experiments on three models and four benchmarks report higher proportions of significant steps and larger effects for arithmetic and GSM8K, with case studies illustrating the decomposition.
Critical strengths
-
The paper studies a meaningful question. Understanding which reasoning steps affect the final answer and how their effects pass through later steps is useful for evaluating how models use their generated reasoning.
-
The related work is thorough and well organized. It covers different approaches to analyzing reasoning traces and explains how the proposed framework differs from them. This makes it easier to understand where the contribution fits.
-
The use of e-values and e-BH gives step selection an explicit false-discovery guarantee under the stated sampling assumptions. Accounting for random variation in continuations and multiple testing is a useful part of the framework.
Critical weaknesses
-
The method is incremental. The paper uses multiple testing to select statistically significant reasoning steps, then decomposes their total effects into direct and indirect effects. However, the question of “which reasoning steps matter and how their effects propagate to the final answer” has already been studied in closely related work, including Thought Anchors and Thought Branches. FRODO also studies direct and indirect effects through causal mediation. Although the granularity differs, the added value over these works is not clear to me.
-
More importantly, the empirical validation is weak. The case studies and the check that TE is approximately IDE + IIE mainly illustrate the method and check numerical consistency. They do not establish that the decomposition provides trustworthy explanations of reasoning. For example, there is no controlled test with known causal paths, or comparison showing that the method gives more reliable attributions than existing approaches.
-
I find the insights limited. The main observations are that mathematical tasks have more steps with significant effects, and earlier steps often affect the answer indirectly while the last step acts directly. Both observations are quite intuitive, and I do not learn much beyond them.
-
The experiments use only two Qwen models and one Mistral model. Whether the findings hold across a broader range of frontier reasoning models remains unclear.
-
There is also an obvious presentation issue: Figure 1 still contains the placeholder “[MODEL]” instead of the model name.
Questions for authors
See the content of weaknesses.
3 comments
Thanks for adding the analysis. I’m raising Soundness to 2. But I still don’t see a test with known direct and indirect effects to check whether the method gets the split right.
Thank you for suggesting this check. We tested the decomposition in a semi-synthetic experiment with known direct and indirect effects. In each of the six scenarios, the mean estimated IDE and IIE differed from their respective true means by less than 0.0005.
We used 766 step–intervention pairs from Qwen3.8-27B on MMLU-Pro and Basic Arithmetic. For each pair, we retained 64 downstream continuations generated after the original step and 64 generated after the edited step. We scored every continuation by the model’s answer accuracy given the edited step followed by , regardless of which step had generated the continuation. We denote this score by . We simulated only final-answer correctness, with probability
where denotes the original step, the edited step, and is the mean score of continuations generated after the original step. This probability rule and the two empirical continuation distributions determine the true IDE and IIE for each pair.
We varied , , and across six scenarios. The first three used all 766 pairs. The last three used the 577 pairs for which the two continuation pools had different mean scores. In each of 50 replications per scenario, we resampled 64 continuations per arm, drew answers per continuation, and applied the paper’s estimator and confidence interval procedure.
The table reports true effects averaged over pairs. Mean error is the estimate minus its pair-specific true effect, averaged over pairs and replications. Coverage is the fraction of pair-specific intervals containing the corresponding true effect.
| Scenario | Pairs | True mean IDE | Mean IDE error | True mean IIE | Mean IIE error | 95% CI coverage (IDE / IIE) |
|---|---|---|---|---|---|---|
| No effect | 766 | 0.000 | −0.0001 | 0.000 | +0.0001 | 94.8% / 95.0% |
| Direct only | 766 | +0.300 | +0.0001 | 0.000 | −0.0001 | 94.1% / 95.2% |
| Indirect only | 766 | 0.000 | −0.0001 | −0.143 | −0.0002 | 95.0% / 94.8% |
| Same sign | 577 | −0.129 | 0.0000 | −0.142 | +0.0001 | 94.2% / 94.4% |
| Opposite signs | 577 | +0.129 | −0.0004 | −0.142 | +0.0001 | 94.1% / 94.5% |
| Zero total effect | 577 | +0.095 | −0.0001 | −0.095 | −0.0001 | 94.2% / 94.7% |
In the cancellation scenario, the true total effect was exactly zero for every included pair. The decomposition recovered the opposing effects on average: the true mean IDE was +0.095, the true mean IIE was −0.095, and the mean estimation error was −0.0001 for each. Across the six scenarios, empirical coverage of the nominal 95% intervals ranged from 94.1% to 95.2%. We will add this experiment to Appendix D.4.
We thank the reviewer for the comments on novelty, validation, and scope.
W1: No prior method combines TE, FDR-controlled selection, and IDE/IIE for the same step. We agree that e-values, e-BH, and mediation analysis each have precedents. Our contribution is one procedure that, for each step, estimates the total effect, selects the step with a finite-sample FDR guarantee (Eqs. 3–5, 7–8), and splits its effect into a direct part and a part through later reasoning (Eqs. 9–10), with no training and no access to model internals. Thought Anchors, Thought Branches, and FRODO each cover only part of this (Table 1; Appendix D.6, Table 10 in the edited PDF). In particular, none of them separates the direct and indirect effects of a single step, which is what reveals the opposing pathways in W3.
W2: Each term is measured by direct intervention, and the decomposition passes checks with known answers. i) Assumptions: IDE and IIE are interventional effects (Vansteelandt & Daniel, 2017), so no cross-world assumption is needed, and since we set the step and the reasoning suffix ourselves, no ignorability assumption is needed either. IDE + IIE equals TE if the answer after a given suffix has the same mean whether the suffix was generated freely or fixed by us (Appendix A.4, Eq. 16). This is checkable: on MMLU-Pro, where TE and IDE + IIE come from separate generations, the two agree exactly for 66% of pairs and within 0.05 for 93% (Appendix C.1, Fig. 4). ii) Known answers: final steps have little reasoning after them, so their IIE should be near zero, and the estimates are 0.01–0.10 (Appendix D.4). The zero-effect controls and the known-effect simulation in our response to Reviewer pgVk (Q3) further show that the selection feeding the decomposition is calibrated and detects moderate effects. Our claims concern pathways through generated text, not internal computation.
W3: The aggregate analyses add three findings that are not obvious a priori (details in our response to Reviewer pgVk). i) Edit size: Math effects are near their maximum even for the smallest edits, whereas Non-Math effects grow with edit size, consistent with Math traces carrying each intermediate quantity forward (Appendix D.1, Fig. 17). ii) Position and function: non-final steps act mostly through later reasoning (85–100% of pairs are dominated by the indirect effect). The definition does not force this, since IIE can be near zero even when later reasoning exists. The channel also depends on function: planning steps act mainly through later reasoning, verification steps mainly directly (Appendix D.4, Table 9). iii) Opposing pathways: TE alone can hide a step that pushes the answer one way directly and the other way through later reasoning; 0.5–8.2% of pairs show this with both intervals excluding zero (Table 9).
W4: We added GLM-4.5-Air and Llama-3.1-8B-Instruct; the cross-task pattern holds for all five models. The evaluation now covers five models from four families, including a large MoE model (Appendix D.5, Fig. 24). On the shared subsets, every model has a much higher discovery rate on GSM8K (88–92%) than on MMLU-Pro (24–50%), so the gap is not specific to one family or size. The method only needs continuation from a given prefix and applies to any model that allows it; closed models that hide their reasoning are out of scope.
W5: Fixed. "[MODEL]" now reads "Qwen3.8-27B". Thanks for catching the bug.