acceptodds
Under review as a conference paper at ICLR 2027

Which Reasoning Steps Drive the Answer? A Causal Analysis of LLM Reasoning Steps

Abstract

Large language models often generate multi-step reasoning traces to solve complex problems, yet many evaluations assess only the final answer. The accuracy of the final answer alone, however, does not reveal which intermediate steps causally influence the answer or how their effects propagate through reasoning. This paper introduces a causal framework for identifying reasoning steps with statistically significant effects on the final answer and for characterizing the pathways through which these effects arise. We use -values to assess the statistical significance of each reasoning step's total effect, apply multiple testing procedures to select significant steps, and decompose their total effects into the interventional direct and indirect effects for visualization. We demonstrate the utility of our approach for evaluating LLMs’ reasoning capabilities across multiple reasoning models and datasets.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, reasoning traces, causal inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_V7XZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes a framework for measuring how individual reasoning steps affect answer correctness. It combines step interventions and repeated continuations with e-value-based multiple testing, then decomposes selected steps’ effects into direct effects and effects through subsequent reasoning. Experiments on three models and four benchmarks report higher proportions of significant steps and larger effects for arithmetic and GSM8K, with case studies illustrating the decomposition.

Critical strengths
  1. The paper studies a meaningful question. Understanding which reasoning steps affect the final answer and how their effects pass through later steps is useful for evaluating how models use their generated reasoning.

  2. The related work is thorough and well organized. It covers different approaches to analyzing reasoning traces and explains how the proposed framework differs from them. This makes it easier to understand where the contribution fits.

  3. The use of e-values and e-BH gives step selection an explicit false-discovery guarantee under the stated sampling assumptions. Accounting for random variation in continuations and multiple testing is a useful part of the framework.

Critical weaknesses
  1. The method is incremental. The paper uses multiple testing to select statistically significant reasoning steps, then decomposes their total effects into direct and indirect effects. However, the question of “which reasoning steps matter and how their effects propagate to the final answer” has already been studied in closely related work, including Thought Anchors and Thought Branches. FRODO also studies direct and indirect effects through causal mediation. Although the granularity differs, the added value over these works is not clear to me.

  2. More importantly, the empirical validation is weak. The case studies and the check that TE is approximately IDE + IIE mainly illustrate the method and check numerical consistency. They do not establish that the decomposition provides trustworthy explanations of reasoning. For example, there is no controlled test with known causal paths, or comparison showing that the method gives more reliable attributions than existing approaches.

  3. I find the insights limited. The main observations are that mathematical tasks have more steps with significant effects, and earlier steps often affect the answer indirectly while the last step acts directly. Both observations are quite intuitive, and I do not learn much beyond them.

  4. The experiments use only two Qwen models and one Mistral model. Whether the findings hold across a broader range of frontier reasoning models remains unclear.

  5. There is also an obvious presentation issue: Figure 1 still contains the placeholder “[MODEL]” instead of the model name.

Questions for authors

See the content of weaknesses.

3 comments

We thank the reviewer for the comments on novelty, validation, and scope.

W1: No prior method combines TE, FDR-controlled selection, and IDE/IIE for the same step. We agree that e-values, e-BH, and mediation analysis each have precedents. Our contribution is one procedure that, for each step, estimates the total effect, selects the step with a finite-sample FDR guarantee (Eqs. 3–5, 7–8), and splits its effect into a direct part and a part through later reasoning (Eqs. 9–10), with no training and no access to model internals. Thought Anchors, Thought Branches, and FRODO each cover only part of this (Table 1; Appendix D.6, Table 10 in the edited PDF). In particular, none of them separates the direct and indirect effects of a single step, which is what reveals the opposing pathways in W3.

W2: Each term is measured by direct intervention, and the decomposition passes checks with known answers. i) Assumptions: IDE and IIE are interventional effects (Vansteelandt & Daniel, 2017), so no cross-world assumption is needed, and since we set the step and the reasoning suffix ourselves, no ignorability assumption is needed either. IDE + IIE equals TE if the answer after a given suffix has the same mean whether the suffix was generated freely or fixed by us (Appendix A.4, Eq. 16). This is checkable: on MMLU-Pro, where TE and IDE + IIE come from separate generations, the two agree exactly for 66% of pairs and within 0.05 for 93% (Appendix C.1, Fig. 4). ii) Known answers: final steps have little reasoning after them, so their IIE should be near zero, and the estimates are 0.01–0.10 (Appendix D.4). The zero-effect controls and the known-effect simulation in our response to Reviewer pgVk (Q3) further show that the selection feeding the decomposition is calibrated and detects moderate effects. Our claims concern pathways through generated text, not internal computation.

W3: The aggregate analyses add three findings that are not obvious a priori (details in our response to Reviewer pgVk). i) Edit size: Math effects are near their maximum even for the smallest edits, whereas Non-Math effects grow with edit size, consistent with Math traces carrying each intermediate quantity forward (Appendix D.1, Fig. 17). ii) Position and function: non-final steps act mostly through later reasoning (85–100% of pairs are dominated by the indirect effect). The definition does not force this, since IIE can be near zero even when later reasoning exists. The channel also depends on function: planning steps act mainly through later reasoning, verification steps mainly directly (Appendix D.4, Table 9). iii) Opposing pathways: TE alone can hide a step that pushes the answer one way directly and the other way through later reasoning; 0.5–8.2% of pairs show this with both intervals excluding zero (Table 9).

W4: We added GLM-4.5-Air and Llama-3.1-8B-Instruct; the cross-task pattern holds for all five models. The evaluation now covers five models from four families, including a large MoE model (Appendix D.5, Fig. 24). On the shared subsets, every model has a much higher discovery rate on GSM8K (88–92%) than on MMLU-Pro (24–50%), so the gap is not specific to one family or size. The method only needs continuation from a given prefix and applies to any model that allows it; closed models that hide their reasoning are out of scope.

W5: Fixed. "[MODEL]" now reads "Qwen3.8-27B". Thanks for catching the bug.

Thanks for adding the analysis. I’m raising Soundness to 2. But I still don’t see a test with known direct and indirect effects to check whether the method gets the split right.

Thank you for suggesting this check. We tested the decomposition in a semi-synthetic experiment with known direct and indirect effects. In each of the six scenarios, the mean estimated IDE and IIE differed from their respective true means by less than 0.0005.

We used 766 step–intervention pairs from Qwen3.8-27B on MMLU-Pro and Basic Arithmetic. For each pair, we retained 64 downstream continuations generated after the original step and 64 generated after the edited step. We scored every continuation SS by the model’s answer accuracy given the edited step followed by SS, regardless of which step had generated the continuation. We denote this score by m(S)m(S). We simulated only final-answer correctness, with probability

Pr⁡(correct∣x,S)=0.5+d,x+i(m(S)−mˉ0)+g,x(m(S)−mˉ0),\Pr(\mathrm{correct}\mid x,S) = 0.5 + d\\,x + i\bigl(m(S)-\bar m_0\bigr) + g\\,x\bigl(m(S)-\bar m_0\bigr),

where x=0x=0 denotes the original step, x=1x=1 the edited step, and mˉ0\bar m_0 is the mean score of continuations generated after the original step. This probability rule and the two empirical continuation distributions determine the true IDE and IIE for each pair.

We varied dd, ii, and gg across six scenarios. The first three used all 766 pairs. The last three used the 577 pairs for which the two continuation pools had different mean scores. In each of 50 replications per scenario, we resampled 64 continuations per arm, drew L=4L=4 answers per continuation, and applied the paper’s estimator and confidence interval procedure.

The table reports true effects averaged over pairs. Mean error is the estimate minus its pair-specific true effect, averaged over pairs and replications. Coverage is the fraction of pair-specific intervals containing the corresponding true effect.

ScenarioPairsTrue mean IDEMean IDE errorTrue mean IIEMean IIE error95% CI coverage (IDE / IIE)
No effect7660.000−0.00010.000+0.000194.8% / 95.0%
Direct only766+0.300+0.00010.000−0.000194.1% / 95.2%
Indirect only7660.000−0.0001−0.143−0.000295.0% / 94.8%
Same sign577−0.1290.0000−0.142+0.000194.2% / 94.4%
Opposite signs577+0.129−0.0004−0.142+0.000194.1% / 94.5%
Zero total effect577+0.095−0.0001−0.095−0.000194.2% / 94.7%

In the cancellation scenario, the true total effect was exactly zero for every included pair. The decomposition recovered the opposing effects on average: the true mean IDE was +0.095, the true mean IIE was −0.095, and the mean estimation error was −0.0001 for each. Across the six scenarios, empirical coverage of the nominal 95% intervals ranged from 94.1% to 95.2%. We will add this experiment to Appendix D.4.

Reviewer_iMq8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies which individual steps in an LLM reasoning trace affect the final answer, and whether their effects arise directly or are transmitted through subsequent reasoning. The authors intervene on each reasoning step by generating a locally modified version of it. They then sample continuations from the original and modified prefixes and estimate the change in answer accuracy as the total effect of the intervention. E-values and the e-BH procedure are used to identify steps with statistically significant effects while controlling the false discovery rate. For the selected steps, the total effect is further decomposed into an interventional direct effect and an indirect effect mediated by the subsequent reasoning. The experiments cover three models and four benchmarks. The main empirical finding is that mathematical tasks contain a higher proportion of significant reasoning steps. The examples also suggest that early calculation steps tend to act through later reasoning, whereas the final calculation or answer-selection step tends to affect the answer directly.

Critical strengths

1.The paper provides a useful step-level view of how a reasoning trace affects the final answer. In particular, separating direct effects from effects transmitted through subsequent reasoning is more informative than assigning each step a single importance score. The examples illustrate this distinction clearly. 2.The statistical treatment of step selection is well motivated. Since multiple steps are tested within each trace, using e-values and e-BH provides a principled way to control false discoveries. The method is also black-box and does not require access to model internals.

Critical weaknesses

1.The results depend strongly on the automatically generated interventions, but the intervention pipeline is not quantitatively validated. The paper mentions manual spot checks but does not report their scale or error rate, nor does it test sensitivity to different valid interventions for the same step. It is therefore unclear whether the selected steps reflect stable properties of the reasoning trace or sensitivity to particular generated modifications. 2.The cross-benchmark comparison may be confounded by baseline accuracy, answer stability, and intervention strength. For example, mathematical steps may simply admit clearer and more damaging local interventions than knowledge or planning steps. In addition, the values of N, B_0, B_1, L, and alpha are not clearly reported, even though they directly affect estimation precision and the power of the selection procedure.

Questions for authors

1.What values of N, B_0, B_1, L, and alpha were used in the main experiments? 2.According to human evaluation, what proportion of the automatically generated interventions satisfy the intended intervention rules? 3.If multiple valid interventions are generated for the same step, how stable are step selection and the estimated effects?

1 comment

We thank the reviewer for the close attention to intervention quality and reproducibility, both of which are now quantified in Appendix D.

W1 & Q2: A blinded human audit finds 97.6% of interventions compliant. We audited a stratified sample of 373 interventions from the main experiments. An intervention passes only if it changes just the target content, keeps the step's function, follows the function-specific rule, and introduces no other issue. Annotators were blind to effects and selection; a second annotator double-coded 90 items (90% agreement), and disagreements were adjudicated. Weighting each function by how often it occurs, compliance is 97.6% (95% CI 94.1–99.6%; unweighted 95.4%; Appendix D.2, Fig. 18). Rerunning e-BH without the non-compliant interventions changes total discoveries from 5561 to 5553 and adds none. Flawed interventions are rare and barely affect which steps are selected.

W1 & Q3: Across new valid interventions, selection is stable for calculation steps and slightly noisier for knowledge steps. For each selected step, we generated new rewrites with the same pipeline, swapped in only that step's intervention, and reran e-BH, with resampling of the original rewrite as the baseline (Appendix D.2, Fig. 19). i) Selection: one new rewrite reselects 96–100% of original discoveries on GSM8K, matching the baseline, but 73–81% on MMLU-Pro (baseline 90–94%). Pooling several new rewrites through the step-level e-value (Eq. 5), without the original, raises MMLU-Pro to 83–91%. ii) Effect size: a new rewrite changes TE by a median of 0 on GSM8K but 0.13–0.31 on MMLU-Pro, against at most 0.02 from resampling, with the same sign in 76–100% of cases.

W2: The Math vs. Non-Math gap persists after controlling for all three factors. (Math: Basic Arithmetic and GSM8K; Non-Math: Knights & Knaves and MMLU-Pro.) The analysis is shared with our response to Reviewer pgVk (W1 & Q1); under every control in Table 7, the gap stays at 17–39 pp with all intervals above zero. i) Baseline accuracy: standardizing for original-answer correctness shrinks the gap the most, so accuracy explains part of it, but not all. ii) Answer stability: keeping only problems where two continuations from the original prefix give the same answer with probability at least 0.95 changes the gap by at most about 7 pp, so stability does not explain it. iii) Intervention strength: the gap remains among calculation steps alone, and even the smallest Math edits move the answer more than the largest Non-Math edits (Appendix D.1, Fig. 17), so it does not come from Math admitting larger edits.

Q1 & W2: N = 64, B_0 = B_1 = 64, L = 4, alpha = 0.05. Quick reminder, N is the number of free continuations per arm, B_0 and B_1 are the numbers of reasoning suffixes per arm used for the decomposition, L is the number of answer regenerations per fixed suffix, and alpha is the target FDR of e-BH within each problem. All supplementary experiments use these values (the budget analysis also uses N = 16, 32, and 48). They are listed at the start of Appendix D, and we will add them to Section 4.

Reviewer_pgVkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method for the statistical causal analysis of reasoning steps in language models. Given a fixed prompt (question and prefix), the authors intervene by modifying a specific reasoning step and regenerating the subsequent reasoning and final answer; the total effect (TE) is then defined by the resulting change in the expected task score. The authors construct and aggregate e-values, employing the e-Benjamini–Hochberg (e-BH) procedure to control the false discovery rate (FDR) for the selected steps. For the steps identified, they further estimate the Interventional Direct Effect (IDE) and the Interventional Indirect Effect (IIE) by fixing or replacing the distribution of the subsequent reasoning text. The experiments cover three models across four benchmarks: Basic Arithmetic, Knights & Knaves, GSM8K, and MMLU-Pro. The authors report a higher discovery rate of significant steps and larger effects on task scores for mathematical tasks, and they provide case studies illustrating the direct and indirect effects of various reasoning steps.

Critical strengths
  1. Integrating step-level interventions with uncertainty quantification is a clear and valuable methodological contribution. Simply observing whether an answer changes following a single modification risks conflating sampling noise with true causal effects. By fixing the intervention and generating independent continuations from two different prefixes—then selecting steps via e-values and multiple-testing correction—the authors ensure that "significant steps" correspond to a well-defined statistical hypothesis. The design, which involves averaging valid e-values per step and applying the e-BH procedure, is well-founded under the stated conditions and does not require the steps to be independent of one another.

  2. Distinguishing between direct and indirect effects offers significant interpretative value. A single sensitivity score cannot clarify whether a specific step influences the final answer directly or by altering the subsequent reasoning process; this paper provides a framework to articulate such distinctions. For instance, as shown in Figure 1, the impact of an early calculation step is primarily mediated through subsequent reasoning, whereas the mapping to a final answer choice exerts a direct influence on the outcome. This represents a promising tool for analyzing "black-box" models.

Critical weaknesses
  1. The observed discovery rates across tasks are not yet sufficient to support the broad conclusion that "mathematical reasoning steps possess a stronger causal role." The outcome function used in the experiments is primarily based on correctness or task scores. This measures the sensitivity of the task score to an intervention, but it does not capture the full scope of changes in the distribution of answers. For example, if an intervention changes an incorrect answer "B" to another incorrect answer "C," the task score remains zero in both cases—yet the final answer is clearly dependent on that specific reasoning step. Baseline accuracy, intervention intensity, and statistical power may vary across different tasks. In Table 3, Qwen3.8-27B shows a discovery rate of 89.3% on GSM8K and 41.0% on MMLU-Pro; however, the table lacks baseline accuracy figures, comparisons stratified by the correctness of the original answer, and a summary that accounts for the comparability of intervention categories and intensities. Consequently, while these results demonstrate varying sensitivity to the given protocol, they do not rule out other factors that could explain the observed differences between tasks.

  2. The core contribution—mediation decomposition—is primarily supported by individual case studies, lacking broader empirical conclusions. The main tables systematically summarize total effects and step-level discovery rates, while IDE (Interventional Direct Effect) and IIE (Interventional Indirect Effect) estimates are presented only through a few examples in the main text and the appendix. Although the "bridge-gap" check supports the agreement between the effect estimates, it cannot substitute for a comprehensive analysis of which specific functions or positions influence the final answer through which particular pathways.

Questions for authors
  1. Is it possible to use existing rollouts to provide baseline accuracy for each model and task, stratified by the correctness of the original answer and the category of intervention? If one examines changes in the specific answer provided—rather than just the correctness score—does the observed ranking (e.g., that mathematical tasks rely more heavily on reasoning steps) still hold?

  2. Is it possible to aggregate the distributions and confidence intervals of IDE/IIE across tasks, models, step functions, and positions—including instances where effects might be in opposite directions or cancel each other out? Do these aggregate results support the patterns suggested by the individual case studies?

  3. Is it possible to demonstrate selection stability and statistical power through experiments—such as sensitivity analysis within a limited budget, or controlled tests involving known zero/non-zero causal effects? This would help illustrate the practical advantages of the proposed method, without requiring the re-validation of existing theorems or the addition of large-scale benchmarks.

3 comments

We thank the reviewer for the precise questions; they shaped the new analyses in Appendix D.

W1 & Q1: The Math > Non-Math gap survives every check the reviewer suggested. (Math: Basic Arithmetic and GSM8K; Non-Math: Knights & Knaves and MMLU-Pro.) i) Outcome: within MMLU-Pro, we replaced correctness with the total-variation distance between answer distributions, which also counts moves between two wrong options. Math questions still show larger answer shifts and 20–30 pp higher discovery rates for all three models (Appendix D.1, Fig. 15, see in the edited PDF), so the ranking holds inside one benchmark and does not depend on scoring by correctness. ii) Baseline accuracy: Knights & Knaves is answered about as accurately as the Math tasks yet has much lower discovery rates; Qwen3.5-9B, for example, scores 99% on it vs. 92% on GSM8K with equally stable answers, but its discovery rate is 47% vs. 90% (Table 6). Standardizing for original-answer correctness shrinks the gap more than any other control, yet it stays at 17–32 pp (Table 7). Accuracy explains part of the gap, not all of it. iii) Intervention intensity: the gap remains when only calculation steps are compared (Table 7), and even the smallest Math edits move the answer more than the largest Non-Math edits (mean |TE| 0.73–0.84 vs. 0.27–0.36; Fig. 17). iv) Power: the gap also shows up in effect sizes (Figs. 15 and 17), not just in discovery rates, so it is not a matter of detection power alone; Q3 reports the power of the test. Section 4 will present the gap as an empirical finding that holds under these controls.

W2 & Q2: Decomposing every selected step confirms the case-study pattern at scale. We decomposed all selected steps in three model–task settings, giving 996 step–intervention pairs from 409 problems (Appendix D.4). i) Position and function: for early, middle, and late steps, 85–100% of pairs are dominated by the indirect effect, whereas the final step, with little reasoning after it, acts mainly directly (Table 9, Fig. 23). By function, planning and organizing steps act mainly through later reasoning, and verification steps mainly directly. A non-final step therefore changes the answer mostly by changing the reasoning that follows it. ii) Opposing pathways: in 0.5–8.2% of pairs, IDE and IIE have opposite signs with both intervals excluding zero. Among pairs with opposite-sign estimates, the median share of |IDE| + |IIE| that cancels in the sum is 0.09–0.19 (Eq. 17). Such opposing pathways are uncommon and usually offset only a small part of the dominant one, but TE alone cannot reveal them. We will replace the case-based presentation in Section 4 with these aggregate results.

Q3: Selection is conservative under the null, detects effects of about 0.3 or more, and is reproducible. i) Calibration: in an A/A control, where both arms are drawn from one pool so the true effect is zero, at most 0.10% of problems have any false discovery across all 12 model–task settings, against a nominal 5% (Appendix D.3, Fig. 20a). An inserted empty step is almost never detected, and detection rises as the inserted text becomes more consequential (Table 8). Selection stays quiet when nothing changes and becomes more sensitive as the change matters more. ii) Power: in a simulation with a known effect δ at our budget of 64 continuations per arm, the test detects δ ≥ 0.3 in at least 70% of runs but often misses δ ≤ 0.2 (Fig. 21). Selected steps thus have moderate to large effects, and our discovery rates are conservative. iii) Reproducibility: independently resampling all continuations for 80 problems reselects 170 of 172 discoveries, and cutting the budget to 48 continuations per arm costs some power but adds almost no new selections (Fig. 22).

Thank you for supplementing the analysis in Appendix D. However, whether the interventions alter the reasoning to a similar extent across tasks remains unresolved. Figure 17 compares the rewriting magnitudes based on their normalized character-level edit distance. But changing a single digit or operator only requires modifying one character, which may alter the premise of subsequent calculations. Even when limited to 'local computation or inference' steps, the editing with similar character-level distances does not necessarily indicate a similar semantic change.

We agree that a one-character edit can change the premise of later calculations. Figure 17 measures textual change and leaves open whether the edits were comparable in semantic strength across tasks.

We asked Qwen3.8-27B to rate how much each of the 8,836 interventions in Figure 17 changed the meaning of the edited step. The rater saw only the original and edited step texts. We asked it to assess the change in the step itself, while TE measures the effect of the edit on the final answer. The scale ran from 1 (only the wording changed) to 5 (the statement was negated, reversed, or replaced with unrelated content). Three responses remained unparseable after one retry, leaving 8,833 interventions for the rating-based analyses. Spearman correlations between the ratings and character-level edit distance ranged from 0.17 to 0.24 for Math and from 0.39 to 0.42 for Non-Math.

For each model, the mean absolute total effect (|TE|) was higher for Math than for Non-Math within each of the three rating bins. The table reports the Math − Non-Math difference (Δ) with 95% bootstrap intervals clustered by question.

ModelRatingn (Math / Non-Math)Mean absolute TE (Math)Mean absolute TE (Non-Math)Δ [95% CI]
Qwen3.8-27B1–2106 / 2740.650.17+0.48 [0.37, 0.57]
Qwen3.8-27B315 / 600.730.21+0.52 [0.29, 0.74]
Qwen3.8-27B4–51,276 / 1,2470.830.32+0.50 [0.46, 0.54]
Qwen3.5-9B1–2102 / 1820.610.27+0.34 [0.22, 0.45]
Qwen3.5-9B312 / 530.640.15+0.48 [0.20, 0.73]
Qwen3.5-9B4–51,185 / 9480.840.31+0.53 [0.49, 0.56]
Mistral-Small-3.2-24B1–2147 / 3290.550.14+0.41 [0.33, 0.50]
Mistral-Small-3.2-24B314 / 790.730.14+0.59 [0.35, 0.81]
Mistral-Small-3.2-24B4–51,349 / 1,4550.760.27+0.49 [0.45, 0.53]

For each of the three models, Math edits rated as small changes (scores 1–2) had a higher mean |TE| than Non-Math edits rated as large changes (scores 4–5). The corresponding means ranged from 0.55 to 0.65 for Math and from 0.27 to 0.32 for Non-Math.

Among Local Computation or Inference steps, the low (1–2) and high (4–5) bins together accounted for over 98% of the rated interventions in that subset. In those two bins, the Math − Non-Math differences ranged from 0.31 to 0.52 across models, with all 95% intervals above zero.

We will add this analysis to Appendix D.1 and acknowledge that edits assigned the same rating may still differ across tasks in how much they change a step’s meaning. In Section 4 and the Discussion, we will narrow the claim to the observed result. Under our function-specific intervention protocol, single-step edits had a larger mean absolute effect on final answers for the Math tasks than for the Non-Math tasks we studied.

Comments on the paper

1 comment

Thank you for reading Appendix D closely. We used up to four distinct new rewrites per step function. The total sampling budget matched the baseline when we used one new rewrite and exceeded it whenever we pooled two or more. On MMLU-Pro, the number of new rewrites ranged from 1 to 4 for both models, averaging 2.6 for Qwen3.8-27B and 2.3 for Mistral-Small-3.2-24B. On GSM8K, the mean was 1.9.

We used the same generator, prompt, and acceptance checks as for the original rewrite. The pooled test included only the new rewrites and averaged their e-values with equal weights using Eq. (5). The resampling baseline and the single-rewrite test each used 64 original-arm and 64 intervened-arm continuations, for a total of 128. With kk new rewrites, the pooled test shared the same 64 original-arm continuations across rewrites and used 64 intervened-arm continuations per rewrite, for a total of 64(1+k)64(1+k) per step function.

Model (MMLU-Pro)New rewrites per function, mean (range)Pooled continuations per function, mean (original + intervened)Reselection, resampling baselineReselection, one new rewriteReselection, pooled rewrites
Qwen3.8-27B2.6 (1–4)232 (64 + 168)94.1%81.3%91.1%
Mistral-Small-3.2-24B2.3 (1–4)210 (64 + 146)90.0%72.7%83.0%

On MMLU-Pro, pooling increased the fraction of original discoveries reselected by about 10 percentage points for each model relative to a single new rewrite. Because pooling increased both the number of distinct rewrites and the number of continuations, this comparison cannot separate their contributions to the gain. We will report the sampling budgets alongside the reselection rates in Appendix D.2.

We agree that the character-level edit distance in Figure 17 measures textual change and that a one-character edit can substantially alter a step's meaning. In an additional analysis, we grouped edits by an LLM's rating of how much each edit changed the meaning of the step. Across the three models and three rating bins, the Math − Non-Math difference in mean |TE| ranged from 0.34 to 0.59, with all 95% bootstrap intervals above zero. For each model, Math edits rated as small changes (scores 1–2) had a higher mean |TE| than Non-Math edits rated as large changes (scores 4–5). The corresponding means ranged from 0.55 to 0.65 for Math and from 0.27 to 0.32 for Non-Math.

We will add these results to Appendix D.1 and narrow the claim in Section 4 and the Discussion to the observed difference in sensitivity under our function-specific intervention protocol. We will explicitly retain differences in semantic intervention strength as a limitation of the cross-task comparison, since edits in the same rating bin may still differ across tasks in how much they change a step's meaning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.