- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how large reasoning models solve planning problems using a modified Tower of Hanoi task. The authors identify six reasoning strategies and further decompose means-ends analysis into four reasoning primitives: subgoal proposal, action proposal, verification, and backtracking. Through controlled interventions on reasoning traces, they demonstrate the causal influence of these primitives on solution accuracy and optimality. Additionally, causal mediation analysis identifies MoE experts associated with verification behavior, which can be manipulated through activation steering.
Critical strengths
S1. Interesting research question. The paper addresses an important question concerning the causal relevance of chain-of-thought reasoning, providing insights into how reasoning models solve structured planning problems.
S2. Systematic and well-designed analysis. The investigation progresses from high-level reasoning strategies to functional primitives and internal model mechanisms. This multi-level approach provides a coherent framework for understanding reasoning behaviors.
S3. Convincing causal interventions. The repair and corruption experiments provide meaningful evidence that specific reasoning primitives influence downstream performance. The additional activation-steering experiments further strengthen the mechanistic investigation.
Critical weaknesses
W1. Limited generalizability. The mechanistic analysis focuses exclusively on two models from the gpt-oss family and a relatively simple Tower of Hanoi task with three elements. It remains unclear whether the identified reasoning primitives and their causal effects generalize to more complex planning tasks or other reasoning model families.
W2. Limited validation of internal mechanisms. Although activation steering successfully induces verification-related behavior, the experiments primarily measure the occurrence of verification rather than its correctness or contribution to downstream performance. Therefore, the evidence connecting the identified MoE experts to functional verification capabilities remains incomplete.
W3. Potential confounding in trace interventions. The repair and corruption experiments involve selecting particular reasoning trajectories and iteratively modifying their content. While paraphrase controls are included, it remains unclear to what extent the observed improvements reflect the targeted reasoning primitives rather than broader changes in subsequent reasoning trajectories.
Questions for authors
Q1. Do the identified reasoning strategies and primitives generalize to other planning tasks or reasoning model families beyond gpt-oss?
Q2. Does activation steering improve the correctness of verification or downstream task performance, rather than merely increasing the frequency of verification-related expressions?
Q3. Can the authors further isolate the causal effects of individual reasoning primitives from the additional reasoning changes introduced by iterative trace interventions?
4 comments
Response to W1/Q1
Response to W1/Q1
We thank the reviewer for this comment. As suggested, we have extended our analyses to a large reasoning model from a different model family, Qwen3-235B, and to a more challenging planning task, Mystery Blocksworld (Section 4 in the revision). The new results show that MEA and its four reasoning primitives generalize to both settings: MEA, the main focus of our analyses, remains the predominant strategy, and the models employ the same four primitives in MEA episodes at broadly similar frequencies (except that Qwen3-235B rarely proposes subgoals). In contrast, exhaustive search, which the gpt-oss models sometimes fall back on in 3-disk ToH, is rarely used by Qwen3-235B or in Mystery Blocksworld, suggesting that MEA is the more general strategy across models and tasks.
- Experiments on More Challenging Planning Tasks.
We have extended our analyses to another more complex planning task, Mystery Blocksworld, from PlanBench [1]. Mystery Blocksworld from PlanBench [1] is an adaptation of the classic Blocksworld planning domain [2]. As in ToH, the goal is to transform an initial configuration of blocks into a goal configuration by moving blocks. However, Blocksworld involves four different actions, each with its own preconditions, and its problems involve 4 or 5 blocks, unlike the 3-disk ToH problems we mainly studied. Furthermore, the Mystery version replaces the names of the actions and of the block relations with unrelated words (e.g., "succumb X" means putting down block X). Please refer to Figures 24 and 25 in the revision for demonstrations of the task.
We evaluated the gpt-oss models on two versions of Mystery Blocksworld: plan generation (Task 1) and cost-optimal planning (Task 2), which also asks the model to reach the goal using the fewest actions. As expected, performance dropped substantially relative to 3-disk ToH, especially for gpt-oss-20b (Figure 14b in the revision).
| Task | Model | Samples | Accuracy | Optimal | Thinking length (characters) |
|---|---|---|---|---|---|
| Plan generation (Task 1) | gpt-oss-20b | 100 | 0.14 (14/100) | 0.12 (12/100) | 106.0k |
| Plan generation (Task 1) | gpt-oss-120b | 100 | 0.78 (78/100) | 0.49 (49/100) | 99.7k |
| Cost-optimal planning (Task 2) | gpt-oss-20b | 100 | 0.20 (20/100) | 0.20 (20/100) | 138.7k |
| Cost-optimal planning (Task 2) | gpt-oss-120b | 100 | 0.67 (67/100) | 0.59 (59/100) | 145.6k |
| *Each task: 50 problems × 2 runs per model. |
We parsed the reasoning traces in the same way we did for ToH tasks. The results showed that both gpt-oss models predominantly adopted MEA in Mystery Blocksworld tasks, rarely relying on exhaustive search or analogy (Figure 16 in the revision) since exhaustive search is not tractable in the complex state space of this task.
We further detected all four primitives in the MEA episodes of Task 1. Please see Figure 23 for an actual example of MEA episode. We then measured how often the model carries out the target primitive when it could. In the specific context, the model would decide to implement a specific primitive (P) or not (N). For example, after proposing a specific move, the model would choose to verify the moves against the rules (P for verification) or directly update the states with the move applied (N for verification). Different Primitives would have different types of context. Figure 20 in the revision illustrates, for each primitive, the context, and the P and N moments. Specifically, we computed the fraction of decision points at which the model carries out the primitive (P) among all decision points of the same context type, . We measured this metric in both ToH and Mystery Blocksworld tasks. We found that they are employed at broadly similar frequencies and patterns in both tasks (Figure 6b in the revision), with backtracking the least frequent primitive.
- Experiments on LRMs from another model family.
We extended our analyses onto Qwen3-235B-A22B-Thinking-2507 which achieved a high accuracy of 94.6% on the same 3-disk ToH task. We parsed its reasoning traces of Qwen3-235B into different reasoning strategies as we did for gpt-oss models. As shown in Figure 15 in the revision, Qwen3-235B, like the gpt-oss models, relies mainly on MEA, but unlike them it almost never resorts to exhaustive search.
We further successfully detected all four primitives in the MEA episodes. Please see Figure 22 in the revision for an example. We then measured how often the model carries out the target primitive when it could using the same metric which we used for gpt-oss models, . Figure 6 in the revision shows that the three LRMs (gpt-oss 20b/120b and Qwen3-235B) have similar tendencies to perform verification, action proposal, and backtracking. However, Qwen3-235B proposes the subgoal of removing the blocker much less when a move is infeasible; instead it tends to propose an alternative move for the blocked number.
Response to W2/Q2
We appreciate the reviewer's comment. In our steering experiments, we mainly focused on extracting embeddings at the decision points where the model decides whether to implement certain primitives or not. Thus, the steering vectors were meant to control whether to do the primitive rather than how to do it, and it is hard to expect that such steering vectors would always induce correct verification.
Steering effects on the downstream performance. We agree that it is still meaningful to examine how such steering affects the correctness of the induced verifications and the downstream performance. Therefore, we designed another intervention loop with steering vectors in the revision. To gain stronger steering effects, we injected the steering vectors at the outputs of all MoE layers at once, with the best steering magnitude selected on a small set of actual traces. In the intervention loop, we always started by detecting the moments when the model chose not to verify (i.e., an N moment) after proposing a move (i.e., the context), then injected the steering vectors at all layers. We let the model continue generation after injection and detected N moments again, repeating the loop until a path was found. Then we forced the model to output answers. We ran this loop on traces whose original answers were incorrect and that contained at least one unverified action proposal (N moment). As Figure 10 in the revision shows, after injection, the frequency of implementing verification dramatically increased compared to the control condition, in which we injected the steering vectors for backtracking with matched vector norm and magnitude (from ~11% to ~85%). However, the accuracy of the induced verifications did not differ between the two conditions. The downstream answer accuracy was also higher than in the control condition, but the difference did not pass the significance tests, so it is hard to confidently claim that steering with verification vectors would increase model accuracy. We looked at some failure cases after we injected the verification vector: the model did verify later but did not implement the verification correctly, thus causing invalid answers. This is consistent with our expectation above: the verification vector controls whether the model verifies, but not whether it verifies correctly.
| Model | α | Episodes | Arm | Accuracy | Verification rate | Verification accuracy | Checks left out |
|---|---|---|---|---|---|---|---|
| gpt-oss-20B | 10 | 35 | Steering | 16/35 (0.46) | 100/115 (0.87) | 86/93 (0.92) | 7 |
| Control | 12/35 (0.34) | 23/198 (0.12) | 19/21 (0.90) | 2 | |||
| p | .45 | < .001*** | .67 | ||||
| gpt-oss-120B | 20 | 9 | Steering | 7/9 (0.78) | 35/41 (0.85) | 24/28 (0.86) | 7 |
| Control | 3/9 (0.33) | 6/54 (0.11) | 6/6 (1.00) | 0 | |||
| p | .22 | < .001*** | 1.00 |
*For measuring verification accuracy, we excluded verifications whose accuracy could not be reliably judged, e.g., when the board state could not be inferred from the text.
*Although the verification accuracy in the control condition is higher than that in the steering condition for gpt-oss-120B, the control induced far fewer verifications.
New steering experiments across all primitives. At the same time, we would argue that studying the internal embeddings responsible for deciding whether to do a primitive is also non-trivial and valuable. In the revision, we adopted a more effective and rigorous way to extract steering vectors and applied it to all four primitives, unlike just verification in the original submission [Section 3.2 in the revision]. The core idea of extracting steering vectors is taking the difference between the average embedding over all the decision points where the model chose to implement the primitive (P) and those where the model chose not to (N), in the same type of context. By extracting steering vectors from each MoE layer's output and injecting them on short synthetic prompts at the same layer, we managed to induce a higher frequency of starting to implement the primitives than both the control conditions, where we injected the steering vectors of another primitive onto the same prompts, and the unsteered baseline, at the most effective layer, across all primitives in both models (Figure 5). The significant steering effects mostly emerge when we inject at the later layers (L18-L22 in gpt-oss 20b, L22-L30 in gpt-oss 120b). Since the short synthetic prompts mainly mention the list configurations and certain moves and lack context about the ToH task (i.e., rules, task goal, etc.), such results demonstrate the generalization of these steering vectors to the synthetic prompts.
We also acknowledge that studying the internal mechanism behind the actual implementation of primitives, i.e., how the model performs a primitive correctly, would be an interesting open question worth further exploration, which we also mention in Section 7 of the revision.
Response to W3/Q3
To test whether the improvements come from the targeted primitive rather than from broader changes in the later reasoning, we checked whether the specific mistake we edited at each step still appears in the model's final answer.
Across all four primitives, the interventions largely remove the edited mistake from the final plan (0–11% carried over), while it persists under the paraphrase control (24–80%). These results measure the impacts of the specific steps we edited rather than overall accuracy, which would indicate that the interventions act through the targeted primitives, not through broad changes in the subsequent reasoning. We have added these results to the manuscript as a new figure (Figure 12).
The below are detailed results for your reference. We would also welcome the reviewer's view on other potential confounds, and would be glad to design additional controls to rule them out.
We first removed every trace in which either the repair intervention or the control run switched to exhaustive search after many edit rounds. This affected 10% of the edited runs (31 of 302 across the four repair experiments).
Verification. We identified checks in which the model wrongly accepted an illegal move. We edited each such check so that the move was rejected, and then asked whether the same move, from the same state, appears in the final answers.
| Model | Repair | Control | p |
|---|---|---|---|
| gpt-oss-20B | 1/36 (3%) | 11/26 (42%) | < .001 |
| gpt-oss-120B | 1/28 (4%) | 8/33 (24%) | .031 |
*The repair and control could have different numbers of edited steps for the same original trace.
After the repair, the illegal move rarely reaches the final answer (3–4%). Under the control, it does in 24–42% of cases.
Action proposal. For the repair of non-optimal action proposals, we asked whether the model's original non-optimal move, from the same state, appears in a final answer.
| Model | Repair | Control | p |
|---|---|---|---|
| gpt-oss-20B | 0/40 (0%) | 14/31 (45%) | < .001 |
| gpt-oss-120B | 0/49 (0%) | 22/66 (33%) | < .001 |
None of the replaced moves appear in the final answers after the intervention, whereas 33–45% remain under the control.
Subgoal proposal and backtracking. The same pattern holds for the other two repairs:
- Subgoal proposal: does the model still remove the worse of the two obstacles first in the final answer?
- Backtracking: after the model leaves the optimal path, do the moves that we forced the model to abandon still appear in a final answer?
Subgoal proposal
| Model | Repair | Control | p |
|---|---|---|---|
| gpt-oss-20B | 0/3 (0%) | 2/3 (67%) | .40 |
| gpt-oss-120B | 0/7 (0%) | 8/10 (80%) | .002 |
*We counted only cases in which the blocked move has two obstacles, one on its source list and one on its target list. Such cases are rare.
Backtracking
| Model | Repair | Control | p |
|---|---|---|---|
| gpt-oss-20B | 11/102 (11%) | 25/67 (37%) | < .001 |
| gpt-oss-120B | 1/23 (4%) | 10/27 (37%) | .006 |
p: Fisher's exact test.*
Summary of New Experiments
Thanks for the reviewer’s suggestions and comments. We first summarize some major new results before replying to the specific concerns raised in the review:
Please refer to the uploaded revision which highlight changes in blue.
[1] Valmeekam, K., et al. (2023). Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. Neurips 2023.
[2] Gerald J Sussman. A computational model of skill acquisition. 1973.