acceptodds
Under review as a conference paper at ICLR 2027

Correct So Far, But Where Next? Diagnosing Next-Step Failures in Multi-Step Reasoning

Abstract

Chain-of-thought reasoning improves multi-step problem solving by decomposing a difficult computation into simpler steps, but decomposition only helps if the steps are composed correctly. When a reasoning chain first goes wrong after a sequence of correct steps, existing evaluations do not distinguish whether the model chose the wrong computation to perform next or chose the right one and executed it incorrectly. Meanwhile, verified, truncated, or relayed reasoning traces are reused in practice on the assumption that a correct intermediate result is a state from which the computation can continue. We show that this assumption can fail even when every supplied step is correct and the remaining computation is well within the model's ability. Across controlled serial tasks with a unique correct next step, models often preserve the correct intermediate result yet continue from the wrong place, repeating or skipping steps or taking the wrong branch. These failures primarily reflect choosing the wrong computation to perform next rather than executing the right one incorrectly. Explicitly identifying the next step or the remaining computation removes much of the gap, whereas merely restating the current result does not. Training and internal-state interventions reinforce this distinction: the same remaining computation can succeed or fail depending on how it is entered, and internal states formed by a fresh run can repair a failed continuation. A correct intermediate result is therefore not necessarily a reasoning state from which the model can reliably continue.

Primary area:
interpretability and explainable AI
Keywords:
language model reasoning, latent reasoning, chain-of-thought, looped transformers, recurrent-depth transformers, intermediate reasoning, mechanistic analysis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_95wsOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This work shows that LLM may still fail even though all the previous steps are provided correctly. Based on controlled tasks, they find that these failures primarily reflect choosing the wrong computation to perform next rather than executing the right one incorrectly. The experiment also shows that identifying the next step or the remaining computation can mitigate this issue.

Critical strengths

The definition of the task and the experiment is carefully controlled.

The proposed method is effective to mitigate the transition errors.

Critical weaknesses
  1. The main concern for this work is its generalization to real-world complex reasoning problem. The experiments are conducted on non-standard benchmarks. Even though the findings provide some insights, they may not be practically useful.

  2. The chosen tasks are simple and the models evaluated are either old or small. It is necessary to evaluate on strong LLMs to validate the claims.

  3. Some statements are difficult to follow due to overly-long sentences, dense terminology or lack of logical connections. For example, in lines 476-478, “A model’s own steps carry the next operation along with each result; a record of results carries the results alone, and the model enters the remaining work at the wrong step while holding the right value.” would be easier to understand if you convert it to “A model's own reasoning steps may encode both the computed result and the next operation, whereas a record containing only intermediate results may leave the next operation unspecified. As a result, a model may resume computation at the wrong step despite having the correct intermediate value.”

Questions for authors
  1. Is it possible to validate the claims on widely-used benchmarks such as gsm8k?
  2. In Figure 1, for fresh problem and continue from a record, should we regard them as two different problems instead?
2 comments
AuthorsOpenReview ↗

Response (Part1)

We thank the reviewer for the comments and for the concrete writing suggestion, which we adopted. We are glad that the reviewer found that "the definition of the task and the experiment is carefully controlled" and that the proposed method "is effective to mitigate the transition errors". Reviewer LrVd likewise describes the comparisons as "carefully matched" to "separate continuation difficulty from execution ability". We ran both requested evaluations, and the revised PDF shows all changes in blue. Bracketed numbers such as [L266] are line numbers of the original submission.

W1 / Q1 — Standard benchmarks such as GSM8K. Yes, and they show the boundary our account predicts. We added GSM8K and the code benchmark CRUXEval (Section 4.5, Table 3, Appendix H), both with prefixes that are correct by construction. Accuracy for the models without explicit reasoning:

ModelCRUXEval, scratchCRUXEval, from correct recordGSM8K, scratchGSM8K, from correct record
Qwen2.5-14B0.6320.5470.9480.958
Gemma 3 12B0.6460.5650.9230.952
Phi-40.7090.6530.9500.966
Llama-3.3-70B0.7400.7650.9510.971
Qwen3.5-27B0.9370.8390.9600.980
Qwen3.5-122B-A10B0.8460.6740.9670.985
gpt-oss-20b, low effort0.8910.8070.9340.963

On CRUXEval, the next line depends on loop and branch state that the record does not spell out. Continuing from the correct execution record is 5.6–17.2 points less accurate than predicting from scratch for six of the seven models, 17 points for Qwen3.5-122B (−0.172 [−0.225, −0.123]), and only 175–227 of 285 continuations start at the next line to execute. On GSM8K, the problem text names each quantity to compute next, so the record suffices, for all ten model configurations including the reasoning models (differences −0.005 to +0.029). A correct record is enough when the next step can be read off the problem, and fails when it must be inferred from the state. Records of completed steps are how reasoning is verified, truncated, relayed, or reused in practice [L090–092], so this boundary matters for real systems. The controlled tasks remain our core evidence because they make every step verifiable; as Reviewer iDiv notes, this "wouldn't be possible in a setting where only the final answer can be verified."

AuthorsOpenReview ↗

Response (Part2)

W2 — Strong LLMs. The failure persists in strong 2026 models, and reasoning models avoid it only by recomputing the completed steps. Our submission covered four model families, including the 2026 Qwen3.5 releases [L266–268, L290–291]. We now add Qwen3.5-27B and Qwen3.5-122B-A10B (thinking off and on), Llama-3.3-70B, and the reasoning models gpt-oss-20b and gpt-oss-120b (Section 4.4, Figure 2b, Table 11). All record conditions share the 192 programs, the correct record, and the demonstrations; only one appended sentence varies.

Modelown trajectoryone steprecord+ value+ next node
Llama-3.3-70B0.9320.9950.6300.5990.943
Qwen3.5-27B, no thinking0.9841.0000.8650.8440.990
Qwen3.5-122B-A10B, no thinking0.9950.9950.8440.8020.995
gpt-oss-20b, low effort1.0001.0000.8440.5420.927
gpt-oss-20b, medium effort1.0001.0000.9740.7710.995
gpt-oss-120b, medium effort0.9951.0000.9580.9381.000
Qwen3.5-27B, thinking1.0001.0001.0000.9951.000
Qwen3.5-122B-A10B, thinking0.9901.0001.0000.9951.000

Without explicit reasoning, Llama-3.3-70B, Qwen3.5-27B, and Qwen3.5-122B reach only 0.630, 0.865, and 0.844 from a correct record, although each executes the next step alone at 0.995–1.000. One sentence naming the next node restores 0.943, 0.990, and 0.995 (+0.312 [0.245, 0.380], +0.125 [0.078, 0.177], +0.151 [0.104, 0.203]); restating the value does not. Reasoning models reach high accuracy by recomputing what the record leaves out: at medium effort, gpt-oss re-tests the last completed node's branch condition in 51–57% of record prompts, and both Qwen3.5 models with thinking re-execute the whole program in every record prompt, spending more tokens than when solving from scratch (4,102 vs 2,325; 3,378 vs 2,339). At lower effort the loss returns (gpt-oss-20b: 0.844), and naming the next node again restores it (0.927).

W3 — Long sentences. We adopted the reviewer's suggested rewrite. Section 6 now reads: "A model's own reasoning steps may encode both the computed result and the next operation, whereas a record containing only intermediate results may leave the next operation unspecified. As a result, a model may continue at the wrong step despite holding the correct intermediate value." We shortened long sentences throughout, and new Tables 1 and 2 list the prompt conditions and the experimental setup.

Q2 — Figure 1: two different problems? They are the same remaining computation by construction [L089–091, L185–186]: the same current value, the same remaining operations or program nodes, and the same answer. The fresh problem states that remainder directly; the continuation presents the full problem with a correct record of the completed steps. The fresh problem certifies that the remaining work is within the model's ability [L184–186], and the cause of the gap is isolated by changing one thing inside the record prompt [L303–310; new Figure 2]: adding the successor to each record line cuts first-step transition errors from 0.24–0.89 to at most 0.009, and appending "The next node is Y." lifts accuracy from 0.45/0.51/0.38 to 0.78/0.82/0.98, whereas restating the value does not.

Reviewer_LrVdOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why language models can fail to continue from a correct intermediate reasoning record. It distinguishes transition errors, where the model enters the wrong operation or uses the wrong input state, from execution errors after selecting the correct operation and input. Paired comparisons hold the remaining computation fixed while presenting it either as a fresh problem or as a continuation of a correct prefix. Across synthetic arithmetic and program-execution tasks, models often retain the correct value but repeat, skip, or enter the wrong step.

Critical strengths
  1. This paper uses carefully matched comparisons to separate continuation difficulty from execution ability.

  2. This paper includes useful interventions across multiple model families. In particular, adding only the next-node pointer while retaining the original program and history substantially improves conditional-program continuation.

Critical weaknesses
  1. The practical scope of the continuation failure remains insufficiently established. In arithmetic, the large gaps mainly occur under immediate-answer evaluation; allowing written continuation raises accuracy to 99-100% at every tested split point for the four evaluated base models (Section 4.2).

  2. The training and hidden-state experiments provide limited additional explanation of the underlying failure. The matched training design controls the remaining computation, but its results remain compatible with learning particular depth-boundary combinations and ways of presenting the continuation.

Questions for authors
  1. How often do transition failures occur when naturally generated, verified prefixes are reused on reasoning tasks beyond the synthetic execution settings studied here?

  2. What evidence distinguishes learning a reusable continuation rule from learning the supervised depth-boundary combinations and prompt format?

  3. Why do explicit variable-state and next-line cues largely close the Python continuation gap for Phi-4 but not for Qwen and Gemma? What additional source of failure does this suggest?

3 comments
AuthorsOpenReview ↗

Response (Part1)

We thank the reviewer for the positive assessment and the precise questions. We are glad that the reviewer found that the paper "uses carefully matched comparisons to separate continuation difficulty from execution ability", and that "adding only the next-node pointer while retaining the original program and history substantially improves conditional-program continuation" across model families. Reviewers iDiv and 95ws likewise found that the design reveals "the nature of the errors" beyond final answers and is "carefully controlled". We answered each question with new experiments, and the revised PDF shows all changes in blue. Bracketed numbers such as [L322–323] are line numbers of the original submission.

W1 — Practical scope. The failure appears wherever the next step must be inferred from the state, and the new experiments show it in strong models and on a standard benchmark. The arithmetic result the reviewer cites is the boundary our account predicts and the submission states [L093–095, L322–323]: when every completed operation is written out against the question's list, the next operation can be read off. When the next step depends on the state, the failure is large even with written continuation and generated reasoning. In conditional programs, the first step written after a correct record is a transition error on 0.39–0.74 of prompts, while execution errors at that step stay at or below 0.06 [L296–302, L371–377]. The new results carry this beyond our tasks and models:

  • Strong models. From a correct record, Llama-3.3-70B, Qwen3.5-27B, and Qwen3.5-122B-A10B reach 0.630, 0.865, and 0.844, although each executes the next step alone at 0.995–1.000; one sentence naming the next node lifts them to 0.943, 0.990, and 0.995 (Section 4.4).
  • A standard benchmark. On CRUXEval, a correct execution trace makes the return value 5.6–17.2 points less accurate than starting from scratch for six of seven models without explicit reasoning, and 17 points for Qwen3.5-122B (Section 4.5).
  • Reasoning models avoid the loss by re-deriving the completed transitions, at a price: Qwen3.5-27B with thinking re-executes the program from its start in every record prompt and spends 4,102 tokens to continue against 2,325 from scratch, and with less reasoning the loss returns (gpt-oss-20b at low effort: 0.844).

GSM8K confirms the boundary from the other side: its problem text names each next quantity, and continuation matches solving from scratch. Records of completed steps are how reasoning is verified, truncated, relayed, or reused in practice [L090–092]; these results tell when such a record can be trusted.

Q1 — How often, beyond synthetic execution. We measured it with verified prefixes on two standard benchmarks (Section 4.5, Appendix H).

  • CRUXEval, verified execution traces of benchmark functions. Only 175–227 of 285 continuations (61–80%) start at the next line to execute, and the return value loses 5.6–17.2 points as above.
  • GSM8K, verified human-written reference solutions. 10–52% of continuations begin by recomputing a quantity that is already done, and more from bare equations (up to 73%). Because the text names every quantity and a recomputed value is unchanged, accuracy matches solving from scratch (differences −0.005 to +0.029).
  • Reasoning models re-derive instead of continuing: on CRUXEval, Qwen3.5-122B with thinking uses 6,807 tokens from the record against 3,566 from scratch.
AuthorsOpenReview ↗

Response (Part2)

W2 / Q2 — A reusable rule, not memorized cells or formats. Three tests show that the broadly supervised adapters learn an entry rule that extends beyond their training (Section 5.3, Appendix K.4, new Table 17).

  1. Unsupervised combinations. Withholding a single interior split point (W52, W42, W53) while keeping several split points per depth costs little [L454–456].
  2. Unseen depth. Adapters trained on depths two to five continue at depth six [L1585–1586, L1614–1617].
  3. Unseen formats (new). Adapters trained only on records written as 12 - 2 = 10, 10 - 5 = 5. keep most of their advantage on records written with arrows, in words, or under a re-templated question: R1 reaches 0.88–0.96 and W52 0.76–0.95 over two seeds, against 0.32–0.35 for the base model, and 0.66–0.94 against 0.23–0.25 at W52's withheld cell (5,2).

The masking control shows how. The broadly supervised adapter reads the record: masking the chain drops it from 0.98 to 0.62. The restricted arm, whose single split point per depth makes depth predictive, keys on depth and is unchanged by masking. What a model learns about entry is thus set by supervision, which the matched design makes visible.

W2 — What the state experiments explain. They locate where the deficit forms (Section 5.2, Appendix J.4). In the symbolic format, every donor in which the remaining expression is not read under the original question restores the receiver in the first pass, a length-matched filler as much as the fresh problem (+0.045 [0.005, 0.083]); a donor that keeps the original question does not (+0.020 [−0.018, 0.058]). In the natural format, the remaining operations are encoded where they are written (an operation-swapped donor transmits them), and the failure lies in deciding, at the answer, which of them remain: the transition failure seen in behavior.

Q3 — Why the cues help Phi-4 but not Qwen and Gemma. The cues remove the failure they target; Qwen and Gemma pay a second, separate cost. We matched every wrong final answer against the results of entering the program at a wrong place from the paused frame (new Table 20, Appendix M.3). Events rendering, 128 programs:

ModelConditioncorrectwrong entryother entryno entry
Phi-4record854003
+ frame + next line1091531
fresh problem1121402
Gemma 3 12Brecord7736015
+ next line854138
+ frame + next line6814937
fresh problem124004
Qwen2.5-14Brecord21172565
+ frame + next line33212846
fresh problem991748

Wrong entry: re-running the current loop iteration, skipping the next line, or repeating the last executed line. Other entry: entering at another line or re-entering an earlier completed event. No entry: matching none of these.

The record induces wrong entries, and the cues remove them: Phi-4 falls from 40 to 15 (14 on the fresh problem) and Gemma from 36 to 4 (0 fresh), with Gemma starting at the cued line on 127 of 128 programs. Qwen's wrong entries were never record-induced; it makes them at its fresh-problem rate in every condition (17–21 vs 17). Its record-specific errors match no entry (46–65 vs 8 on the fresh problem), and Gemma shows the same kind once its entry is fixed (37–38 vs 4). These are execution errors made next to a long execution history, the same context cost we find in the symbolic format even when the remaining instructions are explicit (Figure 9, Appendix I.1). The cues fix the transition; Qwen and Gemma also pay for executing next to a long trace.

Thank you for the additional experiments, which substantially address my concerns. Attributing unmatched errors to execution failures still needs qualification, but overall the response strengthens the paper.

Reviewer_iDivOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
3(+1)
Read review
Paper summary

The paper tries to shed light on the on failures in language modeling multi-step reasoning, analising it on controlled tasks. The paper distinguishes between two kind of mistakes: the ones originating from chosing which step to do next and the ones originating from a mistake when doing the correct step. The core experiment holds the remaining computation fixed and compares continuing from a correct written record against solving the same remainder as a fresh problem. The gap is then probed with prompt edits, residual-state imports in a looped model, and LoRA training that varies which split points are supervised. They identify the primary failure in chosing the correct next step instead of performing the computation, even when the model's own trajectories rarely make transition errors and a single specified step is executed near ceiling.

Critical strengths
  • The idea is interesting and the proposed experimental settings to test it make possible to understand more finely the nature of the errors. This wouldn't be possible in a setting where only the final answer can be verified.
  • Experiments are performed on a wide set of different models, from different families and parameter count.
Critical weaknesses
  • The manuscript is extremely difficult to read. A list of some of the issues:
    • The necessary information to understand is spread out in different points of the manuscript, including the appendix. For example Sec 5.2, 5.3, 6 reference multiple figures and tables in the appendix.
    • Vocabulary and definitions are either defined but never used (ex, eq 1), change as the manuscript progresses (ex, "value + remainder"), defined in the appendix even if they are used in the main body (ex, "ordinary and explicit wordings"), never defined (ex, "entry" and "depth").
    • Sec 4 the experimental settings for the experiments have to be extrapolated from the model used, which is very confusing.
  • Several main-text figures compare conditions that differ in more than one factor. Fig 2 (record vs. own-format record differ in four things: successor, branch outcome, placement, demonstrations). Fig 3 (partial chain vs. value only both removes the derivation and adds a sentence). Figs 3, 4a (fresh vs. partial chain changes the history and the question form, plus the demonstrations in Fig. 4a). Fig 4a and Fig 1C (the “+ remainder” condition adds a pointer and the rules; Fig. 5 shows the pointer does most of the work). Some controlled comparisons are reported in the appendix, which in my opinion is not the right place from a readibility/clarity standpoint of the manuscript.
  • The residual-state import (Sec 5.2, Fig 1C-middle) does not show that a fresh run forms a usable state that continuation lacks:
    • It uses the symbolic format, where the remainder is already in the prompt and the fresh-minus-partial gap is only 0.10–0.17 (App. H.1), against 0.4–0.9 in the natural format and in conditional programs. The repair is tested where the phenomenon is mostly absent.
    • In this format the gap comes from the question: Table 8 shows the original question costs 0.14, and adding the derivation on top of it helps (0.73 to 0.78). App G1 says so itself: "a question describing completed work incurs a step-like penalty".
    • The imported states carry the donor's computation, not its progress: different-item imports transmit the donor's answer (App. H.2), and Sec 7 concedes that no portable progress variable is identified. What the experiment shows is that a different preceding context yields better states at the expression positions. A donor with a length-matched filler context, or the "remainder re-asked" prompt, would separate context interference from a record-specific effect.
Questions for authors
  • Experiments are carried using base and instruct models, not reasoning ones. Do you think conclusions will change on such models?
  • In Sec 5.3 the restricted arm has exactly one split point per depth, so question length alone predicts where to resume. Consistent with this, masking the chain leaves the restricted arm unchanged (Table 10) and withholding only (5,2) in W52 costs little (Table 9). How do you rule out a depth-keyed shortcut rather than a failure to enter at an unsupervised position?
3 comments
AuthorsOpenReview ↗

Response (Part1)

We thank the reviewer for the careful reading and for proposing concrete controls. We are glad that the reviewer found the idea interesting and the setting able "to understand more finely the nature of the errors", which "wouldn't be possible in a setting where only the final answer can be verified", with experiments "on a wide set of different models, from different families and parameter count". Reviewers LrVd and 95ws likewise found the comparisons "carefully matched" and the experiments "carefully controlled". We ran every proposed control, and the revised PDF shows all changes in blue. Bracketed numbers such as [L303–310] are line numbers of the original submission.

W1 — Readability. We reorganized the paper so that its argument can be followed from the main text alone.

  • Section 3.1 defines depth, split point, and entry together in prose (their symbols d, t, and j appear at [L171, L181, L194–195]), and Section 3.2 defines the ordinary and explicit wordings.
  • ρ in Eq. 1, the share of the gap an intervention recovers, is what Figure 1C plots [L075–076]; it is now defined in prose where it is used.
  • New Table 1 defines every main-text condition by its single difference from the record. The condition the reviewer mentions carries one name in the main text, "+ value + remainder".
  • Vocabulary is consistent between the main text and the appendix: "split point" replaces "boundary" throughout, and the appendix tables and figures on conditional programs use the condition names of Table 1.
  • Section 4 opens with one setup table for all comparisons (new Table 2); Sections 5.2–5.3 state their setup and key numbers in the main text, and the former Section 6 is folded into Sections 4.3 and 5.1.

W1 — One factor per comparison. Every main-text comparison now changes exactly one factor, and these comparisons are our strongest evidence (Section 4.2, Figure 2).

  • Successor clause (Figure 2a, in place of record vs own-format record). Within each of four record formats, with placement and same-format demonstrations fixed, adding only "; next is Y" to each line cuts first-step transition errors from 0.24–0.89 to at most 0.009 in all three families. The own-format record without the clause still errs on 0.24–0.81, so neither the branch outcome nor the format is what matters; prompt versus assistant-turn placement is compared directly (Appendix F.3).
  • One appended sentence (Figure 2b). With program, record, and demonstrations identical, "The next node is Y." lifts accuracy from 0.45/0.51/0.38 to 0.78/0.82/0.98 (Qwen2.5-14B/Gemma/Phi-4), whereas restating the value, the visit count, or a neutral sentence moves it by at most 0.07.
  • Arithmetic (Figure 3). An additive ladder that keeps the derivation: + value leaves the gap, and + value + remainder closes it.

We reported these single-factor results in the text of the submission [L303–310]; the revision makes them the figures. The fresh problem serves as a capability reference by design [L184–186], and the Figure 1C caption lists what "+ value + remainder" adds and points to Figure 2b, where the next node is added alone.

AuthorsOpenReview ↗

Response (Part2)

W2 — Residual-state import. We ran both donors the reviewer proposed, plus a donor with the original question and no derivation (new Table 14, Appendix J.4; Ouro-1.4B, symbolic format, four operations left, 398 items; receiver 0.774):

Donor (states imported at the remaining expression)donor accuracyreceiver after first-pass importbest import, gain [95% CI]
fresh problem0.8640.80–0.83+0.073 [0.035, 0.111] (pass 2)
length-matched filler (no question, no record)0.7660.81–0.82+0.045 [0.005, 0.083] (pass 1)
question and record, remainder re-asked as a new question0.8290.78–0.81+0.060 [0.025, 0.098] (pass 2)
original question, no derivation0.7340.77–0.78+0.020 [−0.018, 0.058]

The donors pin down what withholds the repair: reading the remaining expression as the continuation of the original question. Every donor in which the expression is not read under that question restores the receiver: the fresh problem, a length-matched filler, and even the same question and record once the remainder is re-asked as a new question. The donor that keeps the original question directly before the expression does not. The deficit forms in the first pass; this is the question-context effect the reviewer cites (Appendix I.1, formerly G.1), now localized to the first pass. The pattern replicates with five operations left (291 items).

In the natural format, where the gap is large, the same method shows that the remaining operations are encoded where they are written. An operation-swapped donor transmits its operations (up to 0.25 of Ouro's answers apply them to the receiver's own value), yet importing fresh-question states at those positions leaves accuracy at the receiver's level (Qwen2.5-7B 0.05–0.18 vs 0.11; Ouro 0.21–0.25 vs 0.23). What fails is deciding, at the answer, which operations remain, which is exactly where the behavioral results place the failure. Section 5.2 now presents the imports as this localization.

AuthorsOpenReview ↗

Response (Part3)

Q1 — Reasoning models. The conclusions hold, and reasoning models confirm them from the other side: they succeed by recomputing the very transition the record leaves out, and the failure returns when they reason less. We added gpt-oss-20b (low and medium effort), gpt-oss-120b, Qwen3.5-27B and Qwen3.5-122B-A10B with thinking on and off, and Llama-3.3-70B (Section 4.4, Figure 2b, Table 11). All record conditions share the 192 programs, the correct record, and the demonstrations; only one appended sentence varies.

Modelown trajectoryone steprecord+ value+ neutral+ next node
Llama-3.3-70B0.9320.9950.6300.5990.6460.943
Qwen3.5-27B, no thinking0.9841.0000.8650.8440.7860.990
Qwen3.5-122B-A10B, no thinking0.9950.9950.8440.8020.8440.995
gpt-oss-20b, low effort1.0001.0000.8440.5420.4480.927
gpt-oss-20b, medium effort1.0001.0000.9740.7710.8750.995
gpt-oss-120b, medium effort0.9951.0000.9580.9380.9581.000
Qwen3.5-27B, thinking1.0001.0001.0000.9951.0001.000
Qwen3.5-122B-A10B, thinking0.9901.0001.0000.9950.9641.000

At medium effort, gpt-oss re-tests the branch condition of the last completed node in 51–57% of record prompts, and Qwen3.5 with thinking re-executes the program from its start in every record prompt, spending more tokens to continue than to solve from scratch (4,102 vs 2,325; 3,378 vs 2,339). At low effort, gpt-oss-20b drops to 0.844, and a sentence after the record exposes the transition error of Section 4.2: with the value restated, 43 of its 44 wrong answers (77 of 88 at low effort) re-enter the last completed node with the correct value. Naming the next node restores accuracy (0.927 at low effort) and lifts even gpt-oss-120b (+0.042 [0.016, 0.073]).

Q2 — Depth-keyed shortcut. The matched design is built to reveal which cue a model learns, and the controls the reviewer cites show both outcomes.

  • Restricted arm, one split point per depth. Depth predicts the entry, and the model learns that shortcut: masking the chain leaves its withheld-entry accuracy unchanged (0.556 → 0.599), and 73% of its errors execute the operations supervised at that depth [L444–446, L1648–1649].
  • Broad arm, several split points per depth. It reads the record: masking the chain drops it from 0.984 to 0.619.
  • When depth does not predict the split, the learned rule reaches beyond supervision. Withholding a single interior split point (W52, W42, W53) costs little [L454–456], the trained adapters transfer to depth six beyond the training depths [L1585–1586, L1614–1617], and adapters trained only on the original chain format transfer to records written with arrows, in words, or under a re-templated question (new): R1 reaches 0.88–0.96 and W52 0.76–0.95 over two seeds, against 0.32–0.35 for the base model, and 0.66–0.94 against 0.23–0.25 at W52's withheld cell (5,2) (new Table 17, Appendix K.4).

We now state the result in these terms in Section 5.3: supervision decides which cue a model uses to locate the entry.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.