acceptodds
Under review as a conference paper at ICLR 2027

Not Which Path, but What Happens Before You Would Stop: What Reasoning Post-Training Changes

Abstract

Post-training for reasoning makes language models much better at mathematics, yet the abilities it exploits already exist in the base model and its lasting changes touch few tokens, so what those changes do is unclear. We find that they change what happens before the model commits to an answer rather than which path it takes, and that a few words carry the change. Our instrument is a transplant in which one model writes and, wherever the two disagree about the next token, at 3 to 6% of positions, the other model's word is substituted. Given to the base model, the reasoning model's words close 83% of the accuracy gap, taken from the reasoning model they remove 68% of its advantage, and the same dose of foreign words at random positions removes 12%. The difference is not made at the steps that decide a run's outcome. Forcing each option there shows that both models mostly agree and neither chooses reliably better. Where post-training lengthens the run, the words act before the stop rather than at it. In the primary pair they recover the effect when the base model is left to end its run on its own, while the word at the stop alone, or a fixed “Wait”, recovers nothing. The words carry most of the gap in three model pairs, one trained by reinforcement learning alone, and the effect is largest where the base model solves a problem occasionally and small where it has mastered it. Within the base model's reach, post-training thus changes how existing capabilities are deployed rather than what they are. That gives interpretability a concrete target, locates the gain that sampling alone does not recover, and says which problems, measured against the base model itself, post-training reaches.

Primary area:
interpretability and explainable AI
Keywords:
reasoning post-training, mathematical reasoning, mechanistic interpretability, language models, chain-of-thought, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_zBeBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The author expanded the existing method “transplant,” a substitution method that outputs trajectories at gate positions where the base and reasoning models disagree. With dose-adjusted control experiments, the author confirmed that transplant alters task accuracy in both directions for hard-level problems, and this phenomenon is consistent across Qwen variants. Also, the author revealed that forks (positions with top-2 choices that significantly change consequential accuracy) are disjoint from gate positions. When the output length differs between the base and reasoning models, transplant carries output length and commits, which are positions where a model writes its final answer. However, extending reasoning length simply by injecting “wait” does not work. Reasoning models also have the ability to recognize correct runs among sampled trajectories. The author finally provides an interpretation that a reasoning model acquires exploration.

Critical strengths

Considered random control experiments: The author conducts random control experiments with an effective amount of substitution by considering the ratio of off-support words.

Critical weaknesses
  1. Detailed experiments, but still unclear conclusions: The paper seems to fail to provide a clear answer to the research question “what those few tokens buy,” stated in Section 1. In the conclusion section, the author insists that carrying what the model does before termination is the role of transplanted tokens. However, this is not narrowed down enough because the part before termination accounts for most of the output tokens. Furthermore, the author proposes an interpretation that the author insists fits the experimental results. However, further studies to validate the interpretation are not conducted.
  2. Limited target for “fork”: The author identifies “forks” by the discrepancy between the consequences of the top-2 tokens. Rather, I suppose that forcing a reasoning model's word at only one (not a few) of the “gate” positions could change the consequences. In this regime, forks match gates by definition, but this could lead to a more detailed analysis.
  3. Injecting “wait” at the end of the output is a somewhat simple intervention: A model has already completed writing its final answer at the stop position. Injecting “wait” at other positions (e.g., after a period in a sentence at random places) could be considered.
Questions for authors
  1. What does “transplant” carry into base models beyond superficial characters?
  2. Why does running through the transparent code with no substitutions alter performance in Section 3?
3 comments

Thank you for your time and for a careful and constructive review. Your summary is accurate. One small detail is that only the first pair is Qwen3. The second is Olmo-3, a different model family, and the third is SimpleRL-Zoo on Qwen2.5. Since the submission we also ran a fourth pair, Open-Reasoner-Zero on Qwen2.5-7B, trained with reinforcement learning only and with runs about twice as long as its base model's. The result is the same. On contested problems its tokens close .75 [.49, 1.02] of the gap and the base model's tokens remove .81 [.56, 1.09], against .05 [−.17, .24] for tokens at random positions.

Unclear conclusions. We agree in part. The paper shows more than "before the stop". With all substitutions except those at the stop, the transplant closes as much of the gap as the full transplant, and the token at the stop alone closes nothing. The base model with the reasoning model's tokens also writes its answer once, late in the run (83% of the way in), and keeps it. So the tokens change the work that comes before the answer is written. But you are right that this is still a wide region, and that the paper does not say which tokens in it matter or why. We also agree that the interpretation at the end is not tested. It is offered as a hypothesis, and we will mark it more clearly as one.

Forcing one disagreement position at a time. Thank you for this suggestion. We have not run it. Forcing the reasoning model's token at a single position where the two models disagree, and measuring how much the outcome changes, would show directly which positions matter. We think it is a good next step.

"Wait" only at the stop. This is a fair point. We chose the stop because that is where budget forcing is usually applied. But at the stop the base model has usually written its answer already (at about 99% of the run's length), so a "Wait" there may come too late. Our test does not rule out that a "Wait" earlier in the run would help. Adding it after random sentence ends, as you suggest, is a good test, and we have not run it.

What does the transplant carry beyond surface tokens? We cannot answer this fully yet. Here is what we know.

  • The reasoning model supplies very few of the numbers (0.5%) and symbols (about 2%) in a transplanted run. So the base model does the calculations itself.
  • Refusing the doubt words ("wait", "but", "check" and similar) leaves the effect almost unchanged (.79 and .83 of the gap against .81 and .85, see our reply to Reviewer h6dn).
  • 60% of the substituted tokens are ordinary words. Substitutions are also much more frequent in the first 64 tokens (22%) than later (4.4%), and the first one is always "Okay" at the first token. We have not yet measured how much of the effect comes from the start of the run. A test that stops substituting after the first N substitutions would show this.

Why the run with no substitutions differs (Section 3). The difference is within sampling noise. With nothing substituted, the transplant code gives .725 on rare and .925 on contested problems, against .769 and .902 for the reasoning model alone. Compared problem by problem, the differences are −.044 [−.092, +.002] on rare, +.023 [−.010, +.056] on contested and −.010 [−.040, +.019] on both together. They have opposite signs and all intervals include zero. The two numbers also come from different runs (two samples per problem against eight, with different random seeds). We will make this clear in the text.

Thank you for your comment. I understand that the tested models were not only Qwen3 variants, and the performance difference seems to be random noise at inference time. I have read your comment and confirmed additional characteristics of transplanted tokens (e.g., few numbers and symbols). However, further experiments to reveal the points mentioned have not been conducted; the contributions of this work remain limited. I therefore keep the current rating.

Thank you for your comment and time, we will work on the highlighted points in future work.

Reviewer_h6dnOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper analyzes the differences in token-level reasoning pathways between base and reasoning models, showing that replacing a few tokens in base model reasoning that two models disagree can recover the reasoning performance. Specifically, most of replaced tokens are a "deliberation tokens" such as "wait", arguing that the base models already possess reasoning capabilities and that post-training improves them to reach the correct answers.

Critical strengths

The idea of investigating the token-level differences between base and reasoning model trajectories through token replacement is somewhat interesting.

Critical weaknesses

Several critical flaws exist in the experimental analysis.

  1. The substantial performance gain from token replacement largely rely on a subset that "the base model can already solve under pass@k". This challenges the paper's assumtion that "the base models already possess sufficient reasoning capabilities". The gains are limited on unreachable problems, suggesting that post-training can also play an important role beyond mere steering.

  2. The observation that tokens such as "wait" are frequently involved in the replacement does not imply that "such tokens make accuracy gains" and this risks conflating statistical prevalence with causal relevance. Specifically, considering that RLVR often encourages generating such tokens, the critical tokens that improve performance may lie in elswhere.

These distinctions can be critical to understanding the effect of post-training and could drive the other conclusions. The authors should provide a more rigorous analysis.

Questions for authors

The figures and tables are poorly presented and difficult to follow, resembling low-quality AI-generated content. The authors should significantly improve these presentations.

3 comments

Thank you for your time and for your review. We answer your points in order.

The gain relies on problems the base model can already solve. We agree with the observation, and it is one of the paper's own findings (the fourth finding in the introduction, and Table 1). The paper does not assume that the base model has enough ability for every problem. Its claim is limited to problems "within the base model's reach", in the abstract and in the limitations. On problems the base model never solves in 64 tries, we report that the transplant lifts it only from .00 to about .05, against .15 for the reasoning model. So we agree with you that post-training does more than steering there, and the paper says that those problems are outside its claim. We will state this limit earlier and more clearly.

Frequent "wait" tokens do not show that they cause the gain. We agree with the principle, but the paper does not make this claim. It reports that such words are 18% of the substituted tokens against 3% elsewhere, which is a description, and it is not most of them. The paper also gives evidence against these words being the cause. The introduction cites work showing that removing "wait"-style restarts leaves accuracy almost unchanged, and in Section 5 adding a fixed "Wait" recovers nothing. We see that the phrase "the vocabulary of deliberation" in Section 3 invites your reading, and we will change it.

Your comment led us to test this directly. We ran the transplant again and refused these words from the reasoning model (wait, hmm, but, however, actually, alternatively, instead, hold, check, verify, recheck, double-check, reconsider, let). At those positions the base model keeps its own token. The result barely changes. The transplant closes .79 [.65, .95] of the gap on rare problems and .83 [.66, .98] on contested ones, against .81 and .85 with all tokens (two runs per problem). So these words do not produce the gain, which agrees with your point that the important tokens are elsewhere. Which tokens they are is still open. 60% of the substituted tokens are ordinary words, and the reasoning model writes only 0.5% of the numbers in a run. One possibility is that no specific kind of token matters. Post-training may make many small changes to the model's internal representations, and the substituted tokens are only where those changes become visible in the output. In that case the effect would be spread over many ordinary tokens. This is a hypothesis, and we have not tested it.

Figures and tables. We agree that some are dense. We will simplify them.

Thank you for the additional experiments and clarifications. I appreciate the analysis excluding deliberation tokens, which addresses one of my original concerns. However, my broader concern regarding the paper's central contribution remains.

The experiments demonstrate that disagreement-based token transplantation can recover much of the reasoning performance gap, but what makes these substitutions effective remains unclear. The fork analysis does not directly examine the causal effects of the transplanted tokens, while the stopping experiments primarily rule out alternative explanations rather than identify the underlying mechanism. The authors also acknowledge that which tokens matter, why they matter, and whether the proposed sequential-search interpretation holds remain unresolved.

Given that the effectiveness of token transplantation has already been demonstrated in prior work, the paper needs a clearer explanatory account of its findings to establish what new understanding of reasoning post-training it provides. While the additional analyses are informative, I remain unconvinced that they sufficiently support the paper's central claims. I therefore maintain my original assessment.

Thank you for the comment. We understand and agree with the points raised and will work on them in the future.

Reviewer_yRuzOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper investigates the capability differences between a base and a reasoning post-trained model and assesses the benefits of post-training an LLM to be reasoning-capable. To do this, the authors propose to transplant one model into the other by switching the sampling away from the main model (referred to as "host") and onto the other one for individual tokens, in cases where the models disagree most (based on a KL divergence). The paper focuses on accuracy gaps as a result of the above "transplantation". The authors experiment with Qwen3-8B and Qwen3-8B-Base, as well as Olmo-3-7B-Think and Olmo-3-1025-7B. The experiments are conducted on 300 samples each drawn from GSM8K, MATH-500, AMC23, and AIME. The results show that interfering in the reasoning model's sampling process with the base model drops accuracy by 68%, and using the reasoning model for the base model as host increases accuracy by 83%.

Critical strengths

The paper's experimental setup is overall technically sound, and the scope and evaluation setup is well-defined.

Critical weaknesses
  • The paper's motivations for the experimental setup (the transplant approach) are not well-defined or -explained. It is unclear what this experimental setup is aiming to test, and what the main hypotheses of this work are.
  • The paper is written in a convoluted way and in large parts contains metaphorical language typical of text generated with recent LLMs (e.g., "the gate fires", "carry", "the behaviour that hurts"). This language makes the paper hard to read and understand, and thereby seems to somewhat obfuscate key information.
  • The evaluation setup of the paper is not clearly motivated. What is the motivation behind building an evaluation set of evaluation sets (with a subsample of 300 samples per task) as compared to running the experiments across all such evaluation tasks?
  • Even though acknowledged by the authors, the paper is limited in scope by only focussing on the interaction of base and reasoning models in the context of such mathematical tasks.
  • It is not clearly formulated what the implications of this paper's findings and insights are for future work. How can we use the insights obtained in this work in future research?
Questions for authors
  • What exact evaluation tasks does the "AIME pool" cover?
1 comment
AuthorsOpenReview ↗

Reply to Reviewer yRuz

Thank you for your time and for your review. We answer your points in order.

Two details of the summary. The 300 problems are the whole set, not 300 per task (151 from MATH-500, 90 from AIME, 30 from GSM8K and 29 from AMC23). Also, 83% and 68% are fractions of the accuracy gap between the two models, not of accuracy. For example, on the rare band the base model scores .12 and the reasoning model .77. With the reasoning model's tokens the base model reaches about .65, which is 81% of the way. The paper also has a third pair, SimpleRL-Zoo, trained with reinforcement learning only.

Motivation and hypotheses. We respectfully think the paper does state them, in the introduction. Earlier work shows that reasoning post-training changes only a few token positions. Our question is what those few tokens do for accuracy. The transplant is a direct test. One model writes, and where the two models disagree most, the other model's token is used instead. The introduction says what it tests, in three parts.

  1. Are the reasoning model's tokens enough to raise the base model's accuracy?
  2. Are they needed for the reasoning model's own accuracy?
  3. Does their position matter, compared with the same amount of tokens at random positions?

The answer to all three is yes. Then the paper tests two common explanations. One is that the gain is made at the few steps where the outcome of a run is decided (Section 4). The other is that it is made where the base model would stop (Section 5). The results support neither. We agree that we can make this easier to find, for example with a short numbered list at the end of the introduction.

Language. We agree. Terms like "gate" and "host" are defined in Section 2, but phrases like "the gate fires" and "the words carry" are figures of speech, and plain terms would be easier to read. We will change them. There are other readability issues that we will work on.

Why a set of 300 problems. Section 2 gives the reason. The effect depends on how often the base model solves a problem by itself, so difficulty has to be measured against the base model. For each problem we estimate that rate from up to 64 samples. Then we group the problems into five bands, from never solved to always solved, with 60 problems each. The full benchmarks would not give this balance, because most of their problems are either too easy or too hard for the base model. Each transplant run also needs both models at every token, and reasoning runs are about 8,000 tokens long, so the full benchmarks would be very costly.

Mathematics only. We agree. This is a limit of the paper, and we say so.

Implications for future work. The abstract and the conclusions name three, and we can make them more visible.

  1. For interpretability, the positions where the two models disagree are a small set (3 to 6% of tokens) where the effect of post-training can be studied.
  2. Most of the gain is on problems that the base model solves sometimes but rarely. Measuring post-training against the base model's own success rate shows which problems it reaches.
  3. On problems that the base model solves about half the time, a majority vote over 8 base-model runs comes close to the reasoning model (.84 against .90). On rarely solved problems it does not (.19 against .77). So more sampling does not recover the gain where the gain is largest.

The AIME pool. It contains AIME I and AIME II from 2013 to 2023, 327 of the 330 problems (2013-I-2, 2022-II-8 and 2023-I-15 are missing). 120 of them were candidates and 90 are in the final set of 300 (37 unreached, 35 rare, 15 contested, 2 reliable and 1 mastered).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.