acceptodds
Under review as a conference paper at ICLR 2027

Base Models Can Reason by Taking a Cue from Their Training Data

59 views

Abstract

In this paper, we study how training data creates associations between the first tokens a base model outputs and its reasoning behaviors. First, we discover that fixing particular starting *token cues* makes a base model's performance competitive with that of its reinforcement learning (RL)-trained counterparts on math and coding. For instance, the cue ".\n\nOkay" raises Olmo-3-7B's MATH-500 pass@1 accuracy from 42% to 78%, while "Alright," raises Qwen3-14B's from 72% to 87%. Second, we demonstrate that RL itself learns to make these same starting cues more likely, while fixing these cues recovers much of its performance gain over the base model. Third, we trace the reasoning effects of token cues to the training data. We perform causal interventions on the data to turn an arbitrary word, such as "Chicken", into an effective reasoning cue or remove an existing cue's effect. We also find that the hidden state representations induced by different cues correlate with different document types from the training set. Finally, we extend our study of token cues to a safety setting, finding that different cues elicit distinct refusal and compliance behaviors that correspond to different types of training data.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, reasoning, data attribution, reinforcement learning, interpretability

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ATgQOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper focuses on the connections between the first tokens a base LLM model outputs and its reasoning behaviors from the training data aspect. An interesting phenomenon was observed: for some specific base models, the fixing particular starting token cues could lead the model to achieve comparable performance of its RL version. Causal interventions on the data could build new “starting token cues” or remove existing cues.

Critical strengths
  1. This work builds connections between the “starting token cues” that activate reasoning behaviors and the training data for specific base LLM models, which is interesting.
Critical weaknesses
  1. The major weakness of this work locates in the generalization ability. Does this property hold across different LLM backbones? Is the phenomenon or “starting token cues” consistent among them? Does it apply to all types of tasks (including long-context tasks)? The fact that it does not work on Llama is a major concern.
  2. The authors are suggested to answer: should this phenomenon be avoided during pre-training? Could better data cleaning and mixing ratios substantially mitigate this issue? Would this phenomenon affect the effectiveness of future post-training and reinforcement learning? Maybe the current “starting token cues” are “bugs” rather than “features”. In this way, more industrial LLMs should be evaluated.
Questions for authors
  1. Pre-training and pre-training data serve as the foundation of the following post-training/RL stages, which has already gained widespread recognition within the community. I feel the authors need to provide a detailed discussion of the related literature and clearly articulate the contributions that are unique to this paper.
8 comments

Thank you for your review. We are glad that you found the connection between starting token cues and training data interesting and that you “learned something interesting or useful”. We believe that many of the concerns raised are addressed by experiments and discussion already included in the manuscript, which we clarify below.

“The major weakness of this work locates in the generalization ability. Does this property hold across different LLM backbones?”

Our experiments demonstrate token-cue effects across multiple LLM backbones spanning three model families: Olmo, Qwen, and SmolLM. We study model scales from 3B to 32B, across multiple mathematics benchmarks and coding (Appendix C and Tables 5–6). Moreover, we conducted controlled training-data interventions on both Olmo3-7B and SmolLM3-3B (Section 3.1 and App. F). This provides cross-family evidence consistent with the scope of related reasoning studies. For instance, s1 [1] evaluates its proposed recipe on Qwen2.5-32B-Instruct, DAPO [2] studies its RL method on Qwen2.5-32B, and the Spurious Rewards phenomenon [3] is largely confined to the Qwen2.5 family, with the same rewards generally failing to improve Llama and Olmo.

“...industrial LLMs should be evaluated.”

We note that our paper includes Qwen3-4B and Qwen3-14B released by Alibaba, which is considered an “industrial LLM.” We use Olmo3 and SmolLM3 for the causal experiments because their training data and checkpoints are public. To test token cues at an even larger scale, we ran our search on Qwen2.5-72B-Base, yielding the cue “To determine.” Below, we report the pass@1 rates along with standard deviation over 32 rollouts. The token cue improves performance on almost all benchmarks tested.

BenchmarkNo cue" To determine"
MATH-5000.612 ± 0.0170.751 ± 0.012
AIME240.079 ± 0.0510.156 ± 0.034
AIME250.037 ± 0.0280.119 ± 0.042
AMC230.331 ± 0.0680.578 ± 0.044
OlympiadBench0.267 ± 0.0200.448 ± 0.017
HumanEval0.771 ± 0.0300.771 ± 0.028

“Does [the phenomenon of token cues] apply to all types of tasks (including long-context tasks)?”

Our study focuses on mathematical reasoning and coding, established settings for studying reasoning behavior and the effects of post-training [4, 5, 6]. We do not claim generalization to all task types. Extending our analysis to long-context inputs and multi-turn, long-horizon agentic tasks is an interesting direction for future work. We will mention this as future work in the discussion.

“Is the phenomenon of ‘starting token cues’ consistent among [the different LLM backbones]?” and “The fact that it does not work on Llama is a major concern.”

Our paper shows that token cues can be discovered and used to improve the performance of Olmo3, Qwen3, and SmolLM3 base model families across various reasoning benchmarks. The exact token cue is not the same across models. For instance, Olmo3-7B has the token cue “.\n\nOkay” while Qwen3-4B has “To determine.” This variation is consistent with our claim that the effectiveness of token cues depends on associations each model learns from its own training data. This is supported by our controlled data interventions on Olmo3 and SmolLM3.

Our data-centric account offers a plausible explanation for the Llama-3.1-8B result (Sec. 6 and App. C.5). A token cue helps when the training data pairs that opening with reasoning data, so the cue can elicit existing reasoning behavior. We find that Llama-3 shows poor existing reasoning behavior: 94% of its rollouts for MATH-500 don’t even have an extractable answer (App. C.5). Prior work finds similar model dependence for reasoning. Gandhi et al. [7] found that Llama3 lacks reasoning behaviors Qwen already exhibits such as verification and backtracking. As we mention in L487-490: “... we find no effective cue for Llama-3.1-8B among the openings tested (App. C.5), in line with prior reports of limited gains from RL on Llama and stronger gains after math-focused base-model training (Wang et al., 2025b; Liu et al., 2025).” We do not claim for token cues to be a universal phenomenon, and the Llama exception is consistent with our account and does not undermine our broader findings.

“The authors are suggested to answer: should this phenomenon be avoided during pre-training? Could better data cleaning and mixing ratios substantially mitigate this issue? Would this phenomenon affect the effectiveness of future post-training and reinforcement learning? Maybe the current ‘starting token cues’ are ‘bugs’ rather than ‘features’.”

These are all very interesting questions. We kindly note that they concern the broader implications of our work rather than the soundness of our methods or results. We briefly discuss implications for RL and synthetic-data biases in Section 7, and for dataset design and model safety in our Ethics Statement. To elaborate, one implication is that we can deliberately pair cues with useful behaviors in training data to make those behaviors easier to elicit later. Post-training could potentially be made more efficient by focusing mainly on learning when to generate these cues. The cue-forced RL results already suggest that supplying the cue can reduce the updates needed to reach a given accuracy (e.g., Fig. 12). However, changing only the opening can also increase harmful responses to unsafe requests (Sec. 4). So, the cues could be thought of as both a “bug” and a “feature.” Deliberately creating useful cues would also require identifying and addressing cue associations that elicit unsafe behavior.

“Pre-training and pre-training data serve as the foundation of the following post-training/RL stages … need to provide a detailed discussion of the related literature and clearly articulate the contributions that are unique to this paper.”

We agree that the importance of pre-training for subsequent post-training is well established. We note that Section 5 already discusses related work, with the final sentence of each paragraph clarifying how our work builds on and differs from prior studies. We have also updated the PDF with an expanded related works section. Our unique and original contributions are:

  1. To show that short, fixed token openers can elicit reasoning from base models that is competitive with RLVR in several evaluated settings.

  2. To identify a mechanism by which RLVR can improve reasoning by increasing the probability of these cues.

  3. To establish how token cue–behavior associations are learned through controlled training-data interventions.

These are supplemented by additional analyses, such as how cues change representations inside the model (Sec. 3.2), their effect on test-time compute (Sec. 2.4), and a case study on refusal behavior (Sec. 4). While prior work has shown that pre-/mid-training develops reasoning capabilities that RL can build on, as Reviewer fPs5 says, our work “goes beyond merely showing latent base-model reasoning.” Reviewer znSf’s comment captures this distinction: “most of the work on analyzing how RL changes the distribution and behaviours of models is correlational, and the use of training data in this work to provide stronger empirical proof is much stronger and well motivated.”

We appreciate that you found the phenomenon “interesting.” As the review does not seem to point to specific errors in our methods or results, we would be grateful to hear any remaining soundness concerns so we can address them. Otherwise, we hope you might consider revisiting your assessment.

[1] Muennighoff et al. “s1: Simple test-time scaling.” EMNLP 2025.

[2] Yu et al. “DAPO: An open-source LLM reinforcement learning system at scale.” NeurIPS 2025.

[3] Shao et al. “Spurious Rewards: Rethinking Training Signals in RLVR.” ICML 2026.

[4] Wei et al. “Chain of thought prompting elicits reasoning in large language models.” NeurIPS 2022.

[5] Zhao et al. “Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining.” COLM 2025.

[6] Yue et al. “Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?” NeurIPS 2025.

[7] Gandhi et al. “Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs.” COLM 2025.

Thank you for the timely and candid clarification. It offers many useful leads for a thorough discussion of the questions I raised.

My core concern is that work on "starting token cues" needs more rigorous evidence on three aspects. The first is generalizability, across different backbones and different downstream tasks. The second is practicality: is the method actually usable in real settings, and do the latest models, such as this year's flagship open-source models, still show the same issue or phenomenon? The third is how deeply the mechanism of "starting token cues" is understood. Does the method mainly provide some form of instruction-guiding ability, or does it only mitigate a specific "bug"? Could the problem be alleviated through some mechanism during LLM pre/post training (or is it a good feature in training)?

These questions are not meant to be nitpicking. I genuinely hope the discussion can lead to a "starting token cues" method that is effective in practice and grounded in a correct understanding of the mechanism. My main concern is that the proposed method may only uncover certain biases that different backbones acquire during pre-training (possibly from their training data), offer some degree of control over them, and validate this on a few models. However, similar phenomena have already been discussed in prior work. This paper does not yet give a more fundamental explanation, and it is not yet clear how its findings could improve the training pipelines and recipes of current flagship models. I would appreciate it if the authors could address these points with deeper discussions, and I will modify my voting accordingly.

Reviewer_znSfOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how training data creates associations between the first tokens of a base model outputs and the reasoning behaviours they exhibit. The work finds correlations between the accuracy of model outputs for code and reasoning and the presence of starting token cues in model outputs. They demonstrate that RL makes the use of these cues more likely and using these cues in base models recovers gains in RL performance. The work then traces the reasoning effects of the token cues and how it relates to document categories in the training data. The study also presents a case study into the use and impact of such token cues in an AI safety setting.

Critical strengths
  1. Training Data analysis and causal evidence. The experiments for relating the base model performance and correlation of the reasoning cues with the training data are well desgined and implemented. In my experience, most of the work on analyzing how RL changes the distribution and behaviours of models is correlational, and the use of training data in this work to provide stronger empirical proof is much stronger and well motivated.

  2. Clear evidence for how RL gains relate to opening tokens for MATH-500. The experiments to show that RL training uses and focuses on the first few generated tokens for MATH-500 during training are comprehensive and informative. The experiments on how a base model using an RL generated opening performs well while an RL post trained model with a base opening performs similar to the base model provide strong empirical support for the key claim in the paper. However, I have some concerns certain parts of this experimental setup, which will be outlined in the weakness.

Critical weaknesses
  1. The analysis looks only at the opening, while the evidence suggests RL also changes behavior during the trajectory, especially on harder problems. The study is built on the hypothesis that RL mainly makes good opening tokens more likely. However RL may also instill intermediate behaviours that a base model lacks, such as backtracking, switching strategies and recovery after an error which are triggered by tokens like "wait" or "hold on" during the reasoning generation. While we could hypothesize that such tokens are only generated in reasoning traces incentivized by the opening cues the work finds, there is no clear evidence to support this. Additionally, I believe we can see how this tracks as the difficulty of the evaluations increases in the experiments reported in the paper, with the prefilled base model performing worse on AMC 23, AIME 2024, AIME 2025 and so on as compared to MATH-500, where it almost matches the RL performance. I think the caption in figure 2 which mentions "often matching or exceeding" is actually going past this without deeper investigation which may be warranted here. I believe that the per position KL averaging which the work does cannot properly tackle this issue. Backtracking tokens are rare and occur at varying positions during the course of response generation, so large but sparse differences at these decision points will not affect the average being measures at these positions. There have been prior work which has already found that RL gains in math and code can concentrate on such high-entropy fork tokens spread across the reasoning chain (https://arxiv.org/abs/2506.01939). This goes against the key claims of the paper on the opening tokens being the key driver for RL improvements, and I believe that the selection of the evaluation and the measurement strategy using mean of per position KL may bias the evidence in favour of the hypothesis.. The swap experiment for opening cues is only performed on MATH-500, which is arguably the easiest benchmark selected. I believe such a pattern would hold up in this evaluation since it contains problems which require little recovery from mid-trajectory errors.

I believe a few experiments could help address this and make the findings more meaningful. The most direct would be to repeat the swap experiment on the harder benchmarks such as AIME 2024, AIME 2025 and OlympiadBench, and to make the switch between models at later points in the response (for example after 500 or 2,000 tokens) instead of only after the opening. If RL’s advantage on these benchmarks happens in later parts of the reasoning trajectory, the base model continuing an RL opening would fall short of RL there. The KL analysis could also be made sensitive to these mid-trajectory changes by conditioning on events instead of simply averaging by position (for example by measuring KL at positions where either model generates backtracking tokens like “wait” or “hold on”, or at high entropy positions in general).

To check whether the cued base model can actually recover from errors in the way an RL trained model can, recovery could be tested directly by prefilling a trajectory with a plausible wrong step, and comparing how often each model detects and corrects it. Similarly, it would be useful to count how often backtracking events occur in the cued base and RL responses, and how often each actually leads to a corrected answer, since the phrase counts currently only show that such phrases appear and not whether they help. Lastly, I believe running RL with gradients blocked at the first two response positions while still letting the model sample its own opening would be helpful. If this still reaches RL level accuracy, then RL is finding gains beyond the opening, which would go directly against the paper’s central claim.

  1. The 'base' models have already been trained on synthetic reasoning traces, which impacts the findings in the paper. The Olmo 3 7B mid training mix contains synthetic thinking data drawn from other reasoning models, which contain such reasoning behaviours and cues which the work identifies. As noted in the paper, "okay" opens in 83% of these traces. Similarly, SmolLM 3 also introduces reasoning traces in its decay stage and qwen3 base models also have a reasoning focused pretraining whos contents are not fully documented. Based on this, I have concerns about how the paper attributes the reasoning and the observed gains on base models.

The paper attributes the effect of the cues to training data associations, and frames the result as evidence that reasoning is already present in the base model and that RL mainly amplifies/sharpens it. However, there are at least two possible sources for the reasoning that follows the cue. The first is general pretraining on math textbooks and web text, which would support the paper’s hypothesis. The second is distillation of reasoning model outputs through these synthetic traces during mid training. The experiments in the paper do not separate the two. The data edit experiments show convincingly that the trigger, i.e. which token unlocks the behaviour, is learnt from training data. However, they do not show where the reasoning ability behind the trigger comes from, which is the more important question for the paper’s claims about RL. I believe the paper’s own results point towards the traces as a major source. In Fig. 20, the cue provides almost no gain at the end of pretraining, and the gain only appears and grows during mid training, which is exactly when the traces are introduced. Llama-3.1-8B, which was trained before such traces were common, has no effective cue at all (App. C.5). The behaviour that follows the cue also looks very much like a reasoning model trace, with checking and backtracking phrases such as “Wait, let me double-check”. This ties into my first concern as well, since the backtracking the cued model produces may be reproduced from these traces rather than coming from the model’s own pretraining.

If this is the case, then the comparison with RL is affected too. The finding that two tokens “match RL” would then show that RL-Zero adds little to a model which has already been distilled on the outputs of RL trained reasoning models. It would not show that RL mainly surfaces reasoning acquired in general pretraining, which is different from framing in the paper. Since the authors already re-run Olmo mid training for the Chicken experiments, a run with the synthetic trace sources removed, followed by both a cue search and RL-Zero, would directly test this attribution. Varying the share of these traces in the mid training mix would be even more informative, since it would show how the cue gains and the RL gains each change as the model is exposed to more of these traces. If the cue gains remain without the traces, this would support the paper’s attribution of the reasoning to the base model’s pretraining. However, if the cue gains disappear without the traces, then much of the reasoning the paper attributes to the base model would instead come from distillation of reasoning model outputs, and the comparison with RL would need to be interpreted accordingly.

  1. The novelty of the work is overstated, and several closely related works are not discussed. The paper presents its core findings as discoveries, but I believe several of them have clear precedents in prior work which the paper does not cite. The idea that a short prefix can make a base model behave like its post-trained counterpart has already been shown by The Unlocking Spell on Base LLMs (https://arxiv.org/abs/2312.01552), which found that base and aligned models decode almost identically at most token positions, with the differences concentrated on stylistic tokens and at earlier positions. The same work showed that base models behave much like aligned models when given the suitable prefix, and Nudging (https://arxiv.org/abs/2410.09300) built on this by inserting aligned model tokens at those positions to steer a base model. The per position KL analysis in Sec. 2.3 is essentially the same analysis applied to RLVR instead of alignment tuning, and I think the paper should position its findings as an extension of this line of work rather than as a new discovery.

The paper also makes two specific claims about RL which I believe overlap with prior work. The first is that RL changes the base policy mainly at the first few output positions. This again is close to the findings of The Unlocking Spell on Base LLMs. The second is that RL gains largely come from up-weighting reasoning behaviours already present in the base model, and that fixing the cue recovers much of this gain. This fits a well established line of work, including Spurious Rewards (https://arxiv.org/abs/2506.10947) and Understanding R1-Zero-Like Training (https://arxiv.org/abs/2503.20783), which found that RL amplifies behaviours the base model already favours. Cognitive Behaviors that Enable Self-Improving Reasoners (https://arxiv.org/abs/2503.01307) also shows that whether RL can make use of such behaviours depends on whether they were present in the training data, which closely relates to the paper’s data edit experiments. I believe the RL findings should be positioned as confirming these results through a new mechanism (the opening token cue) rather than as new findings in themselves.

I believe the genuinely novel contributions of the work are the controlled mid training data edits, which establish causally that a specific trigger token may be learnt from training data, and the specific finding that two fixed tokens can match a released RL-Zero model on MATH-500. These are valuable, but they are narrower than the current claims of the work. I would like the paper to position the work clearly against the above, and to state which of their findings are new and which confirm existing results in a new setting.

Questions for authors
  1. Could you run the ablation experiments suggested in the weakness 1? It would help improve the findings and insights from the experiments.
  2. Please report paired problem level CI for the main comparisons. The reported delta is the SD over the rollout indices, which does not account for variation across problems on small sets like AIME and AMC 23.

Clarifications

  • The main text refers to "Qwen3-14B," while the appendix tables say "Qwen3-14B-Base," and App. C.4 evaluates the post-trained Qwen3-14B. Since Qwen3-14B was released with reasoning post-training already applied, please state exactly which checkpoint each main-text experiment uses.
6 comments

Thanks for the detailed comments! We appreciate your positive assessment of our work, especially of the controlled causal training-data interventions. Your suggestions helped us clarify how RLVR provides additional gains beyond opening cues.

“RL also changes behavior during the trajectory, especially on harder problems…”

We agree that token cues do not explain all of RLVR gains. Our claim is that token cues “can explain a large portion of RL gains” (L482–483) in our settings. We have run the suggested experiments below, and will add them to highlight the remaining benefits of RLVR beyond the opening. We will explain the larger gap on harder benchmarks, and have also adjusted Figure 2’s caption accordingly in the updated PDF.

“...repeat the swap experiment on the harder benchmarks…”

Following your suggestion, we repeated the swap experiment comparison on AIME 2024/2025 and OlympiadBench using Olmo3-7B and its RL-Zero variant. We let the base model continue the RL opener at three points: right after the opening cue, after 500 tokens, and after 2,000 tokens. We report the pass@1 below along with 95% CI paired differences to RL. The RL opening brings the base model to approximately cued-base performance, but longer RL prefixes do not close the remaining gap. These results support some additional RLVR benefits during the continuation after the opener.

BenchmarkSwitch pointRL → baseRL aloneDifference [95% CI]
AIME 2024After the opening0.2350.352−0.117 [−0.185, −0.033]
After 500 tokens0.2290.352−0.123 [−0.181, −0.048]
After 2,000 tokens0.2290.352−0.123 [−0.165, −0.065]
AIME 2025After the opening0.2290.300−0.071 [−0.156, 0.000]
After 500 tokens0.2330.300−0.067 [−0.144, −0.010]
After 2,000 tokens0.2330.300−0.067 [−0.123, −0.031]
OlympiadBenchAfter the opening0.3650.416−0.051 [−0.088, −0.027]
After 500 tokens0.3550.416−0.061 [−0.097, −0.038]
After 2,000 tokens0.3830.416−0.033 [−0.064, −0.015]

“...recovery could be tested directly by prefilling a trajectory with a plausible wrong step…”

We tested a proxy for the suggested error-recovery experiment. We selected responses from RL-Zero and the cued Olmo-3-7B base model that ended with an incorrect answer on MATH-500. We took at most one per problem from each model, resulting in 114 RL and 67 cued-base rollouts. We then truncated each response after 512 tokens or halfway through, and then let both models continue from the same prefix. The table reports pass@1 and paired bootstrap 95% CIs. RL reaches a correct answer from the same failing prefix more often, which could help explain its stronger performance on harder benchmarks.

Prefix source# TrajectoriesHandoffBase continuesRL continuesRL − base [95% CI]
RL-Zero114512 tokens0.3250.482+0.157 [+0.053, +0.263]
RL-Zero114Halfway0.1490.351+0.202 [+0.105, +0.298]
Cued base67512 tokens0.2090.403+0.194 [+0.090, +0.313]
Cued base67Halfway0.0900.299+0.209 [+0.090, +0.328]

“...it would be useful to count how often backtracking events occur…how often each actually leads to a corrected answer…”

We follow your suggestion using the paper’s Olmo-3-7B and RL-Zero rollouts across all math benchmarks we tested. For each backtracking marker (a sentence starting with “Wait”, “Hmm”, “Actually”, “Let me check”, … similar to App. G.2), we check whether a wrong answer was stated before it and corrected later. RL writes more of these markers per response than the base model with a cue (e.g., about 3x on AIME). Most are “Wait” loops in responses that hit the token cap. In both models, over 90% of markers come before any answer is given. After a wrong answer, both models correct it at the same rate (8.2% base vs. 7.9% RL pooled over benchmarks). Together with the recovery experiment above, whose prefixes end before any answer is stated, this suggests RL’s advantage lies in continuing partial solutions rather than revising stated answers. We will add this to App. G.2.

“…high-entropy fork tokens spread across the reasoning chain…goes against the key claims of the paper on the opening tokens being the key driver for RL improvements.” and “The KL analysis could also be made sensitive … at high entropy positions in general.”

Following your suggestion, we measured KL(RL ‖ base) at the opening, at backtracking markers (“wait”, “hmm”, “hold on”, ...), and at the 20% highest-entropy positions. Similar to Fig. 3, we teacher-force RL-Zero rollouts through both models (2 per problem on AIME 2024). Per token, the opening is where RL changes the policy most, and backtracking markers don’t change as much. In total, however, most of the KL lies at high-entropy positions spread through the response, in line with [1].

We will report these results and cite [1]. The token cues are compatible with the spread-out fork tokens. The opening is a fixed position (which allows prefilling) with the largest per-token KL. The remaining policy change is smaller per token but distributed across high-entropy fork tokens, where most of the total KL lies. [1] selects 10% or more of tokens by entropy at positions that vary across responses. This doesn't isolate the two opening tokens, which make up under 0.05% of a response. We also extend the evidence beyond Qwen3, the main focus of [1], by showing opening effects in Olmo3 and SmolLM3 and using controlled data edits to establish how these cues acquire their effects during mid-training.

PositionsKL vs. average tokenShare of total KL
Opening (first two tokens)60×1.1%
Backtracking markers2.3×1.9%
Top 20% entropy3.1×62%

“RL with gradients blocked at the first two response positions…”

Our paper claims that fixing the cue “can explain a large portion of RL gains” (L482–483), but not all of it. For instance, our own results show cases of RL beating the token cue, especially on harder benchmarks. We therefore read “gains beyond the opening” as asking whether RL can reach strong performance by a route that does not use the opening cue. We ran GRPO for 300 steps on MATH (App. D setup) with gradients blocked at the first two tokens for Olmo3-7B as you suggested. Note that this is different from the Olmo3 RL-Zero model. Even with no gradient at the opening, RL selects the cue “.\n\nOkay” indirectly because updates from later positions can change the opener through shared parameters. As shown in the table below, the masked model reaches the standard RL-level accuracy (pass@1±SD) while starting almost all of its rollouts with the token cue. The fact that RL still converges on ".\n\nOkay" is consistent with the learned association between the cue and the reasoning that follows.

ModelMATH-500Opens with .\n\nOkayAIME 2024Opens with .\n\nOkay
Base0.422 ± 0.02227.4%0.120 ± 0.05039.4%
Base + cue0.779 ± 0.015100% (forced)0.233 ± 0.063100% (forced)
GRPO (ours, App. D)0.832 ± 0.02692.6%0.267 ± 0.06993.1%
GRPO (ours), opening masked0.830 ± 0.02785.7%0.250 ± 0.06287.1%

“The ‘base’ models have already been trained on synthetic reasoning traces…”

We appreciate this point and think we are largely in agreement. The review refers to “the paper’s attribution of the reasoning to the base model’s pretraining,” and writes that a trace-based account “would not show that RL mainly surfaces reasoning acquired in general pretraining, which is different from framing in the paper.” We want to note that we make neither claim. Our hypothesis is that “particular tokens trigger reasoning behavior because of associations learned during training” (L078–079), and the data we point to are the synthetic reasoning traces (e.g., Fig. 1 and Fig. 7), and Section 6 states that both models’ “training mixes include synthetic reasoning traces” (L492). We agree that the synthetic traces can be more prominently foregrounded in the abstract and introduction, and have updated the PDF accordingly.

Our use of “base model” follows common usage, as in the Olmo 3 report [2], whose Section 2.1 states: “We develop Olmo 3 Base in three stages of pretraining … midtraining …” Thus, “already present in the base model” describes capabilities present before the RLVR stage we study. It does not imply that they arose without synthetic data. We have updated the introduction to state that base models are trained on general text corpora and, more recently, on mid-training mixes that include synthetic data such as reasoning traces.

The reviewer notes that our data edits show the trigger is learned from training data, but not where the ability behind it comes from. We agree with this distinction, and it matches the question our paper asks: “if reasoning may already be present in the base model, then how can the model actually access that ability?” (L047–048). As we state in Section 5, we investigate “one mechanism by which RLVR can promote existing behaviors: increasing the probability of response openings that already elicit stronger reasoning” (L469–471), and we do not claim to show where the reasoning ability behind the trigger comes from. A full account of the origins of these capabilities is beyond the scope of this paper, and an open research question.

Finally, we thank the reviewer for the suggestion that “varying the share of these traces in the mid-training mix would be even more informative.” The paper includes a version of this experiment in App. F.2.1. Raising the amount of traces in SmolLM3-3B’s data from 0.55% to 10% increases cued MATH-500 pass@1 from 0.193 to 0.251 and uncued pass@1 from 0.054 to 0.095.

“...novelty of the work…”

Thank you for the pointers! We have uploaded a PDF with an extended related work section. We cited The Unlocking Spell (URIAL) [3] and Nudging [4] alongside the closely related “Safety Alignment Should Be Made More Than Just a Few Tokens Deep“ [5], which we already cite in Sec. 4. The works you brought up study alignment tuning and highlight changes in stylistic tokens, such as discourse markers and disclaimers. URIAL uses demonstrations and a system prompt to steer behavior. Nudging relies on an additional aligned model to select tokens dynamically throughout generation. So, the token-level interventions differ from ours. While alignment tuning can involve RL, we study reinforcement learning with verifiable rewards (RLVR) for reasoning. RLVR has its own line of work and is credited with improving reasoning, so findings about stylistic shifts from alignment do not tell us whether reasoning gains are also concentrated at the opening. We will clarify throughout that we mean RLVR for reasoning rather than RL in general.

Regarding RLVR amplifying existing behaviors, Sec. 5 cites such work and frames our result as “one mechanism by which RLVR can promote existing behaviors” (L469-471). We have added Spurious Rewards [6] and Cognitive Behaviors [7] in our updated related works section. Understanding R1-Zero is already cited in Sec. 6. Overall, we view our contributions as:

  1. Showing that short, fixed token openers can elicit reasoning from base models that can be competitive with RLVR in several evaluated settings.

  2. Identifying a mechanism by which RLVR can improve reasoning by increasing the probability of these fixed cues.

  3. Establishing how token cue–behavior associations are learned through controlled training-data interventions.

Beyond these points, the paper examines the phenomenon from several independent angles: such as looking at hidden-state representations (Sec. 3.2, App. G), interpreting RL weight geometry (App. D.4–D.5), analyzing the use of test-time compute (Sec. 2.4, App. E).

Reviewer_fPs5OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how short response openings access reasoning behaviors already present in base language models. It selects a fixed one- or two-token cue using answer agreement on MATH training problems, evaluates transfer to held-out math and coding benchmarks, and analyzes how RL changes opening probabilities. Controlled mid-training interventions rename reasoning-associated words and reassign their contexts, creating or redirecting cue effects in Olmo and SmolLM3. Hidden-state comparisons and an XSTest case study provide supporting behavioral analyses. The principal contribution is evidence connecting particular training associations to the behavior elicited by a response opening.

Critical strengths
  1. The controlled data interventions are the strongest and most informative contribution. On Olmo, renaming makes Chicken effective, while reassignment sharply reduces Okay-cued accuracy without eliminating the effects of other cues (Fig. 5; Tables 20–23). SmolLM3 reproduces the directional pattern at its published trace share, as well as after upsampling (Tables 16–17). This goes beyond merely showing latent base-model reasoning. Wang and Zhou already explored alternative initial decoding tokens, Karan and Du elicited strong reasoning through sampling, and Zhao et al. connected RL to pretraining output formats. Here, the specific lexical association is experimentally changed. Ruis et al. also include retraining controls, so the distinction is the targeted manipulation of behavioral cues, rather than causal retraining in general.

  2. The evidence is triangulated unusually well. Olmo-3-7B improves from 42.2% to 77.9% on MATH-500, and Qwen3-14B from 72.2% to 86.6% (Table 5). The comparison holds prompts and decoding fixed within model families. Common-cap rescoring and repeated-sampling controls address the simple explanation of longer generation (Fig. 4; Appendix E). Opening swaps provide more direct evidence than the KL plots alone: an RL opening followed by the base model reaches 77.7%, versus 75.1% for RL throughout (Table 13). Negative results and context dependence are reported, including Llama, code-transfer failures, and SmolLM3's retained Okay effect after a think tag.

Critical weaknesses
  1. The empirical results are not yet fully auditable at the level needed to assess generalization and close comparisons with RL. Appendix A specifies a training-only cue-selection procedure, and D.1 explicitly separates MATH training from MATH-500 evaluation; these address direct search/train/test separation. However, B and F do not report an overlap audit between evaluation problems and the synthetic continued-training corpora, or sufficiently identify the answer extractor/grader and its version. Code is deferred until acceptance. B.1 correctly explains that most error bars measure rollout variation, not independent training-run uncertainty or uncertainty over new problems. This does not undermine the large cue gains, but limits conclusions about small differences or equivalence to RL. The exact recipe also matters: on AIME 2024, cued Olmo reaches 23.3%, versus 35.2% for released RL (Tables 5–6).

  2. The safety case study supports changes in refusal behavior more strongly than improved helpfulness or broadly comparable safety. Figure 8 labels its horizontal axis as answering safe prompts, although the measured quantity is non-refusal. Table 33 includes a non-refusing response that merely reformulates the user's question. Appendix H.4 explicitly acknowledges that non-refusal need not mean compliance, but this does not resolve the axis interpretation. H.1 also grades the full base-model response, including deliberation and the prefill, while grading Think-SFT after its think block. With one automated judge and no reported manual calibration, the 0.2% harmful-response result and comparison to the post-trained references require care. H.3 usefully reports that the selective-refusal benefit does not transfer to Qwen.

Questions for authors
  1. What does RL improve beyond selecting the opening? Table 13 provides strong evidence on Olmo-3-7B/MATH-500, and Appendix D.2 already shows little additional gain from cue-forced training in some settings. Does this near-equivalence persist when the base and RL models receive exactly the same opening on a task with a remaining performance gap, such as Olmo on AIME 2024 (Tables 5–6)? A matched-opening, matched-token-budget comparison in one such setting, with paired problem-level differences, would clarify whether the remaining RL benefit comes from choosing better openings or improving the reasoning conditional on an opening.

  2. When label-free cue search fails, is the bottleneck candidate discovery or entropy-based selection? The 500-opening Olmo control in Appendix A is reassuring, but Appendix C.5 reports a lowest-entropy end-of-text candidate for Llama, and C.6 notes that search misses an effective Okay cue for SmolLM3. Could you use the existing candidate pools and evaluations to separate these two failure modes, and quantify how well training-set answer entropy predicts held-out accuracy after accounting for missing answers? This would clarify whether weak cue results indicate limited latent reasoning or merely unsuccessful elicitation.

6 comments

Thank you for your thoughtful review. We’re encouraged by your positive assessment of the paper’s clarity and informativeness. We also appreciate your observations that our “evidence is triangulated unusually well” and that our controlled data interventions go “beyond merely showing latent base-model reasoning.” We address the remaining concerns and questions below.

"...report an overlap audit between evaluation problems and the synthetic continued-training corpora…"

We compared every evaluation problem against Ai2's Olmo3 mid-training mix. Following [1], we flag a problem as contaminated if it shares any 13-gram with a training document. The table below reports pass@1±SD for Olmo3 over 32 rollouts on all problems and on the clean subset with the flagged problems removed. The cue's improvement over the uncued base is still there after excluding problems that were detected to overlap.

BenchmarkProblemsUncued baseCued baseRL-Zero
MATH-500Original (500)0.422 ± 0.0220.779 ± 0.0150.750 ± 0.015
Clean (406)0.424 ± 0.0240.778 ± 0.0160.750 ± 0.016
AMC 23Original (40)0.309 ± 0.0970.595 ± 0.0480.663 ± 0.063
Clean (28)0.272 ± 0.1160.557 ± 0.0630.663 ± 0.076
AIME 2024Original (30)0.120 ± 0.0500.233 ± 0.0630.352 ± 0.059
Clean (18)0.163 ± 0.0660.330 ± 0.0880.469 ± 0.073
AIME 2025Original (30)0.118 ± 0.0620.227 ± 0.0450.300 ± 0.058
Clean (20)0.141 ± 0.0750.291 ± 0.0560.345 ± 0.069
OlympiadBenchOriginal (300)0.247 ± 0.0220.365 ± 0.0190.416 ± 0.014
Clean (204)0.255 ± 0.0280.384 ± 0.0260.417 ± 0.019

“...identify the answer extractor/grader and its version.”

We extract the final answer in the format each prompt requests: the last \boxed{} under the boxed prompt, and the last "Answer:" line under the RL-Zero prompt, falling back to the last \boxed{} so that no model is penalized for its answer format alone. We check equivalence with Math-Verify v0.9.0, and responses without an extractable answer count as incorrect. We will add these details to App. B.1.

"B.1 correctly explains that most error bars measure rollout variation… This does not undermine the large cue gains, but limits conclusions about small differences or equivalence to RL.”

Thank you for the suggestion to report uncertainty over problems rather than over rollouts. We’ve recomputed the main pass@1 comparisons for Olmo-3-7B under the RL-Zero prompt. We report 95% CIs bootstrapped over problems using 10,000 resamples, and pair two models on each problem (table below). The cue’s improvement over the base model is significant on every math benchmark and on HumanEval. Compared with RL, the cued base model is slightly ahead on MATH-500, the differences on AMC 23, AIME 2025 and HumanEval are not significant, and RL is ahead on AIME 2024 and OlympiadBench.

BenchmarkUncued baseCued baseRL-ZeroCued − uncued [95% CI]RL − cued [95% CI]
MATH-5000.4220.7790.750+0.357 [+0.336, +0.372]−0.029 [−0.047, −0.005]
AMC 230.3090.5950.663+0.286 [+0.203, +0.343]+0.068 [−0.002, +0.165]
AIME 20240.1200.2330.352+0.113 [+0.065, +0.150]+0.119 [+0.060, +0.195]
AIME 20250.1180.2270.300+0.109 [+0.051, +0.188]+0.073 [−0.005, +0.141]
OlympiadBench0.2470.3650.416+0.118 [+0.095, +0.138]+0.051 [+0.031, +0.075]
HumanEval0.5020.6980.685+0.196 [+0.163, +0.226]−0.013 [−0.040, +0.014]

"What does RL improve beyond selecting the opening? … Does this near-equivalence persist when the base and RL models receive exactly the same opening on a task with a remaining performance gap, such as Olmo on AIME 2024…A matched-opening, matched-token-budget comparison in one such setting…clarify whether the remaining RL benefit comes from choosing better openings or improving the reasoning conditional on an opening."

Thank you for this suggestion. We ran this comparison on AIME 2024 using Olmo3-7B and the Ai2 RL-Zero model. The RL model writes the first set of tokens of each response, and the base model continues from there, with the same total 32k token budget as RL alone. We let the base model continue the RL opener at three points: right after the opening (through RL's first blank line and the two tokens after it, as in Table 13), after 500 tokens, and after 2,000 tokens written by the RL model. We report the pass@1 below along with 95% CI paired differences to RL.

ConditionAIME 2024 pass@1Paired difference to RL [95% CI]
Base0.120−0.232 [−0.316, −0.151]
Cued base0.233−0.119 [−0.195, −0.060]
RL start → base continues, after opening0.235−0.117 [−0.185, −0.033]
RL start → base continues, after 500 tokens0.229−0.123 [−0.181, −0.048]
RL start → base continues, after 2,000 tokens0.229−0.123 [−0.165, −0.065]
RL alone0.352—

With RL's opening on this benchmark, the base model reaches the level of the cued base model but not fully RL's. Letting RL write the first 500 or 2,000 tokens doesn't close the gap either. So on this benchmark, the results support that the remaining RL benefit comes from better reasoning given an opening, not from choosing a better opening. In fact, the RL-Zero model opens with the token cue “Okay," right after its first blank line in 99.6% of the rollouts across AIME 2024. We note that we do not claim an equivalence between token cues and RL, but that “Token cues can explain a large portion of RL gains” (L482–483) in our settings. We’ll add this new analysis to show how RL can still improve the reasoning conditioned on an opening.

"When label-free cue search fails, is the bottleneck candidate discovery or entropy-based selection? … quantify how well training-set answer entropy predicts held-out accuracy after accounting for missing answers?"

Thanks for this suggestion. We went back to the existing candidate pools from the search described in App. A.1 and found examples of both candidate-discovery and entropy-based selection failures. For SmolLM3 under the RL-Zero prompt, the search didn’t propose “Okay,” as we note in App. C.6, but it did find another effective cue. Under the boxed prompt, where the search selected no opening (App. C.1, Table 5), an effective cue was actually in the pool, but the entropy criterion didn’t select it.

We also checked how training-set answer entropy relates to held-out accuracy. When we keep only candidates that produce an extractable final answer in at least half of the screening responses, the Spearman correlation strengthens from −0.53 to −0.75 for Olmo and from −0.11 to −0.75 for SmolLM3, while remaining −0.91 for Qwen. So lower entropy is a useful signal, but can be misleading when responses rarely contain an extractable answer. We’ll add these results to clarify where the search works and where it can miss useful cues. Our goal in using the search is to find cues that let us study how these behaviors arise and how they connect to training data. Future work could look into more effective ways to discover these token cues.

"The safety case study supports changes in refusal behavior more strongly than improved helpfulness or broadly comparable safety. Figure 8 labels its horizontal axis as answering safe prompts, although the measured quantity is non-refusal. … H.1 also grades the full base-model response, including deliberation and the prefill, while grading Think-SFT after its think block. With one automated judge and no reported manual calibration, the 0.2% harmful-response result and comparison to the post-trained references require care."

We will emphasize that Sec. 4 mainly supports changes in refusal behavior. Our claim is that opening tokens change which requests the base model refuses, and that these changes accompany shifts toward different training sources. Following your suggestion:

  1. We will relabel Figure 8’s horizontal axis as “Non-refusal rate on safe prompts”, as well as remove the “Helpful and safe” and “Answers everything” quadrant labels.

  2. We will move the note that non-refusal doesn't necessarily mean compliance (App. H.4) into the main text, and the Qwen non-transfer (H.3) as well.

  3. We will state the grading-span difference (full base response vs. Think-SFT) as a limitation of the reference comparison. We’ll also report the 0.2% harmful-response rate as a WildGuard label rather than a guarantee of safe behavior.

We appreciate your suggestions, which have helped us strengthen the evidence and clarify the scope of our claims. We’re glad to hear that our evidence is “triangulated unusually well,” and given our additional experiments and the revisions above, we hope you will reconsider your overall assessment in favor of acceptance. We are happy to clarify anything else.

[1] Brown et al. “Language Models are Few-Shot Learners.” NeurIPS 2020.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.