- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper focuses on the connections between the first tokens a base LLM model outputs and its reasoning behaviors from the training data aspect. An interesting phenomenon was observed: for some specific base models, the fixing particular starting token cues could lead the model to achieve comparable performance of its RL version. Causal interventions on the data could build new “starting token cues” or remove existing cues.
Critical strengths
- This work builds connections between the “starting token cues” that activate reasoning behaviors and the training data for specific base LLM models, which is interesting.
Critical weaknesses
- The major weakness of this work locates in the generalization ability. Does this property hold across different LLM backbones? Is the phenomenon or “starting token cues” consistent among them? Does it apply to all types of tasks (including long-context tasks)? The fact that it does not work on Llama is a major concern.
- The authors are suggested to answer: should this phenomenon be avoided during pre-training? Could better data cleaning and mixing ratios substantially mitigate this issue? Would this phenomenon affect the effectiveness of future post-training and reinforcement learning? Maybe the current “starting token cues” are “bugs” rather than “features”. In this way, more industrial LLMs should be evaluated.
Questions for authors
- Pre-training and pre-training data serve as the foundation of the following post-training/RL stages, which has already gained widespread recognition within the community. I feel the authors need to provide a detailed discussion of the related literature and clearly articulate the contributions that are unique to this paper.
8 comments
“The authors are suggested to answer: should this phenomenon be avoided during pre-training? Could better data cleaning and mixing ratios substantially mitigate this issue? Would this phenomenon affect the effectiveness of future post-training and reinforcement learning? Maybe the current ‘starting token cues’ are ‘bugs’ rather than ‘features’.”
These are all very interesting questions. We kindly note that they concern the broader implications of our work rather than the soundness of our methods or results. We briefly discuss implications for RL and synthetic-data biases in Section 7, and for dataset design and model safety in our Ethics Statement. To elaborate, one implication is that we can deliberately pair cues with useful behaviors in training data to make those behaviors easier to elicit later. Post-training could potentially be made more efficient by focusing mainly on learning when to generate these cues. The cue-forced RL results already suggest that supplying the cue can reduce the updates needed to reach a given accuracy (e.g., Fig. 12). However, changing only the opening can also increase harmful responses to unsafe requests (Sec. 4). So, the cues could be thought of as both a “bug” and a “feature.” Deliberately creating useful cues would also require identifying and addressing cue associations that elicit unsafe behavior.
“Pre-training and pre-training data serve as the foundation of the following post-training/RL stages … need to provide a detailed discussion of the related literature and clearly articulate the contributions that are unique to this paper.”
We agree that the importance of pre-training for subsequent post-training is well established. We note that Section 5 already discusses related work, with the final sentence of each paragraph clarifying how our work builds on and differs from prior studies. We have also updated the PDF with an expanded related works section. Our unique and original contributions are:
-
To show that short, fixed token openers can elicit reasoning from base models that is competitive with RLVR in several evaluated settings.
-
To identify a mechanism by which RLVR can improve reasoning by increasing the probability of these cues.
-
To establish how token cue–behavior associations are learned through controlled training-data interventions.
These are supplemented by additional analyses, such as how cues change representations inside the model (Sec. 3.2), their effect on test-time compute (Sec. 2.4), and a case study on refusal behavior (Sec. 4). While prior work has shown that pre-/mid-training develops reasoning capabilities that RL can build on, as Reviewer fPs5 says, our work “goes beyond merely showing latent base-model reasoning.” Reviewer znSf’s comment captures this distinction: “most of the work on analyzing how RL changes the distribution and behaviours of models is correlational, and the use of training data in this work to provide stronger empirical proof is much stronger and well motivated.”
We appreciate that you found the phenomenon “interesting.” As the review does not seem to point to specific errors in our methods or results, we would be grateful to hear any remaining soundness concerns so we can address them. Otherwise, we hope you might consider revisiting your assessment.
[1] Muennighoff et al. “s1: Simple test-time scaling.” EMNLP 2025.
[2] Yu et al. “DAPO: An open-source LLM reinforcement learning system at scale.” NeurIPS 2025.
[3] Shao et al. “Spurious Rewards: Rethinking Training Signals in RLVR.” ICML 2026.
[4] Wei et al. “Chain of thought prompting elicits reasoning in large language models.” NeurIPS 2022.
[5] Zhao et al. “Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining.” COLM 2025.
[6] Yue et al. “Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?” NeurIPS 2025.
[7] Gandhi et al. “Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs.” COLM 2025.
Thank you for the timely and candid clarification. It offers many useful leads for a thorough discussion of the questions I raised.
My core concern is that work on "starting token cues" needs more rigorous evidence on three aspects. The first is generalizability, across different backbones and different downstream tasks. The second is practicality: is the method actually usable in real settings, and do the latest models, such as this year's flagship open-source models, still show the same issue or phenomenon? The third is how deeply the mechanism of "starting token cues" is understood. Does the method mainly provide some form of instruction-guiding ability, or does it only mitigate a specific "bug"? Could the problem be alleviated through some mechanism during LLM pre/post training (or is it a good feature in training)?
These questions are not meant to be nitpicking. I genuinely hope the discussion can lead to a "starting token cues" method that is effective in practice and grounded in a correct understanding of the mechanism. My main concern is that the proposed method may only uncover certain biases that different backbones acquire during pre-training (possibly from their training data), offer some degree of control over them, and validate this on a few models. However, similar phenomena have already been discussed in prior work. This paper does not yet give a more fundamental explanation, and it is not yet clear how its findings could improve the training pipelines and recipes of current flagship models. I would appreciate it if the authors could address these points with deeper discussions, and I will modify my voting accordingly.
Thank you for your review. We are glad that you found the connection between starting token cues and training data interesting and that you “learned something interesting or useful”. We believe that many of the concerns raised are addressed by experiments and discussion already included in the manuscript, which we clarify below.
“The major weakness of this work locates in the generalization ability. Does this property hold across different LLM backbones?”
Our experiments demonstrate token-cue effects across multiple LLM backbones spanning three model families: Olmo, Qwen, and SmolLM. We study model scales from 3B to 32B, across multiple mathematics benchmarks and coding (Appendix C and Tables 5–6). Moreover, we conducted controlled training-data interventions on both Olmo3-7B and SmolLM3-3B (Section 3.1 and App. F). This provides cross-family evidence consistent with the scope of related reasoning studies. For instance, s1 [1] evaluates its proposed recipe on Qwen2.5-32B-Instruct, DAPO [2] studies its RL method on Qwen2.5-32B, and the Spurious Rewards phenomenon [3] is largely confined to the Qwen2.5 family, with the same rewards generally failing to improve Llama and Olmo.
“...industrial LLMs should be evaluated.”
We note that our paper includes Qwen3-4B and Qwen3-14B released by Alibaba, which is considered an “industrial LLM.” We use Olmo3 and SmolLM3 for the causal experiments because their training data and checkpoints are public. To test token cues at an even larger scale, we ran our search on Qwen2.5-72B-Base, yielding the cue “To determine.” Below, we report the pass@1 rates along with standard deviation over 32 rollouts. The token cue improves performance on almost all benchmarks tested.
“Does [the phenomenon of token cues] apply to all types of tasks (including long-context tasks)?”
Our study focuses on mathematical reasoning and coding, established settings for studying reasoning behavior and the effects of post-training [4, 5, 6]. We do not claim generalization to all task types. Extending our analysis to long-context inputs and multi-turn, long-horizon agentic tasks is an interesting direction for future work. We will mention this as future work in the discussion.
“Is the phenomenon of ‘starting token cues’ consistent among [the different LLM backbones]?” and “The fact that it does not work on Llama is a major concern.”
Our paper shows that token cues can be discovered and used to improve the performance of Olmo3, Qwen3, and SmolLM3 base model families across various reasoning benchmarks. The exact token cue is not the same across models. For instance, Olmo3-7B has the token cue “.\n\nOkay” while Qwen3-4B has “To determine.” This variation is consistent with our claim that the effectiveness of token cues depends on associations each model learns from its own training data. This is supported by our controlled data interventions on Olmo3 and SmolLM3.
Our data-centric account offers a plausible explanation for the Llama-3.1-8B result (Sec. 6 and App. C.5). A token cue helps when the training data pairs that opening with reasoning data, so the cue can elicit existing reasoning behavior. We find that Llama-3 shows poor existing reasoning behavior: 94% of its rollouts for MATH-500 don’t even have an extractable answer (App. C.5). Prior work finds similar model dependence for reasoning. Gandhi et al. [7] found that Llama3 lacks reasoning behaviors Qwen already exhibits such as verification and backtracking. As we mention in L487-490: “... we find no effective cue for Llama-3.1-8B among the openings tested (App. C.5), in line with prior reports of limited gains from RL on Llama and stronger gains after math-focused base-model training (Wang et al., 2025b; Liu et al., 2025).” We do not claim for token cues to be a universal phenomenon, and the Llama exception is consistent with our account and does not undermine our broader findings.